How to Use Celery for Scheduling Tasks

There are multiple ways to schedule tasks in your Django app, but there are some advantages to using Celery. It’s supported, scales well, and works nicely with Django. Given its wide use, there are also lots of resources for learning more about it, and once learned, that knowledge is likely to be useful on other projects.

By the way, this blog is an updated version of one I wrote in 2014 on the same topic, but it applied to Celery 3.0.x.

Celery Version 5.0.x

This documentation applies to Celery 5.0.x. Earlier or later versions of Celery might behave differently. Also, links to Celery documentation might stop working if newer versions of Celery reorganize the documentation, which does happen.

Introduction to Celery

The purpose of Celery is to allow you to run code according to a schedule. Why might this be useful? Here are a couple of common cases:

Case 1: Suppose a web request has come in from a user, who is waiting for the request to complete so a new page can load in their browser. Based on their request, you have some code to run that's going to take a while (longer than the person might want to wait for a web page), but you don't really need to run that code before responding to the web request. You can use Celery to have your long-running code called later, and go ahead and respond immediately to the web request.

This is common if you need to access a remote server to handle the request. Your app has no control over how long the remote server will take to respond, or the remote server might be down.

Case 2: Another common situation is wanting to run code regularly. For example, maybe every hour you want to look up the latest weather report and store the data. You can write a task to do that work, then ask Celery to run it every hour. The task runs and puts the data in the database, and then your web application has access to the latest weather report.

Some Celery Terminology:

A task is just a Python function. You can think of scheduling a task as a time-delayed call to the function. For example, you might ask Celery to call your function task1 with arguments (1, 3, 3) after five minutes. Or you could have your function batchjob called every night at midnight.

When a task is ready to be run, Celery puts it on a queue, a list of tasks that are ready to be run. You can have many queues, but we'll assume a single queue here for simplicity.

Putting a task on a queue just adds it to a to-do list, so to speak. In order for the task to be executed, some other process, called a worker, has to be watching that queue for tasks. When it sees tasks on the queue, it'll pull off the first and execute it, then go back to wait for more. You can have many workers, possibly on many different servers, but we'll assume a single worker for now.

We'll talk more later about the queue, the workers, and another important process that we haven't mentioned yet, but that's enough for now.

Back to Top