Automatically Retrying Failed Celery Tasks

In this article, we'll look at how to automatically retry failed Celery tasks.

Objectives

After reading, you should be able to:

  1. Retry a failed Celery task with both the retry method and a decorator argument
  2. Use exponential backoff when retrying a failed task
  3. Use a class-based task to reuse retry arguments

Celery Task

You can find the source code for this article on GitHub.

Let's assume we have a Celery task like this:

@shared_task
def task_process_notification():
    if not random.choice([0, 1]):
        # mimic random error
        raise Exception()

    requests.post('https://httpbin.org/delay/5')

In the real world this may call an internal or external third-party service. Regardless of the service, assume it's very unreliable, especially at peak periods. How can we handle failures?

It's worth noting that many Celery beginners get confused as to why some articles use app.task while others use shared_task. Well, shared_task lets you define Celery tasks without having to import the Celery instance, so it can make your task code more reusable.

Back to Top