Celery and Redis Command Overload Despite Optimizations in Django App

I’m facing an issue with my Celery + Redis setup for a Django project where the number of Redis commands being executed is skyrocketing despite having made several optimizations. I’m using Upstash Redis with the free tier, which has a daily limit of 10,000 commands. However, within an hour of starting the app, I’m already hitting thousands of Redis commands, primarily BRPOP, PUBLISH, and PING. This is happening even when the app is idle.

Context: Django Version: 4.2 Celery Version: 5.4 Redis Broker: Upstash Redis Free Tier Deployment: DigitalOcean (App platform) App Details: The app is primarily used for scheduling and sending reminder emails using Celery tasks. Here’s a brief overview:

I have Celery workers running for email sending tasks. There is no celery.beat setup. Tasks are invoked on-demand by the application logic. The app is relatively idle most of the time, with tasks being triggered only a few times a day. Optimizations I’ve Already Tried: Result Backend Disabled: I set CELERY_RESULT_BACKEND = None to avoid unnecessary reads from Redis. Heartbeat Adjustment: I increased the CELERY_BROKER_TRANSPORT_OPTIONS['heartbeat'] to 120 seconds, hoping it would reduce the command load. Prefetch Multiplier: Set CELERY_WORKER_PREFETCH_MULTIPLIER = 1 to ensure that only one task is fetched by workers at a time. SSL Configurations: Enabled SSL in the transport options as recommended by Upstash:

CELERY_BROKER_TRANSPORT_OPTIONS = {
    'visibility_timeout': 3600,
    'ssl': {
        'ssl_cert_reqs': 'CERT_REQUIRED',
        'ssl_ca_certs': BASE_DIR / 'certs/upstash-ca-chain.pem',
    },
    'heartbeat': 120,
}

Even with these optimizations, Redis command usage rapidly increases to thousands of commands within an hour, even when the app is not actively performing any tasks. For example, within the last hour, Redis logged over 1,000 BRPOP commands, 525 PUBLISH commands, and several PING and AUTH commands.

Below is a graph showing Redis command usage over the past hour:

Redis Command Breakdown (for 1 hour): BRPOP: 1,000 commands PUBLISH: 525 commands PING: 67 commands AUTH: 36 commands Others like LLEN, MULTI, EXISTS, etc.

Is it normal for Celery workers to generate this many Redis commands even when the app is idle? What could be the cause of such high BRPOP and PUBLISH command usage? Are there any additional optimizations I can make to reduce the Redis command load? Would switching from Upstash to a fully managed Redis instance on DigitalOcean (or elsewhere) help reduce command overhead, or is this behavior unrelated to the Redis provider? Any insights or advice would be greatly appreciated! I'm hoping to stay within the limits of the free tier while ensuring my Celery tasks can still run efficiently.

Thanks in advance!

Back to Top