Celery polling Redis every second

I have a Django application with two celery tasks set up. Both the tasks need to run only once at one minute after midnight (12:01 AM). This is just to update some statuses which change from "ACTIVE" to "INACTIVE" at 12:01 AM.

My goal is to reduce the number of times Celery polls Redis. Currently, it polls every second which is problematic.

I took three steps

1. I tried to change the settings in Django for Celery inlcuding using paramters such as CELERY_IGNORE_RESULT, CELERY_WORKER_CONCURRENCY, CELERY_BROKER_HEARTBEAT and lastly CELERY_BEAT_MAX_LOOP_INTERVAL

   CELERY_RESULT_BACKEND = None
   CELERY_IGNORE_RESULT = True  # Ignore task results to reduce Redis usage
   
   CELERY_ACCEPT_CONTENT = ["json"]
   CELERY_TASK_SERIALIZER = "json"
   
   CELERYD_PREFETCH_MULTIPLIER = 1  # Fetch only 1 task at a time
   CELERY_ACKS_LATE = True  # Acknowledge tasks after completion to avoid re-execution on failure
   
   # Celery transport options
   CELERY_BROKER_TRANSPORT_OPTIONS = {
       'ssl': {
           'ssl_cert_reqs': ssl.CERT_NONE  # Suggestion: CERT_REQUIRED for stronger security
       },
       'visibility_timeout': 3600,  # Timeout for long-running tasks
       'polling_interval': 30  # Poll every 30 seconds
   }
   
   CELERY_RESULT_BACKEND_TRANSPORT_OPTIONS = {
       'ssl': {
           'ssl_cert_reqs': ssl.CERT_NONE  # Same as broker for consistency
       }
   }
   # Reduce the number of threads/processes
   CELERY_WORKER_CONCURRENCY = 1
   
   # Disable sending task events
   CELERY_WORKER_SEND_TASK_EVENTS = False
   
   # Disable remote control commands
   CELERY_WORKER_ENABLE_REMOTE_CONTROL = False
   
   # Disable broker heartbeat messages
   CELERY_BROKER_HEARTBEAT = None
   
   CELERY_BEAT_MAX_LOOP_INTERVAL = 3600  # Check for scheduled tasks every hour
   

2. Secondly, I even tried to set up dynamic polling so that the polling schedule is more frequent in during night time and reduced again

   def set_polling_interval():
       now = datetime.now()
       if now.hour == 23 and now.minute >= 59 or now.hour == 0 and now.minute <= 15:
           # Between 11:59 PM and 12:15 AM, poll more frequently
           app.conf.broker_transport_options = {'polling_interval': 30}  # Poll every 30 seconds
           logger.info("Polling every 30 seconds between 11:59 PM and 12:15 AM.")
       else:
           # During the rest of the day, poll less frequently (every 24 hours)
           app.conf.broker_transport_options = {'polling_interval': 86400}  # Poll every 24 hours
           logger.info("Polling every 24 hours outside the midnight window.")
   

3. Lastly, because I have Django Celery Beat installed, I tried tweaking the settings there directly in the "Intervals" and "Crontabs" section, but that did not help either.

Despite these efforts, Celery is still polling Redis every second. I checked the documentation for optimization, but struggled to find a solution there https://docs.celeryq.dev/en/stable/userguide/optimizing.html

The following article was also referred so that maybe I can do the opposite of it but that did not work either

Can I use celery for every-second task?

Вернуться на верх