Celery Beat sending all periodic tasks as due on restart even if they have just run

I am running a celery worker and celery beat with DatabaseScheduler.

I am running these on Heroku along with a django web application. They are being used to send emails and texts at regular intervals, Once per day at midnight for emails and 730am for texts.

I have defined the tasks manually using a chrontab in Periodic Tasks table, but I have also tried defining them in the codebase using celeryapp.conf.beat_schedule, both behave the same.

The tasks look like this

periodic tasks in django admin

The issue I am having is that Heroku restarts its dynos once per day as a policy, and when this happens, the celery beat dyno runs my periodic tasks for some reason. The debug log reads "Scheduler: Sending due task" as if it is normal, whether they are due then or not. I do not know why this is happening.

I have celery results running as well with postgres backed task results and that looks like this, where I have the tasks running at the correct times (midnight and 730am), but also at some random time when heroku restarts the dynos.

How can I stop the tasks from running when this restart happens? Why are they running at all after the restart?

task results in django admin

While I didn't verify the exact cause, I think I found the solution as it didn't happen again today, will update if its not fixed though. If beat doesn't see last_run_at it uses datetime.now(). I believe somehow the calculation of (last run + delta - now) was coming back negative, causing it to run at the start.

I noticed a setting inside the code of beat called "DJANGO_CELERY_BEAT_TZ_AWARE" that I did not have set. I also think there was a mistake in my settings with USE_TZ false as well.

I have celery settings mapped to "CELERY_", and this was my before and after in settings.py:

Before:

CELERY_ENABLE_UTC = False
TIME_ZONE = "America/New_York"
CELERY_TIMEZONE = "America/New_York"
USE_TZ = False 

After (No longer ocurring):

CELERY_ENABLE_UTC = False
CELERY_TIMEZONE = "America/New_York"
DJANGO_CELERY_BEAT_TZ_AWARE = True
TIME_ZONE = "America/New_York"
USE_TZ = True
Back to Top