Django Celery Beat SQS slow scheduling
Beat seems to be sending the messages into SQS very slowly, about 100/minute. Every Sunday I have a sendout to about 16k users, and they're all booked for 6.30pm. Beat starts picking it up at the expected time, and I would expect a huge spike in messages coming into SQS at that time, but it takes its time, and I can see on the logs that the "Sending tasks x..." goes on for a few hours. I expect ~16k messages to go out around 6.30pm, and for the number of messages processed and deleted to pick up as the autoscale sets in.
I have autoscaling on for my Celery workers, but because the number of messages doesn't really ever spike, the workers don't really scale until later, when the messages start backing up a bit.
I'm really puzzled by this behaviour, anyone there know what I could be missing? I'm running celery with, some cron tab tasks but this one task in specific is a PeriodicTask
celery_beat: celery -A appname beat --loglevel=INFO
