Celery Worker Disconnects But Continues To Run Task

I have 8 workers in celery all running on different machines that run very long tasks (multiple hours)

These tasks are able to be stopped with app.control.revoke(task_id, terminate=True) and are visible in app.control.instpect()

What happens is that some order of events causes the workers to disconnect from RabbitMQ and are no longer visible within the app.control.instpect() method, but the underlying code continues to run within the worker.

These tasks are also not able to be terminated. And what would cause something like this to happen?

Is there a way to prevent this disconnect from happening? And what would cause something like this to happen?

The worker is started with celery -A project worker -l info -P eventlet --concurrency 5 > /dev/null

Back to Top