For running celery queues in docker, I want to specify multiple queues with different concurrency

Currently I am using docker compose in which I have specified celery as a service where q1,q2,q3 ,q4 and q5 are some different queues , and i want to assign each queue a different concurrency.

  celery:
    restart: always
    build: 
      context: 
      dockerfile: Dockerfile
    container_name: celery
    command: celery --app=celery:app worker --concurrency=8 --loglevel=DEBUG -Q q1,q2,q3,q4,q5
    depends_on:
      - redis
      - web

I read somewhere about celery multi , but dont know whether it fits my use case.

Multi does fit your use case.

celery multi start \
      q1w q2w q3w 
      -c:q1w 2 -c:q2w,q3w 4 
      -Q:q1 q1w -Q:q2 q2w -Q:q3 q3w

  • q1w -> q1 worker (we can start named nodes, q1w is the name).

  • We can define queue (-Q) and concurrency (-c) options for each node or multiple nodes (comma separated).

You can extend this with other options.

Refer this link for more details https://docs.celeryq.dev/en/stable/reference/celery.bin.multi.html

PS: you can do celery multi show ... and it will give you worker commands it will run.

Back to Top