Why does Celery with Redis backend create hundreds of connections despite max_connections limits

I am using Django + Celery with Redis as both the broker and the result backend. Here is my Celery configuration:

# Celery settings
CELERY_BROKER_URL = 'redis://redis:6379/1'
CELERY_RESULT_BACKEND = 'redis://redis:6379/2'

CELERY_BROKER_POOL_LIMIT = 10
CELERY_REDIS_MAX_CONNECTIONS = 10
CELERY_BROKER_TRANSPORT_OPTIONS = {'max_connections': 10}
CELERY_RESULT_BACKEND_TRANSPORT_OPTIONS = {'max_connections': 10}

CELERY_ACCEPT_CONTENT = ['application/json']
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'

WORKER_MAX_MEMORY_PER_CHILD_KB = 200_000
CELERY_WORKER_MAX_MEMORY_PER_CHILD = 200_000
CELERY_WORKER_MAX_TASKS_PER_CHILD = 1000

CELERY_BEAT_SCHEDULE = celery_beat_schedule
CELERY_TASK_DEFAULT_QUEUE = 'default'
CELERY_TASK_QUEUES = (
    Queue('default'),
    Queue('bind_status'),
)

CELERY_ONCE = {
    'backend': 'celery_once.backends.Redis',
    'settings': {
        'url': "redis://redis:6379/3",
        'default_timeout': 60*60,
    }
}

app = Celery('')
app.config_from_object('django.conf:settings', namespace='CELERY')
app.conf.worker_send_task_events = True
app.conf.task_send_sent_event = True
app.conf.task_track_started = True
app.conf.result_extended = True
app.conf.broker_connection_retry_on_startup = True
app.autodiscover_tasks()

I launch my workers like this:

celery -A giberno worker -Q bind_status --loglevel=info -P gevent -c 4

Expected Behavior

Given the max_connections=10 settings, I expected the total number of TCP connections to Redis (broker + result backend) to be around 10–20, especially with concurrency=4.

Observed Behavior

When I check the number of connections to Redis DB 2 (used as the result backend), I see hundreds of connections (100–400) even though the `max_connections` limits are set.

Questions

  1. Why does Celery create so many connections despite the max_connections limits?

  2. How can I correctly limit the number of TCP connections when using Redis as the result backend with gevent workers?

  3. Is there a way to make the connection pool actually shared across all greenlets and workers?

Вернуться на верх