Celery + RabbitMQ random Connection reset by peer

I'm using celery in django with rabbitmq, it works fine but some times it gives ConnectionResetError

traceback:

[2024-09-18 07:08:31,427: ERROR/MainProcess] Error cleaning up after event loop: RecoverableConnectionError(None, 'Socket was disconnected', None, '')
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/celery/worker/loops.py", line 97, in asynloop
    next(loop)
  File "/usr/local/lib/python3.11/site-packages/kombu/asynchronous/hub.py", line 306, in create_loop
    item()
  File "/usr/local/lib/python3.11/site-packages/vine/promises.py", line 161, in __call__
    return self.throw()
           ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/vine/promises.py", line 158, in __call__
    retval = fun(*final_args, **final_kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/kombu/transport/base.py", line 230, in _read
    drain_events(timeout=0)
  File "/usr/local/lib/python3.11/site-packages/amqp/connection.py", line 526, in drain_events
    while not self.blocking_read(timeout):
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/amqp/connection.py", line 531, in blocking_read
    frame = self.transport.read_frame()
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/amqp/transport.py", line 312, in read_frame
    payload = read(size)
              ^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/amqp/transport.py", line 629, in _read
    s = recv(n - len(rbuf))
        ^^^^^^^^^^^^^^^^^^^
ConnectionResetError: [Errno 104] Connection reset by peer

when checking rabbitmq logs

(172.21.0.1:50078 -> 172.21.0.2:5672): user 'test' authenticated and granted access to vhost '/'
rabbitmq-1  | 2024-09-19 07:43:51.816951+00:00 [error] <0.43132.0> closing AMQP connection <0.43132.0> (172.21.0.1:50072 -> 172.21.0.2:5672):
rabbitmq-1  | 2024-09-19 07:43:51.816951+00:00 [error] <0.43132.0> missed heartbeats from client, timeout: 60s
rabbitmq-1  | 2024-09-19 07:45:10.121070+00:00 [notice] <0.86.0>     alarm_handler: {set,{system_memory_high_watermark,[]}}
rabbitmq-1  | 2024-09-19 07:46:30.182591+00:00 [info] <0.43323.0> accepting AMQP connection <0.43323.0> (172.21.0.1:45394 -> 172.21.0.2:5672)
rabbitmq-1  | 2024-09-19 07:46:30.183948+00:00 [info] <0.43323.0> connection <0.43323.0> (172.21.0.1:45394 -> 172.21.0.2:5672): user 'test' authenticated and granted access to vhost '/'
rabbitmq-1  | 2024-09-19 07:46:30.185316+00:00 [info] <0.43144.0> closing AMQP connection <0.43144.0> (172.21.0.1:50078 -> 172.21.0.2:5672, vhost: '/', user: 'test')
rabbitmq-1  | 2024-09-19 07:46:30.185762+00:00 [info] <0.43335.0> accepting AMQP connection <0.43335.0> (172.21.0.1:45402 -> 172.21.0.2:5672)
rabbitmq-1  | 2024-09-19 07:46:30.187055+00:00 [info] <0.43335.0> connection <0.43335.0> (172.21.0.1:45402 -> 172.21.0.2:5672): user 'test' authenticated and granted access to vhost '/'
rabbitmq-1  | 2024-09-19 07:49:30.185926+00:00 [error] <0.43323.0> closing AMQP connection <0.43323.0> (172.21.0.1:45394 -> 172.21.0.2:5672):
rabbitmq-1  | 2024-09-19 07:49:30.185926+00:00 [error] <0.43323.0> missed heartbeats from client, timeout: 60s

I'm using a dockerized rabbimtq

compose service:

rabbitmq:
    image: "rabbitmq:3-management"
    ports:
      - "5672:5672"
    environment:
        RABBITMQ_DEFAULT_USER: "test"
        RABBITMQ_DEFAULT_PASS: "test"
        RABBITMQ_DEFAULT_VHOST: "/"

any ideas?

it happens a lot in many queues celery -A Celery_App worker --pool=solo -l info --logfile=/chatgbt-api/logs/report.logs -Q send_report_queue -c 5

Back to Top