Как настроить кластер redis для celery.py?

Я пытаюсь запустить свой celery worker, но постоянно сталкиваюсь с ошибками. Я использую кластер aws elasticache redis и запускаю его на экземпляре ec2.

# celery.py

from __future__ import absolute_import, unicode_literals
import os
from celery import Celery
from django.conf import settings  # Import Django settings
from celery.schedules import crontab
from redis.cluster import RedisCluster

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'shipninja.settings')

redis_cluster_nodes = [
    {'host': 'rediscluster-0001-001.rediscluster.m0rn9y.use1.cache.amazonaws.com', 'port': 6379},
    {'host': 'rediscluster-0002-001.rediscluster.m0rn9y.use1.cache.amazonaws.com', 'port': 6379},
    {'host': 'rediscluster-0003-001.rediscluster.m0rn9y.use1.cache.amazonaws.com', 'port': 6379},
]

broker_url = 'redis://' + ','.join([f'{node["host"]}:{node["port"]}' for node in redis_cluster_nodes])
result_backend = broker_url


app = Celery('shipninja')
app.config_from_object('django.conf:settings', namespace='CELERY')
app.autodiscover_tasks(['app'])  # This will discover tasks in all registered Django apps


app.conf.beat_schedule = {
    'aggregate-and-charge-users-weekly': {
        'task': 'path.to.aggregate_and_charge_users_weekly',  # Use the correct path to your task
        'schedule': crontab(hour=0, minute=0, day_of_week='monday'),
    },
    'update_inventory_items': {
        'task': 'app.tasks.update_inventory_items',
        'schedule': 60,
    },
    'update-shipments-every-minute': {
        'task': 'app.tasks.update_shipments_cache',  
        'schedule': 300,  
    },
    'update_package_statuses_every_minute': {
        'task': 'app.tasks.update_package_statuses',
        'schedule': 28800,  
    },
    'refresh-access-tokens': {
        'task': 'app.tasks.refresh_tokens_task',
        'schedule': 3600,  
    },
}

Я пробовал использовать redis-py-cluster с redis 3.5.3:

          ^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/rediscluster/connection.py", line 160, in __init__
    self.nodes.initialize()
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/rediscluster/nodemanager.py", line 300, in initialize
    raise RedisClusterException("Redis Cluster cannot be connected. Please provide at least one reachable node.")
rediscluster.exceptions.RedisClusterException: Redis Cluster cannot be connected. Please provide at least one reachable node.

При использовании redis-py с redis 5.0.2 я получаю:

    self.parse_response(connection, "_")
  File "/home/ubuntu/venv/lib/python3.10/site-packages/redis/client.py", line 1430, in parse_response
    result = Redis.parse_response(self, connection, command_name, **options)
  File "/home/ubuntu/venv/lib/python3.10/site-packages/redis/client.py", line 560, in parse_response
    response = connection.read_response()
  File "/home/ubuntu/venv/lib/python3.10/site-packages/redis/connection.py", line 536, in read_response
    raise response
redis.exceptions.ResponseError: Command # 1 (LLEN 25194542-8c99-351f-94a5-a5db13ebad14.reply.celery.pidbox) of pipeline caused error: MOVED 1545 rediscluster-0001-001.rediscluster.m0rn9y.use1.cache.amazonaws.com:6379

Любая помощь будет оценена по достоинству. Спасибо.

Вернуться на верх