Django-redis/uWSGI- Connection Closed errors after timeout set to 0

I am using Nautobot (Django-based application) that utilises Redis and Celery for asynchronous task execution.

Randomly, when accessing the jobs page, I get the below error stating the Redis server connection was closed.

enter image description here

Originally, I had Redis running in a container and later migrated to Google MemoryStore as part of the troubleshooting. Both platforms have the same issue.

I've tried updating the redis configuration on both to disable timeouts:

timeout 0

As well as setting the configuration in Django but the issue continues.

# CACHE Configuration
CACHE_RETRY = Retry(ExponentialBackoff(), 3)
CACHES = {
    "default": {
        "BACKEND": "django_prometheus.cache.backends.redis.RedisCache",
        "LOCATION": f"{os.getenv('CACHE_BROKER_URL', 'redis://127.0.0.1:6379')}/0",
        "TIMEOUT": 300,
        "OPTIONS": {
            "CLIENT_CLASS": "django_redis.client.DefaultClient",
            "PASSWORD": f"{os.getenv('CACHE_AUTH_STRING', '')}",
            "SOCKET_CONNECT_TIMEOUT": 10,
            "SOCKET_TIMEOUT": 0,
            "CONNECTION_POOL_KWARGS": {
                "ssl_cert_reqs": os.getenv('CACHE_CERT_REQUIRED', 'None'),
                "retry": CACHE_RETRY,
                "retry_on_timeout": True
            }
        },
    }
}

CONTENT_TYPE_CACHE_TIMEOUT = int(os.getenv("CACHE_CONTENT_TYPE_TIMEOUT", "0"))
REDIS_LOCK_TIMEOUT = int(os.getenv("CACHE_LOCK_TIMEOUT", "0"))
CELERY_BROKER_URL = os.getenv("CACHE_BROKER_URL", "redis://127.0.0.1:6379")
CELERY_TASK_SOFT_TIME_LIMIT = int(os.getenv("CACHE_TASK_SOFT_TIME_LIMIT", str(5 * 60)))
CELERY_TASK_TIME_LIMIT = int(os.getenv("CACHE_TASK_TIME_LIMIT", str(10 * 60)))
CELERY_BROKER_TRANSPORT_OPTIONS = {
    "ssl": {
        "ssl_cert_reqs": os.getenv('CACHE_CERT_REQUIRED', 'None')
    }
}
CELERY_RESULT_BACKEND_TRANSPORT_OPTIONS = {
    "ssl": {
        "ssl_cert_reqs": os.getenv('CACHE_CERT_REQUIRED', 'None')
    }
}

if os.getenv('CACHE_CERT_REQUIRED', 'None') == 'required':
    CACHES['default']['OPTIONS']['CONNECTION_POOL_KWARGS']['ssl_ca_certs'] = os.getenv('CACHE_CERT_LOCATION', '')
    CELERY_BROKER_TRANSPORT_OPTIONS['ssl']['ssl_ca_certs'] = os.getenv('CACHE_CERT_LOCATION', '')
    CELERY_RESULT_BACKEND_TRANSPORT_OPTIONS['ssl']['ssl_ca_certs'] = os.getenv('CACHE_CERT_LOCATION', '')

    # Construct the new URL for Celery
    schema, host = os.getenv("CACHE_BROKER_URL", "redis://127.0.0.1:6379").split('://')
    CELERY_BROKER_URL = f"{schema}://:{os.getenv('CACHE_AUTH_STRING', '')}@{host}"

We are not seeing any connectivity problems between the hosts, they were running on the same machine before moving to Memorystore.

Versions:

django-redis==5.4.0
redis==5.0.8
uwsgi==2.0.25.1-r1

The application is running in k8s and I'm using Istio Service Mesh to act as the 'web server' while uWSGI is serving the application.

Back to Top