Django SocketIO Connection Fails After RDS Restart – How to Handle Database Connectivity Issues?

I'm developing a Django application using SocketIO for real-time communication, and I'm encountering an issue where SocketIO connections fail after an RDS (Relational Database Service) restart, while my Django HTTP APIs continue to work fine.

Problem Description My Django application integrates with SocketIO for real-time features. After an RDS instance restart, the HTTP APIs function normally, but SocketIO connections encounter issues and fail to access database models. Specifically, I get errors related to database connectivity when attempting to handle SocketIO connections.

Code Snippets Here's how I configure my ASGI application and handle SocketIO connections:

ASGI Configuration (asgi.py):

import os
from django.core.asgi import get_asgi_application
import socketio
from backend.socketio import socketio_server

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'backend.settings.dev')
django_asgi_app = get_asgi_application()
application = socketio.ASGIApp(socketio_server=socketio_server.socketio_server, other_asgi_app=django_asgi_app)

SocketIO Connection Handler:

async def on_connect(self, sid: str, environ: dict):
    try:
        query_string = environ['asgi.scope']['query_string']
        token, chat_id = get_token_chat_id_from_query(query_string=query_string)
        if not token or not chat_id:
            raise ConnectionRefusedError("Invalid connection parameters.")
        
        user = await get_user_from_token(token=token)
        chat_obj = await get_chat_from_id(chat_id=chat_id, user=user)
        await update_all_chat_redis(chat_obj=chat_obj)
        
        async with self.session(sid=sid, namespace=self.namespace) as session:
            session['user_id'] = user.id
            session['chat_id'] = chat_obj.machine_translation_request_id
        
        await self.enter_room(sid=sid, room=chat_id)

    except ConnectionRefusedError as e:
        logger.error(f"Connection refused: {e}")
        raise

    except UserErrors as exc:
        logger.error(f"User error: {exc.message}")
        raise ConnectionRefusedError(exc.message)

    except Exception as e:
        logger.error(f"Unexpected error: {e}")
        raise ConnectionRefusedError("An unexpected error occurred. Please try again later.")

Database Access Function (@sync_to_async):

import logging
from django.utils import timezone
from django.db import OperationalError
from yourapp.exceptions import UserErrors

logger = logging.getLogger(__name__)

@sync_to_async
def get_user_from_token(token: str):
    retries = 3
    for attempt in range(retries):
        try:
            token_obj = AccessToken.objects.filter(token=token, expires__gt=timezone.now()).last()
            if not token_obj:
                raise UserErrors("Invalid Token!")
            return token_obj.user
        except OperationalError as e:
            logger.error(f"Database operational error on attempt {attempt + 1}: {e}")
            await asyncio.sleep(2)  # Wait before retrying
        except Exception as e:
            logger.error(f"Unexpected error on attempt {attempt + 1}: {e}")
            if attempt == retries - 1:
                raise
            await asyncio.sleep(2)  # Wait before retrying

What I've Tried

• Error Handling: Implemented detailed error logging in both SocketIO handlers and database access functions.
• Retry Logic: Added retry logic for database operations.
• SocketIO Integration: Verified that SocketIO is correctly integrated with Django ASGI. Question How can I resolve the issue where SocketIO connections fail after an RDS restart while HTTP APIs continue to function correctly? Are there specific best practices for handling database connectivity issues in a Django application that uses SocketIO? Any guidance on improving resilience or troubleshooting steps would be greatly appreciated.

Back to Top