Django logger only logging to console from celery worker process

I have a following logging config:

"root": {
    "level": "ERROR",
    "handlers": ["console", "server_file"],
},
"handlers": {
    "celery_file": {
        "class": "logging.handlers.RotatingFileHandler",
        "level": "DEBUG",
        "filename": os.path.join(ROOT_LOGS_DIR, "celery", "celery.log"),
        "formatter": "standard",
        "maxBytes": 1024 * 1024 * 50,  # 50 MB
        "backupCount": 30,
    },
    "server_file": {
        "class": "logging.handlers.RotatingFileHandler",
        "level": "DEBUG",
        "filename": os.path.join(ROOT_LOGS_DIR, "server.log"),
        "formatter": "standard",
        "maxBytes": 1024 * 1024 * 50,  # 50 MB
        "backupCount": 30,
    },
"loggers": {
    # '': {'handlers': ["root"], "level": "DEBUG", "propagate": True, },
    'django': {'handlers': ['django'], 'level': 'DEBUG', 'propagate': False, },        
    "celery": {
        "handlers": ["celery_file"],
        "level": "INFO",
        "propagate": True,
    },

From celery task,

log = logging.getLogger("celery.inbound_outbound_jobs")
log.error("Hello, World!") # this won't be written to files. Only Console.

I run this task using celery-beat and worker.

celery -A log_project beat --loglevel=info
celery -A log_project worker --loglevel=info

But this is not writing logs to server.log and celery.log.

Only console logging is happening.

When running this task directly from django server and not as celery worker process, this is working in all celery.log, server.log and console.

enter image description here

Can anyone help understand this?

the issue is that your celery.inbound_outbound_jobs logger is not configured in your logging configuration and since your celery logger is set to propagate: True, it should propagate logs to the root logger but however your root logger only has the handlers "console" and "server_file" but not "celery_file".

explicitly define celery.inbound_outbound_jobs in the Logging Configuration Add the logger explicitly in the loggers section

"loggers": {
    "django": {"handlers": ["django"], "level": "DEBUG", "propagate": False},
    "celery": {"handlers": ["celery_file"], "level": "INFO", "propagate": True},
    "celery.inbound_outbound_jobs": {
        "handlers": ["celery_file"],
        "level": "INFO",
        "propagate": True,
    },
}

Also ensure celery uses django’s logging settings in your celery.py file inside your django project, modify it as follows

import logging
from celery import Celery
app = Celery("log_project")

app.config_from_object("django.conf:settings", namespace="CELERY")

from django.conf import settings
import logging.config
logging.config.dictConfig(settings.LOGGING)
app.autodiscover_tasks()

when you log using logging.getLogger("celery.inbound_outbound_jobs"), it should correctly write to celery.log

Back to Top