Celery for separate django docker microservices

I have 2 django microservices main_ms and Oms1_ms.

the project name of both is config and both have these installed

redis==4.3.4
celery==5.2.7
celery[redis]==5.2.7

both have celery.py. main_ms's is like this

import os
from celery import Celery
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'proj.settings')
app = Celery('proj')
app.config_from_object('django.conf:settings', namespace='CELERY')
app.autodiscover_tasks()

and for Oms1_ms, instead of proj I have oms1_oms1

in both __init__.pys I have

from .celery import app as celery_app
__all__ = ('celery_app',)

in both settings I have provided

CELERY_BROKER_URL='redis://redis:6379/0'
CELERY_RESULT_BACKEND='redis://redis:6379/0'

from one of views I have in main_ms which calls the other celery

from config.celery import app
from rest_framework import views
from rest_framework import status
from rest_framework.response import Response
class Oms1_Oms1ListView(views.APIView):
    def get(self, request, *args, **kwargs):
        remoteResponse = app.send_task('oms1_oms1.fun')
        print(remoteResponse)
        return Response({}, status=status.HTTP_200_OK)

and in Oms1 app in Oms1_ms container in tasks.py I have

from config.celery import app
@app.task()
def fun():
    return 'as always!!'

so when I run celery -A oms1_oms1 worker -l DEBUG command in Oms1 container I get this error

Error: Invalid value for '-A' / '--app':
Unable to load celery application.
The module oms1_oms1 was not found.

so how to set celery with django docker microservices correctly?

I also know one of celery uses is primarily to process workloads within a service. but finding an example of celery for microservices with conditions below has been a cumbersome task to me.

  1. all django projects are separate, each on their own container and they don't share a parent folder, so when calling tasks we cant import the function handling it unlike the @shared_task of celery.
  2. it has depicted how to run celery worker command.
  3. how to setup and attach celery instances to rabbit/redis or mongodb.

you need to set path of the directory where your celery config file resides with the __init__.py, example if I have my celery config file in settings/celery_config.py, the command is

celery -A settings worker -l info
# or
celery -A settings.celery_config worker -l info
Back to Top