Celery for separate django docker microservices
I have 2 django microservices main_ms
and Oms1_ms
.
the project
name of both is config
and both have these installed
redis==4.3.4
celery==5.2.7
celery[redis]==5.2.7
both have celery.py
. main_ms
's is like this
import os
from celery import Celery
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'proj.settings')
app = Celery('proj')
app.config_from_object('django.conf:settings', namespace='CELERY')
app.autodiscover_tasks()
and for Oms1_ms
, instead of proj
I have oms1_oms1
in both __init__.py
s I have
from .celery import app as celery_app
__all__ = ('celery_app',)
in both settings I have provided
CELERY_BROKER_URL='redis://redis:6379/0'
CELERY_RESULT_BACKEND='redis://redis:6379/0'
from one of views I have in main_ms
which calls the other celery
from config.celery import app
from rest_framework import views
from rest_framework import status
from rest_framework.response import Response
class Oms1_Oms1ListView(views.APIView):
def get(self, request, *args, **kwargs):
remoteResponse = app.send_task('oms1_oms1.fun')
print(remoteResponse)
return Response({}, status=status.HTTP_200_OK)
and in Oms1
app in Oms1_ms
container in tasks.py
I have
from config.celery import app
@app.task()
def fun():
return 'as always!!'
so when I run celery -A oms1_oms1 worker -l DEBUG
command in Oms1
container I get this error
Error: Invalid value for '-A' / '--app':
Unable to load celery application.
The module oms1_oms1 was not found.
so how to set celery
with django docker microservices correctly?
I also know one of celery uses is primarily to process workloads within a service. but finding an example of celery for microservices with conditions below has been a cumbersome task to me.
- all django projects are separate, each on their own container and they don't share a parent folder, so when calling tasks we cant import the function handling it unlike the
@shared_task
of celery. - it has depicted how to run celery worker command.
- how to setup and attach celery instances to rabbit/redis or mongodb.
you need to set path of the directory where your celery config file resides with the __init__.py
,
example if I have my celery config file in settings/celery_config.py
, the command is
celery -A settings worker -l info
# or
celery -A settings.celery_config worker -l info