How to properly integrate Opentelemetry while using fork process model for multiprocessing in python django app

We have a python/django application, in our app we are connecting to snowflake data warehouse and we have long running requests for reporting (sql queries are being used to fetch data from snowflake). To handle time consuming queries(and achieve multi processing) we were using Threading (Code is attached below). And We also had telemetry configured and it was working fine. Opentelemetry works fine while using Threads.

Now We have shifted from Thread approach to fork process model for multi processing and telemetry broke down, its not working as expected. We are unable to get it work.

We are using background_process_job decorator function to create a Thread on windows and Process on Linux, , When we test our code on windows(Thread is created), Telemetry works fine BUT When we test the same code on linux, a process is created instead of thread and telemetry does not work. Are we missing something?
Why telemetry @tracer.start_as_current_span("send-message-core-thread") does not work while using fork process and works fine when using Threads?

def background_process_job(function):
    def decorator(*args, **kwargs):
        # If windows, start a thread.
        if platform.system() == 'Windows':
            thread = Thread(target=function, args=args, kwargs=kwargs)
            thread.daemon = True
            thread.start()
            logger.debug(f"Thread started with id {thread.native_id}")
        else:
            # if linux, start a background process.
            process = Process(target=function, args=args, kwargs=kwargs)
            process.daemon = True
            process.start()
            logger.debug(f"Process started with id {process.pid}")
    return decorator

@background_process_job #will create a Thread on Windows and process on linux
@tracer.start_as_current_span("send-message-core-thread")
def send_message_core(request):
    #heavy processing sending traces

Solution that i have tried: (but did not work)
I tried post_fork function

# I created a gunicorn-telemetry.py file in root project with code 
def post_fork(server, worker):     
    server.log.info("server log: Worker spawned with pid: %s", worker.pid)     
# Ensure Azure Monitor tracing is set up for each worker process     
    configure_azure_monitor()

and in my docker-compose.yml

entrypoint: [
  "gunicorn",
  "-c" , 
  "gunicorn-telemetry.py", 
  "-b", "0.0.0.0:8000", 
  "algo_api.wsgi:application", 
  "--timeout 300"
]  

Back to Top