Asyncio coroutine execution stopped and didn't return any error

I'm facing an odd behavior on one of my applications. So I have an application that has an POST entrypoint that receives a list of order id's and after processing each order it must inform an external API about the monetary value of each order. So what we did was the following:

round_details_tasks = [
        asyncio.create_task(
            self.launch_individual_order_details(order),
            name=f"task_order_{order.id}",
        )
        for order in active_orders
    ]
    results = await asyncio.gather(*order_details_tasks, return_exceptions=True)
    for task, result in zip(order_details_tasks, results):
        if isinstance(result, Exception):
            print(f"⚠️ Task '{task.get_name()}' raised an exception: {result}")
        else:
            print(f"✅ Task '{task.get_name()}' succeeded with result: {result}")

The launch_individual_order_details(order) function does the following stuff and other non I/O logic before this block:

logger.debug(f"Sending order details with success for order: {order.id}")    
await order_service.send_order_request(order)

Inside send_order_request we create a entry on a table called Transaction with the order id and the corresponding order amount and in pending state and send http request using aiohttp.CLient library. Afterwards we update the transaction status to Success if the request response is succesful and to error if the request fails.

So the problem we are facing is that when our system is with a relative amount of load in our pods when the pod uses 70% of the CPU limits we give to them we notice that some of our tasks simply break execution and don't inform the event loop about any possible error.

We discovered this because after some load testing we ran some scripts to check transaction status and noticed that some order id's only had the transaction status to pending so they didn't even execute the request to the external api. Furthermore we noticed that for all the orders that were in this state they didn't appear in the succes task log:

print(f"✅ Task '{task.get_name()}' succeeded with result: {result}").

and they also don't appear on the error task log:

print(f"⚠️ Task '{task.get_name()}' raised an exception: {result}").

Furthermore we also turned the event loop logs debug to true and don't find any kind of error associated with these tasks. The only thing we found out is that the execution reaches this log:

logger.debug(f"Sending order details with success for order: {order.id}") 

and that the transaction state for this order is created with pending state but is never updated again.

Does anyone have any idea why these cases may be happening?

Вернуться на верх