Django Celery: 6-second delay to register task from UI in Kubernetes environment – resource allocation?
We are seeing a ~6-second delay when a Celery task is triggered via the Django UI (e.g., my_task.delay()). Our stack runs on Kubernetes, and I'm wondering if this lag is due to resource constraints or something else.
Key Services and Their Resource Configuration:
We are running several stateful services. Their approximate pod resource configurations are:
- 1 x JanusGraph: Requests 3 CPU, Limits 4 CPU
- 2 x Cassandra: Each requests 3 CPU, Limits 4 CPU
- 2 x Elasticsearch: Each requests 3 CPU, Limits 4 CPU
- We also have other applications running on these nodes.
Kubernetes Node Allocation:
Here's the kubectl describe node output for our two relevant nodes, showing current resource allocation:
Node 1:
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 12410m (78%) 17400m (110%)
memory 14706Mi (23%) 25744Mi (41%)
ephemeral-storage 9Gi (1%) 18Gi (3%)
hugepages-1Gi 0 (0%) 0 (0%)
hugepages-2Mi 0 (0%) 0 (0%)
Node 2:
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 8510m (54%) 15900m (101%)
memory 16076Mi (25%) 28824Mi (46%)
ephemeral-storage 17Gi (3%) 34Gi (6%)
hugepages-1Gi 0 (0%) 0 (0%)
hugepages-2Mi 0 (0%) 0 (0%)
The Problem:
When a user action in the UI triggers a Celery task (e.g., my_task.delay()), there's a noticeable 6-second pause before the request completes and the task is presumably acknowledged by the Celery broker.
Questions:
Could the level of resource utilization on the Kubernetes nodes, particularly the overcommitted CPU requests and limits, be a primary cause for the delay in Celery task registration or might there be another reason??