Celery chord callback executes before tasks completes

a drawing representing the tasks flow

I have a celery workflow, as presented in the above image. I am facing a scenario where the parent_task_callback is executed before all the sub_parent_task callbacks are executed. None of these tasks has ignore_result set to false nor have I globally set it to false. What am I missing?

The code looks like this:

@app.task
def sub_parent_task():
    chord(child_tasks_list)(sub_parent_task_callback.si())

@app.task
def parent_task():
    chord(sub_parent_tasks_list)(parent_task_callback.si())

Since you're triggering a chord inside each sub_parent_task, Celery doesn't automatically wait for those inner callbacks to finish before executing the parent_task_callback. To ensure the final callback only runs after all sub-callbacks are done, you should restructure your workflow so that the outer chord wraps all the inner chords directly.

@app.task
def parent_task():
    # all_child_task_groups is a list of task lists
    header = [
        chord(child_tasks)(sub_parent_task_callback.s())
        for child_tasks in all_child_task_groups
    ]
    return chord(header)(parent_task_callback.s())
Back to Top