Celery chord, execute code after callback

I am working on a Django application that uses celery tasks for complex and time intensive calculations. I have a task in which subtasks are used to process a large number of combinations of values and determine the best combination based on predefined parameters. The subtasks are created like this:

combinations = itertools.product(*bm)

# setup for multicore processing using celery tasks
static_args = (nt, mp, f, levels)
chunk_size = 10000
tasks = _chunked(combinations, chunk_size)

chunk_tasks = group(evaluate_combination_chunk.s(list(chunk), static_args) for chunk in tasks)

final_task = compare_chunk_results.s()

best_result = chord(chunk_tasks)(final_task)
# do some stuff with this result
return best_result

the subtask methods like this:

def evaluate_combination_chunk(chunk, static_args):
# evaluate a chunk of combinations
best_result = None
highest_power = -1
for comb in chunk:
    result = evaluate_combination(comb, *static_args)
    if result[0] and result[1] > highest_power:
        best_result = result
        highest_power = result[1]
return best_result

and

def compare_chunk_results(results):
best_result = None
highest_power = -1
for result in results:
    if result:
        if result[1] > highest_power:
            best_result = result
            highest_power = result[1]
return best_result

both marked with @shared_task(). so i am finding the best result in each chunk (that works just fine based on what i see in the console) and then compare those to find the best result of all chunks. My Issue is that i want to take that best_result, do something with it and then return that to the parent task. but i cant seem to run code after best_result = chord... because code execution doesnt wait for all tasks and the callback of the chord to be finished, I cant seem to find a solution to that, so any help would be very much appreciated. Thanks!

Вернуться на верх