Expensive django calculation needs to run on model change

I am working on a django application that has a model with a field that needs to be calculated when the model changes. The calculation itself is relatively expensive (1+ seconds) especially if/when this field is recalculated for a large queryset. It is important that this field is recalculated immediately following the model change because it determines the workflow which the user needs to follow and the field value is propagated to the UI for the user to see. At a minimum the calculation needs to be guaranteed to run after update or some way to indicate the calculation is stale.

The current implementation which was not an issue because the calculation was previously less complicated/expensive (before requested changes by the client) is as follows:

  • user makes edit in UI and model is updated
  • API receives changes and applies changes to model
  • model .save() is overridden so the field is calculated and set

With this implementation we also disabled the queryset manager on the model to prevent the use of .update() to ensure that field is always recalculated on .save().

I am curious on how others would implement something like this to be as efficient as possible and not relying on the blocking save to finish.

This question has no one-fits-all answer.

Option A: The function/ calculation could potentially fail.
In this case you would want to notify the user whether his model instance was saved successfully or not. In this case you would want the user to wait for the calculation to then be happy, or see the error page.

Option B: The function/ calculation can (almost) not fail.
As it is the case for all expensive functions you could 'outsource' the model saving, including the expensive calculation, to another service (most likely celery). When you deploy a django project the response serving is taking care of one worker pool (e.g. uwsgi). The outsourced functionality is taking care of another worker pool (implemented through celery). This allows you to handle functions and tasks asynchronously while the response is sent fast to the user. The only downside of this approach is that the success of outsourced functionality is difficult to feed back to the user and needs to be looked at.

Back to Top