Task Queue with cached libraries in Django

I am creating a WEB interface for various Python scripts through Django.

Example in calculation.py I would have :

import datetime
def add_time(a, b):
    return = a + b + int(datetime.datetime.now())

Usages :

  • A user could say "I want to run add from calculation.py with arguments [1, 3]" and that returns him the result when it's ready.
  • A user could say "I want this to run add from calculation.py with arguments [1, 3] every 10 minutes and check if result is greater than X" and that would action something if it's true.

Most of my script functions are quick but they need to import libraries that takes a lot of time to load.

I am currently doing this directly with my Django service ; it's simple and load the libraries once which allows most of the next calls to be very fast but sometimes a heavy call is made and that is slowing down all my Django application and I seem limited if I want to have CRON scheduling for some scripts. Therefore I am looking at another solution.

I started to look at :

  • Celery (but it seems not supported anymore on Windows)
  • Huey and Dramatiq
  • Django-Q2 (easy to use with Django)

From my understanding nothing will cache the already imported libraries. (I am fine having a "start up bulk import" if needed). Would anyone have a guess on where I should look or what above solution I can tweak ?

Requirement : I need this to run on Windows as some of my libraries are Windows dependant.

Imagine you have a bunch of heavy tasks in your Django app, like complex calculations or data processing, which are slowing down your web interface. You want a way to handle these tasks separately so they don't bog down your app.

Here's a more user-friendly breakdown:

Celery: It's like a superstar worker who helps with tasks, but it doesn't play well with Windows, so it might not be the best fit here.

Huey and Dramatiq: They're like reliable assistants who can take on tasks separately from your Django app. They work well with Windows and won't cause compatibility issues.

Django-Q2: This one is like a specialized assistant tailor-made for Django tasks. It's easy to use and integrates smoothly with your Django setup.

Now, when it comes to handling libraries:

None of these assistants will automatically remember the libraries you've used before. But you can make them smarter by telling them to load the necessary libraries at the beginning of each task. This way, they won't waste time loading libraries repeatedly. In simple terms, you can think of these tools as helpers who take on heavy tasks for your web app, keeping it fast and responsive. And by loading libraries strategically, you can make them even more efficient. Overall, I'd recommend trying out Huey, Dramatiq, or Django-Q2 to see which one suits your needs best.

In the end I used Django-q2 and used the app init to load libraries for qcluster calls

Back to Top