Will requests to my site lag and work slowly in django while waiting for celery results?
I use django to create pdf to docx converter using pdf2docx library, and I need to wait a celery task to done and get result from it. Will my site lag and work slowly if a lot of users use it, and how can I do it better? here my views and celery code
views.py '''
from django.shortcuts import render
from django.http import JsonResponse, HttpResponse
from django.http.request import HttpRequest
import tempfile
import os
from .forms import FileUploadForm
from django.views.decorators.csrf import csrf_protect
from . import tasks
from celery.result import AsyncResult
@csrf_protect
def main_page(request: HttpRequest):
if request.method == "GET":
# get
form = FileUploadForm(request.POST, request.FILES)
context = {
"form": form
}
return render(request, 'main/main_page.html', context)
if request.method == 'POST' and request.FILES.get('file'):
form = FileUploadForm(request.POST, request.FILES)
if form.is_valid():
# get file
file = request.FILES['file']
size_limit = 2 * 1024 * 1024
# save pdf
with tempfile.NamedTemporaryFile(delete=False, suffix=".pdf") as temp_file:
if file.size > size_limit:
for chunk in file.chunks():
temp_file.write(chunk)
else:
temp_file.write(file.read())
temp_pdf_path = temp_file.name
# start convertor task
task: AsyncResult = tasks.convert_pdf_to_docx.delay(temp_pdf_path)
# get docx path
temp_docx_path = task.wait(timeout=None, interval=0.5)
converted_file_name = str(file).replace(".pdf", "")
# read docx file and set it in response
with open(temp_docx_path, 'rb') as docx_file:
file_data = docx_file.read()
response = HttpResponse(file_data, content_type='application/vnd.openxmlformats-officedocument.wordprocessingml.document')
response['Content-Disposition'] = f'attachment; filename={converted_file_name}.docx'
response['Content-Length'] = len(file_data) # calculate length of content
# Clean up temp files
os.remove(temp_pdf_path)
os.remove(temp_docx_path)
return response
else:
context = {
"form": form,
"submit_error": True,
}
return render(request, 'main/main_page.html', context)
'''
tasks.py '''
from celery import shared_task
from pdf2docx import Converter
@shared_task()
def convert_pdf_to_docx(temp_pdf_path):
# Convert to DOCX
temp_docx_path = temp_pdf_path.replace(".pdf", ".docx")
cv = Converter(temp_pdf_path)
cv.convert(temp_docx_path)
cv.close()
return temp_docx_path
'''
to run celery I use: '''
celery -A pdfconverter worker --loglevel=info -P gevent
'''
Yes, your site will lag and work slowly. Using task.wait() blocks the Django worker thread, preventing it from handling other requests. When you call task.wait(), the Django process stops and waits for Celery to finish, which defeats the purpose of using Celery for background tasks. If you have limited Django workers (typical with gunicorn/uwsgi deployments), concurrent users will queue up waiting for available workers, creating a bottleneck where each file conversion monopolizes a worker thread and significantly reduces your application's ability to handle multiple simultaneous requests.
To solve this issue, you can split the process into two steps: upload and download.
Step 1 - Upload and startthe task:
@csrf_protect
def upload_file(request):
if request.method == 'POST' and request.FILES.get('file'):
form = FileUploadForm(request.POST, request.FILES)
if form.is_valid():
file = request.FILES['file']
# Save file temporarily
with tempfile.NamedTemporaryFile(delete=False, suffix=".pdf") as temp_file:
for chunk in file.chunks():
temp_file.write(chunk)
temp_pdf_path = temp_file.name
# Start task
task = tasks.convert_pdf_to_docx.delay(temp_pdf_path)
# Return task ID to frontend
return JsonResponse({
'task_id': task.id,
'status': 'processing'
})
Step 2 - Check status and download:
def check_task_status(request, task_id):
task = AsyncResult(task_id)
if task.state == 'PENDING':
return JsonResponse({'status': 'processing'})
elif task.state == 'SUCCESS':
docx_path = task.result
with open(docx_path, 'rb') as docx_file:
response = HttpResponse(
docx_file.read(),
content_type='application/vnd.openxmlformats-officedocument.wordprocessingml.document'
)
response['Content-Disposition'] = 'attachment; filename=converted.docx'
# Clean up
os.remove(docx_path)
return response
else:
return JsonResponse({'status': 'error', 'message': str(task.info)})
Frontend JavaScript:
// After uploading
function checkStatus(taskId) {
fetch(`/check-status/${taskId}/`)
.then(response => response.json())
.then(data => {
if (data.status === 'processing') {
setTimeout(() => checkStatus(taskId), 2000);
} else if (data.status === 'success') {
window.location.href = `/download/${taskId}/`;
}
});
}
This approach keeps your Django workers free to handle other requests while Celery processes the file conversion in the background. The frontend polls the server every 2 seconds to check if the conversion is complete, providing a much better user experience without blocking your application.
For more advanced real-time updates, consider using Django Channels with WebSockets to push conversion progress directly to the user's browser instead of polling. This eliminates the need for repeated HTTP requests and provides instant feedback when the conversion completes.
In this case, ideally, Celery would be used to allow your HTTP server to respond quickly to the HTTP request by simply "queueing" the work involved with converting the pdf.
This configuration would allow many requests to queue work to be done, without blocking further HTTP requests. Given the processing is on a different server or not impeding system resources by limiting workers or memory allocated to the workers.
However, by calling wait you're saying, wait until the work is done then continue, effectively nullifying any benefit from using Celery. And therefore blocking up your HTTP servers ability to respond to further requests.
There is no benefit to using Celery in this current configuration, in-fact it's simply creating more overhead.
I'm not sure of the end result you're specifically looking for but I'd imagine it's to start the job and once the the conversion is complete have the front end respond by presenting the user with a download.
To not bottleneck your HTTP server you'll need to remove the wait call (which says I'll sit here and do nothing until you're all done) and instead offer a different method of checking for completion of the conversion.
This is a much larger endeavor. You'll need a way to ID a conversion, I'm short on Celery experience but I'd imagine a Task is given an ID and you can query if that ID is done or not. Otherwise you'll need to implement a way to uniquely identify a file conversion and if it is complete or not.
Once you've done that you'll need the front end to check if it is done, and if so, trigger a download.
There are many ways to do so. I'd imagine you'd be after polling or web sockets. However those topics are best googled if you're approaching the problem for the first time.
The current configuration you have however removes any purpose Celery may have served. The idea you're after is telling Celery you have some work you'd like it to do, then continue to process/return the HTTP request. And then allowing Celery to get to it at some point in the future.