Fastcgi_buffers vs. proxy_buffers when serving Django API using Daphne
I'm running two containers in a docker network (either through docker-compose or AWS ECS):
- Daphne for ASGI to Django
- Nginx for forwarding requests to Daphne and serving static files
This is working fine. I can request API endpoints, use the Django admin site, load static files, etc. The problem is that some API responses are enormous (>10MiB), and while the endpoints returning these massive responses are very optimized and fast themselves, I'm getting timeouts due to buffering of the the response. I can tell by logs such as:
[warn] 28#28: *252 an upstream response is buffered to a temporary file /var/cache/nginx/proxy_temp/1/00/0000000001 while reading upstream, client: 1.2.3.4, server: _, request: "GET /my-app/123/very-big-response/ HTTP/1.1", upstream: "http://172.17.0.2:9000/my-app/123/very-big-response/", host: "app.acme.com", referrer: "https://app.acme.com/"
I have spent the past few hours reading about various nginx buffer settings, and while the docs explain fully what the options are and mean, I cannot find clear and reliable information on:
- Strategies for determining ballpark values for these parameters
- Which exact nginx directives to use
To reiterate, I have two containers: #1 (daphne/django), and #2 (nginx).
Container #1 (daphne/django) uses supervisord to run daphne. (Note before I continue: I'm fully aware of some other deviations from best practices here, like using supervisord in the first place, and comments on those are welcome, but I'm only asking about proper nginx buffer configs here.)
[supervisord]
user = root
nodaemon = true
[fcgi-program:startup-operations]
process_name=asgi%(process_num)d
# Socket used by nginx upstream
socket=tcp://my-app:9000
numprocs=1
command=/bin/bash -c "./.docker/services/my-app/files/startup-operations.sh"
autostart=true
autorestart=false
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
redirect_stderr=true
directory=/home/docker/my-app
startsecs=0
exitcodes=0
killasgroup=true
environment=PROCESS_NUM=%(process_num)d
My nginx configuration (with every irrelevant directive removed, including gzip, ssl, timeouts, etc.):
upstream fastcgi-socket {
server my-app:9000;
}
server {
listen 80;
listen [::]:80;
server_name _;
return 301 https://$host$request_uri;
}
server {
listen 443 ssl default_server;
server_name _;
# NOTE FOR STACK OVERFLOW:
# I am setting a "small" max body size here, and a larger one in specific locations.
# Does that make sense? There's only one location where large body sizes are expected.
client_max_body_size 1M;
underscores_in_headers on;
# NOTE FOR STACK OVERFLOW:
# Are these relevant here?
fastcgi_connect_timeout 300;
fastcgi_send_timeout 300;
fastcgi_read_timeout 300;
fastcgi_buffer_size 16k;
location @honeypot {
include honeypot.conf;
}
location /static {
alias /home/docker/my-app/static;
}
location / {
try_files $uri @proxy_to_app;
}
location @proxy_to_app {
client_max_body_size 20M;
# NOTE FOR STACK OVERFLOW:
# This is OPTION A - `proxy_buffers`
proxy_buffers 8 1280k;
# NOTE FOR STACK OVERFLOW:
# This is OPTION B - `fastcgi_buffers`
fastcgi_buffers 8 1280k;
# Probably not relevant to the question, but included just in case
proxy_read_timeout 30;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $http_x_forwarded_proto;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-Host $server_name;
proxy_redirect off;
proxy_pass http://fastcgi-socket;
}
My main confusion is about the difference between proxy_buffers and fastcgi_buffers. I realize there is a similar question, which includes this in the question:
From my understanding, the
proxy_buffersdirectives apply when you're proxying an http service such as apache, tomcat, or a third party website and would not be applicable to php-fpm with fastcgi.And conversely,
fastcgi_buffersis used when you are proxying a dynamic language engine (or whatever they're called) such as php-fpm, unicorn, or passenger.
The answer generally agrees with this, but also implies there are plenty of exceptions, which are not specified.
Besides this, all I can find is just different formulations of the minimal info the docs, as well as some example configs.
Main question: Which buffers directive to use? What exactly am I proxying here? My understanding of sockets, proxies, and the protocols involved, is generally poor. But as I understand it:
- We are saying that for any request matching
location @proxy_to_app, we will write those requests from Container #2 to a FastCGI socket that exists in Container #1, but: - However, because that FastCGI program is running in a different container (Container #1), I am actually connecting to it as a
servervia HTTP to its hostname (which in this docker network happens to bemy-app). Nginx does not see this as a fastcgi socket. - Instead, the response is proxied to
http://my-app:9000, and supervisord handles the writing of that data to a FastCGI socket (somewhere in Container #1) thanks to thesocket=tcp://my-app:9000line in my supervisor config teling it to do so. - Therefore, I should not use
fastcgi_buffersin my nginx config, but insteadproxy_buffers?
Secondary question #1: Understanding the flow of data
Assuming I answered my own main question correctly (I should be using proxy_buffers here, not fastcgi_buffers), I'm still left with holes in my understanding.
The script started by the fastcgi-program (see the supervisord config), startup-operations.sh, ends with this:
daphne -u "/run/daphne/daphne$PROCESS_NUM.sock" --fd 0 --proxy-headers main.asgi:application
There is so much here that's been stitched together from various guides and tutorials, often with extremely contradicting advice/information and differing approaches, that I must admit I don't have a full understanding of how everything comes together.
I don't understand how the socket=tcp://my-app:9000 statement allows requests proxied there to be intercepted by these /run/daphne/daphne$PROCESS_NUM.sock sockets.
How exactly are requests from the upstream in Container #1 (neinx) being written to tcp://my-app:9000 in Container #2 (daphne/django), and then somehow processed by daphne?
The choice of port 9000 was arbitrary. Is there some "magic" behind the fcgi-program supervisor program that allows this tcp://my-app:9000->daphne.sock communication to happen?
Secondary question 2: Correctly configuring the buffers Presuming that I now understand the flow of the data (and therefore which directives to use in my nginx config), and I know what the correct directives do (creates x buffers of y size for buffering responses), I still don't know the significance of the number of buffers vs. the size of each buffer.
What is the difference between having 1 buffer of 10 MiB, and having 2 buffers of 5 MiB each? Why would you split them up? If there is a benefit to splitting them up, why not have 1000x1k buffers instead of 100x10k buffers? How are they used, or rather, how would I even begin to determine how many buffers I should have and what size each buffer should be?
I understand that:
- If my largest expected response is 10 MiB, but that only happens occasionally, I probably don't want 1x10MiB buffer for every request just to completely eliminate the chance of writing to disk.
- I also don't want 2x5MiB buffers, because that's still 10 MiB per request. But what's the difference?
- I also don't want something like 8x4KiB buffers, because that's far too small for this largest request, basically guaranteeing that it will time out and render my service unavailable.
Sorry about the length and number sub-questions, I feel completely lost and can't find the explanations I need to grasp this stuff.
(Also, I would like to remove supervisor from the image for Container #1, but because I don't have a good enough understanding of this stuff, I'm currently relying on it purely for the cross-container communication aspect. Maybe I could have daphne write a unix socket to a shared volume to get around this?)