nginx server over https using up all available file handles (upd: infinite loop?)
- by mmr
Hi all,
So I have an nginx server that's working over https with Sinatra. When I try to download a jnlp file in a configuration that works fine over Mongrel and http (no s), the nginx server fails to serve the file with a 504 error. Subsequent checking of the logs states that this error is due to overflowing the available number of file handles, ie, "24: too many open files". Running
sudo lsof -p <nginx worker pid>
gets me a huge list of files, all looking like:
nginx 1771 nobody 11u IPv4 10867997 0t0 TCP localhost:44704->localhost:https (ESTABLISHED)
nginx 1771 nobody 12u IPv4 10868113 0t0 TCP localhost:https->localhost:44704 (ESTABLISHED)
nginx 1771 nobody 13u IPv4 10868114 0t0 TCP localhost:44705->localhost:https (ESTABLISHED)
nginx 1771 nobody 14u IPv4 10868191 0t0 TCP localhost:https->localhost:44705 (ESTABLISHED)
nginx 1771 nobody 15u IPv4 10868192 0t0 TCP localhost:44706->localhost:https (ESTABLISHED)
nginx 1771 nobody 16u IPv4 10868255 0t0 TCP localhost:https->localhost:44706 (ESTABLISHED)
nginx 1771 nobody 17u IPv4 10868256 0t0 TCP localhost:44707->localhost:https (ESTABLISHED)
nginx 1771 nobody 18u IPv4 10868330 0t0 TCP localhost:https->localhost:44707 (ESTABLISHED)
nginx 1771 nobody 19u IPv4 10868331 0t0 TCP localhost:44708->localhost:https (ESTABLISHED)
nginx 1771 nobody 20u IPv4 10868434 0t0 TCP localhost:https->localhost:44708 (ESTABLISHED)
Increasing the number of files that can be opened is no help, because then nginx just blows right past that limit. And no wonder, it looks like it's in some kind of loop to pull all available files.
Any idea what's going on, and how to fix it?
EDIT: nginx 0.7.63, ubuntu linux, sinatra 1.0
EDIT 2: Here's the offending code. It's sinatra serving jnlp, which I finally figured out:
get '/uploader' do
#read in the launch.jnlp file
theJNLP = ""
File.open("/launch.jnlp", "r+") do |file|
while theTemp = file.gets
theJNLP = theJNLP + theTemp
end
end
content_type :jnlp
theJNLP
end
If I serve this with Sinatra via Mongrel and http, everything works fine. If I serve this with Sinatra and nginx via https, I get the above error. All other parts of the website appear to be equivalent.
EDIT: I have since upgraded to passenger 2.2.14, ruby 1.9.1, nginx 0.8.40, openssl 1.0.0a, and no change.
EDIT: The culprit appears to be infinite redirects due to using SSL. I don't know how to fix this, other than hosting the jnlp file in the root directory of the server (which I'd rather not do, since it limits me to one jnlp-based app at a time).
The relevant lines from nginx.conf:
# HTTPS server
#
server {
listen 443;
server_name MyServer.org
root /My/Root/Dir;
passenger_enabled on;
expires 1d;
proxy_set_header X-FORWARDED_PROTO https;
proxy_set_header X_FORWARDED_PROTO https;#the almighty google is not clear on which to use
location /upload {
proxy_pass https://127.0.0.1:443;
}
}
The funny thing about this is, first, I was putting the jnlp into a directory called 'uploader', not 'upload', but that still appeared to trigger the problem, since that proxy_pass directive appeared in the logs. Second, again, moving the jnlp into root avoided the problem, because there wasn't any of this proxying due to ssl.
So, how can I avoid the infinite proxy_pass loop in nginx?
