nginx + Jetty - thousands of connections stuck in LAST_ACK
- by virulence
I have a FreeBSD machine with jails -- two in particular, one that runs nginx and another that runs a Java program that accepts requests via Jetty (embedded mode)
Jetty receives upwards of 500 requests/sec constantly and there has been an issue lately where I will constantly have over 60,000 connections in the LAST_ACK state between nginx and jetty.
Distribution of all connections (includes some other services, particularly php-fpm)
root@host:/root # netstat -an > conns.txt
root@host:/root # cat conns.txt | awk '{print $6}' | sort | uniq -c | sort -n
18 LISTEN
112 CLOSING
485 ESTABLISHED
650 FIN_WAIT_2
1425 FIN_WAIT_1
3301 TIME_WAIT
64215 LAST_ACK
Distribution of nginx - jetty connections
root@host:/root # cat conns.txt | grep '10.10.1.57' | awk '{print $6}' | sort | uniq -c | sort -n
1
3 CLOSE_WAIT
3 LISTEN
18 FIN_WAIT_2
125 ESTABLISHED
64193 LAST_ACK
I'd prefer every request to fully close the connection. Clients requests are about 10 minutes apart from each other so connections must be closed.
Some of the connections,
tcp4 0 0 10.10.1.50.46809 10.10.1.57.9050 LAST_ACK
tcp4 0 0 10.10.1.50.46805 10.10.1.57.9050 LAST_ACK
tcp4 0 0 10.10.1.50.46797 10.10.1.57.9050 LAST_ACK
tcp4 0 0 10.10.1.50.46794 10.10.1.57.9050 LAST_ACK
tcp4 0 0 10.10.1.50.46790 10.10.1.57.9050 LAST_ACK
tcp4 0 0 10.10.1.50.46789 10.10.1.57.9050 LAST_ACK
tcp4 0 0 10.10.1.50.46771 10.10.1.57.9050 LAST_ACK
etc..
On Jetty's end I've set maxIdleTime to 2000 -- before this all connections were in ESTABLISHED but they are now LAST_ACK
On Jetty's end I've set Connection: close (i.e response.setHeader(HttpHeaders.CONNECTION, HttpHeaderValues.CLOSE);)
Jetty never reports a lot of open connections -- always very few.
PF/IPFW is not currently being used
nginx - reset_timedout_connection is on
I cannot figure out how to get nginx or jetty to forcibly close the connection, is this simply something that needs to be fixed in Jetty so that it fully closes the socket after the request finishes?
Thanks a lot in advance
EDIT: forgot my nginx config for the proxy setup-
proxy_pass http://10.10.1.57:9050;
proxy_set_header HTTP_X_GEOIP $http_x_geoip;
proxy_set_header GEOIP_COUNTRY_CODE $geoip_country_code;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_set_header Connection "";
proxy_http_version 1.1;
EDIT2: Forcing Jetty to close the connection via request.getConnection().getEndPoint().close() does nothing -- it's obvious the connection IS being closed (as it's in LAST_ACK) but why isn't it getting past this? Is Nginx keeping the connection open to the backend for some reason?
