Intermittent extrememly long response times when downloading documents
- by pap
I have a Java web application running om Tomcat 7 with an Apache httpd 2.2 fronting with mod_jk/AJP.
One part of the application is serving files (up to 4mb size). Now, normally this all runs very smooth with stable, low response-times. However, in rare instances (<0.1% of downloads), the downloadtime will go beyond 1 minute. After activating the ThreadStuckValve in Tomcat, I can see that the long responses seem to be stuck at
org.apache.tomcat.jni.Socket.sendbb(Native method)
i.e network I/O. At most, these long-running downloads take 5 minutes, which I strongly suspect is because of the default 300 second timout in Apache 2.2 (http://httpd.apache.org/docs/2.2/mod/core.html, "TimeOut directive").
To me, this looks like network problems. The Apache timeout (if that is what is kicking in at the 5 minute mark) indicates that ACK packets are not being transmitted correctly. My questions are what could be causing this? Closed browser at receiving end but socket not signaled as closed properly? Packet loss or some other network failure in transit? Where would I start troubleshooting this?
We're running Tomcat and Apache on Windows server 2008-R2 in a vmware virtualized server.
