We have an apache webserver in front of Tomcat hosted on EC2, instance type is extra large with 34GB memory.
Our application deals with lot of external webservices and we have a very lousy external webservice which takes almost 300 seconds to respond to requests during peak hours.
During peak hours the server chokes at just about 300 httpd processes. ps -ef | grep httpd | wc -l =300
I have googled and found numerous suggestions but nothing seems to work.. following are some configuration i have done which are directly taken from online resources.
I have increased the limits of max connection and max clients in both apache and tomcat. here are the configuration details:
//apache
<IfModule prefork.c>
StartServers 100
MinSpareServers 10
MaxSpareServers 10
ServerLimit 50000
MaxClients 50000
MaxRequestsPerChild 2000
</IfModule>
//tomcat
<Connector port="8080" protocol="org.apache.coyote.http11.Http11NioProtocol"
connectionTimeout="600000"
redirectPort="8443"
enableLookups="false" maxThreads="1500"
compressableMimeType="text/html,text/xml,text/plain,text/css,application/x-javascript,text/vnd.wap.wml,text/vnd.wap.wmlscript,application/xhtml+xml,application/xml-dtd,application/xslt+xml"
compression="on"/>
//Sysctl.conf
net.ipv4.tcp_tw_reuse=1
net.ipv4.tcp_tw_recycle=1
fs.file-max = 5049800
vm.min_free_kbytes = 204800
vm.page-cluster = 20
vm.swappiness = 90
net.ipv4.tcp_rfc1337=1
net.ipv4.tcp_max_orphans = 65536
net.ipv4.ip_local_port_range = 5000 65000
net.core.somaxconn = 1024
I have been trying numerous suggestions but in vain.. how to fix this? I'm sure m2xlarge server should serve more requests than 300, probably i might be going wrong with my configuration..
The server chokes only during peak hours and when there are 300 concurrent requests waiting for the [300 second delayed] webservice to respond.
Please help..
