Why is my concurrency capacity so low for my web app on a LAMP EC2 instance?
- by AMF
I come from a web developer background and have been humming along building my PHP app, using the CakePHP framework. The problem arose when I began the ab (Apache Bench) testing on the Amazon EC2 instance in which the app resides. I'm getting pretty horrendous average page load times, even though I'm running a c1.medium instance (2 cores, 2GB RAM), and I think I'm doing everything right.
I would run:
ab -n 200 -c 20 http://localhost/heavy-but-view-cached-page.php
Here are the results:
Concurrency Level: 20
Time taken for tests: 48.197 seconds
Complete requests: 200
Failed requests: 0
Write errors: 0
Total transferred: 392111200 bytes
HTML transferred: 392047600 bytes
Requests per second: 4.15 [#/sec] (mean)
Time per request: 4819.723 [ms] (mean)
Time per request: 240.986 [ms] (mean, across all concurrent requests)
Transfer rate: 7944.88 [Kbytes/sec] received
While the ab test is running, I run VMStat, which shows that Swap stays at 0, CPU is constantly at 80-100% (although I'm not sure I can trust this on a VM), RAM utilization ramps up to about 1.6G (leaving 400M free). Load goes up to about 8 and site slows to a crawl.
Here's what I think I'm doing right on the code side:
In Chrome browser uncached pages typically load in 800-1000ms, and
cached pages load in 300-500ms. Not stunning, but not terrible either.
Thanks to view caching, there might be at most one DB query per page-load to write session data. So we can rule out a DB bottleneck.
I have APC on.
I am using Memcached to serve the view cache and other site caches.
xhprof code profiler shows that cached pages take up 10MB-40MB in
memory and 100ms - 1000ms in wall time.
Pages that would be the worst offenders would look something like this in xhprof:
Total Incl. Wall Time (microsec): 330,143 microsecs
Total Incl. CPU (microsecs): 320,019 microsecs
Total Incl. MemUse (bytes): 36,786,192 bytes
Total Incl. PeakMemUse (bytes): 46,667,008 bytes
Number of Function Calls: 5,195
My Apache config:
KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 3
<IfModule mpm_prefork_module>
StartServers 5
MinSpareServers 5
MaxSpareServers 10
MaxClients 120
MaxRequestsPerChild 1000
</IfModule>
Is there something wrong with the server? Some gotcha with the EC2? Or is it my code? Some obvious setting I should look into? Too many DNS lookups? What am I missing? I really want to get to 1,000 concurrency capacity, but at this rate, it ain't gonna happen.
