We are using nginx-tornado-django to provide web services. That is, no web page frontend. The nginx server serves as a load-balancer. The server has 8 cores, so we launched 8 tornado-django processes on every server. Memcached is also deployed to gain better performance. The requests per day is about 1 million per server. We use MySQL as backend DB.

The code is tested and correct. Our profiling shows that normally every request are processed within 100ms. The problem is, we find that about 10 percent of the requests suffers from time-out issue. Many requests didn't even reach tornado. I really don't have much experience on tuning of nginx/tornado/MySQL. Right now I don't have a clue on what is going wrong. Any advise is appreiciated.