My Drupal 6 site has been running smoothly for years but recently has experienced intermittent periods of extreme slowness (10-60 sec page loads). Several hours of slowness followed by hours of normal (4-6 sec) page loads. The page always loads with no error, just sometimes takes forever.

My setup:

• Windows Server 2003
• Apache/2.2.15 (Win32) Jrun/4.0
• PHP 5 MySql 5.1
• Drupal 6
• Cold fusion 9
• Vmware virtual environment
• DMZ behind a corporate firewall

• Traffic: 1-3 hits/sec avg

  Troubleshooting

• No applicable errors in apache error log
• No errors in drupal event log
• Drupal devel module shows 242 queries in 366.23 milliseconds,page execution time 2069.62 ms. (So it looks like queries and php scripts are not the problem)
• NO unusually high CPU, memory, or disk IO
• Cold fusion apps, and other static pages outside of drupal also load slow
• webpagetest.org test shows very high time-to-first-byte

The problem seems to be with Apache responding to requests, but previously I've only seen this behavior under 100% cpu load. Judging solely by resource monitoring, it looks as though very little is going on.

Here is the kicker - roughly half of the site's access comes from our LAN, but if I disable the firewall rule and block access from outside of our network, internal (LAN) access (1000+ devices) is speedy. But as soon as outside access is restored the site is crippled.

Apache config? Crawlers/bots? Attackers? I'm at the end of my rope, where should I be looking to determine where the problem lies?