My Munin notifications set up for our (Debian) LAMP cluster have been notifying me continuously that our load on our production machine has been at dangerous levels. While the average load all year typically runs between 2 and 8, the load in the past month and only the past month -- has been skyrocketing to 10, 18, and occasionally even 50-60. The spikes last only 5-10 minutes at a time and occur about every 2-3 hours. The spikes do not effect performance only because I have a script that sends traffic off our server to a mirror CDN when the load goes above 10. I've looked for cron jobs that correlate with this timeframe but there is nothing I can see that would cause this. Site traffic is also normal (we receive about 200K visits per day). I'm also trying to think of anything I've changed around the time this problem began, and I really cannot think of anything.
This is probably not much to go on. Maybe there is a clue in the top print-out (below) that I'm not seeing.
How do I proceed to find the cause?
--
Typical top when the load is NOT spiking:
top - 11:13:09 up 472 days, 25 min, 1 user, load average: 6.08, 4.29, 3.80
Tasks: 105 total, 1 running, 104 sleeping, 0 stopped, 0 zombie
Cpu(s): 41.2%us, 5.8%sy, 0.0%ni, 49.5%id, 2.7%wa, 0.1%hi, 0.7%si, 0.0%st
Mem: 3369592k total, 2166980k used, 1202612k free, 559504k buffers
Swap: 2650684k total, 1892k used, 2648792k free, 1129116k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
32046 apache 15 0 36300 12m 9828 S 20 0.4 0:01.97 apache2
32679 apache 15 0 36568 13m 10m S 19 0.4 0:01.69 apache2
31441 apache 15 0 36616 13m 10m S 19 0.4 0:04.13 apache2
31477 apache 15 0 36596 13m 9.8m S 15 0.4 0:01.99 apache2
31993 apache 15 0 36876 16m 12m S 12 0.5 0:02.01 apache2
31782 apache 15 0 36836 14m 10m S 8 0.4 0:02.17 apache2
32198 apache 15 0 36536 13m 10m S 7 0.4 0:01.59 apache2
880 apache 15 0 36508 9708 6236 S 7 0.3 0:00.42 apache2
31945 apache 17 0 36876 16m 13m S 5 0.5 0:03.17 apache2
32197 apache 16 0 36636 10m 7504 S 5 0.3 0:02.70 apache2
32326 apache 15 0 37024 11m 7632 S 5 0.3 0:02.15 apache2
32565 apache 15 0 37280 13m 9.8m S 5 0.4 0:03.75 apache2
32676 apache 15 0 36896 16m 12m S 4 0.5 0:00.95 apache2
32678 apache 15 0 36536 12m 9692 S 4 0.4 0:02.27 apache2
974 apache 16 0 37064 9888 6016 D 4 0.3 0:00.13 apache2
32150 apache 16 0 36832 13m 10m S 3 0.4 0:01.74 apache2
31780 apache 16 0 36848 11m 7660 S 3 0.3 0:02.87 apache2
