For about 5 days now, and seemingly out of the blue, my linux server has started locking up from time to time.
The pattern is always the same as far as I can tell from top and iotop commands around the time it starts happening:
One or more httpd processes (usually one) hang and start using up 100% of CPU power, the %wa goes close to 100% and in the iotop I see several httpd processes with 99.99% in the IO column.
I'm also running an SVN server on this machine through apache and the one way that I've been consistently able to reproduce this is to do an SVN commit of new files or an SVN update from the repository on this server (I am the only one using this SVN repository). This will always reproduce this scenario successfully, but until very recently I had no problems at all checking in/out of SVN.
But sometimes it just happens for no detectable reason at all it seems.
So it seems like there is some issue with my Apache that leads it to have processes use up a lot of read/write upon certain triggers.
I was wondering if anyone could help me uncover that issue.
EDIT: OK now it's happening again:
This is top:
[root@server ~]# top
top - 10:56:54 up 2:59, 5 users, load average: 171.46, 70.35, 27.01
Tasks: 328 total, 2 running, 326 sleeping, 0 stopped, 0 zombie
Cpu(s): 1.9%us, 2.0%sy, 0.0%ni, 0.0%id, 96.1%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 2021144k total, 1968192k used, 52952k free, 2500k buffers
Swap: 4194288k total, 2938584k used, 1255704k free, 39008k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
10390 apache 20 0 2774m 936m 6200 D 2.0 47.4 1:52.27 httpd
2149 root 20 0 927m 13m 1040 S 0.7 0.7 1:50.46 namecoind
11 root 20 0 0 0 0 R 0.3 0.0 0:30.10 events/0
23 root 20 0 0 0 0 S 0.3 0.0 0:17.88 kblockd/1
2049 root 20 0 382m 4932 2880 D 0.3 0.2 0:03.67 httpd
2144 root 20 0 1702m 69m 1164 S 0.3 3.5 5:19.68 bitcoind
6325 root 20 0 15164 1100 656 R 0.3 0.1 0:11.09 top
10311 apache 20 0 387m 9496 7320 D 0.3 0.5 0:01.89 httpd
10313 apache 20 0 391m 10m 7364 D 0.3 0.5 0:02.40 httpd
10466 apache 20 0 399m 12m 7392 D 0.3 0.7 0:02.41 httpd
10599 apache 20 0 391m 9324 7340 D 0.3 0.5 0:00.15 httpd
10628 apache 20 0 384m 7620 4052 D 0.3 0.4 0:00.01 httpd
10633 apache 20 0 384m 7048 3504 D 0.3 0.3 0:00.01 httpd
10634 apache 20 0 384m 8012 4048 D 0.3 0.4 0:00.02 httpd
10638 apache 20 0 400m 22m 9.8m D 0.3 1.1 0:01.93 httpd
10640 apache 20 0 385m 8288 4028 D 0.3 0.4 0:00.03 httpd
10641 apache 20 0 401m 21m 6376 D 0.3 1.1 0:01.45 httpd
10759 apache 20 0 385m 8816 3480 D 0.3 0.4 0:01.45 httpd
10773 apache 20 0 384m 8044 3464 D 0.3 0.4 0:00.02 httpd
This is an iotop snapshot:
Total DISK READ: 5.93 M/s | Total DISK WRITE: 0.00 B/s
TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
10732 be/4 apache 3.76 K/s 0.00 B/s 0.00 % 58.48 % httpd
876 be/3 root 0.00 B/s 52.68 K/s 0.00 % 52.98 % [jbd2/dm-1-8]
10906 be/4 root 124.17 K/s 0.00 B/s 0.00 % 23.03 % sh -c [ -x /usr/local/psa/admin/sbin/backupmng ] && /usr/local/psa/admin/sbin/backupmng >/dev/null 2>&1
2156 be/4 root 206.94 K/s 0.00 B/s 0.00 % 21.15 % bitcoind
10904 be/4 mysql 0.00 B/s 0.00 B/s 0.00 % 18.94 % mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --log-error=/var/log/mysqld.log --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/lib/mysql/mysql.sock
10773 be/4 apache 7.53 K/s 0.00 B/s 0.00 % 14.77 % httpd
10641 be/4 apache 15.05 K/s 0.00 B/s 0.00 % 11.57 % httpd
10399 be/4 apache 1057.29 K/s 0.00 B/s 43.16 % 10.56 % httpd
10682 be/4 sw-cp-se 158.03 K/s 0.00 B/s 0.00 % 7.45 % sw-engine-cgi -c /usr/local/psa/admin/conf/php.ini -d auto_prepend_file=auth.php3 -u psaadm
10774 be/4 apache 3.76 K/s 0.00 B/s 0.00 % 6.53 % httpd
10624 be/4 apache 0.00 B/s 0.00 B/s 0.00 % 5.53 % httpd
10356 be/4 apache 899.26 K/s 0.00 B/s 35.52 % 4.01 % httpd
10795 be/4 apache 0.00 B/s 0.00 B/s 0.00 % 3.93 % httpd
10804 be/4 apache 7.53 K/s 0.00 B/s 0.00 % 3.08 % httpd
4379 be/4 root 2.89 M/s 0.00 B/s 99.99 % 0.00 % namecoind
10619 be/4 apache 462.80 K/s 0.00 B/s 7.80 % 0.00 % httpd
10636 be/4 apache 3.76 K/s 0.00 B/s 0.00 % 0.00 % httpd
10716 be/4 mysql 105.35 K/s 0.00 B/s 5.92 % 0.00 % mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --log-error=/var/log/mysqld.log --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/lib/mysql/mysql.sock
1988 be/4 root 18.81 K/s 0.00 B/s 0.00 % 0.00 % spamd_full.sock
I also ran lsof -p for pid 10390 which was way up top under the top command and this is the bottom line where I can sort of see what request this was and it says CLOSE_WAIT:
httpd 10390 apache 34u IPv6 315879 0t0 TCP default-domain.com:https->crawl-66-249-65-91.googlebot.com:42907 (CLOSE_WAIT)
I'm still not sure what exactly is causing this all to happen though?
I killed that service but %wa and load average remain high, I also stopped mysqld and other services. It really only goes down once I stop httpd altogether, and even then I can't start it without finding remaining hanging httpd processes via "netstat -tulpn", killing those or doing "killall -9 httpd" and after waiting a while for it to cycle through all those then doing /etc/init.d/httpd start
