This website has been running fine without issues, Recently it went down.
After some investigation it looks like the combo of MYSQL and Apache bring the box to its knees. Apache can run find serving static web pages and MYSQL can run fine when the website isn't working.
As soon as the website is enabled with SQL running the CPU on the box remains at 100%.
Picture of the usage:
http://i.stack.imgur.com/GG2NC.png
I've checked the sql database for errors, tried tuning nearly every parameter in apache/sql's conf file for performance. The server is a redhat based box running the latest software packages.
Any help/suggestions are welcome.
Doing an strace on a high cpu apache process I see the following:
read(14, "", 8192) = 0
close(14) = 0
socket(PF_FILE, SOCK_STREAM, 0) = 14
fcntl64(14, F_SETFL, O_RDONLY) = 0
fcntl64(14, F_GETFL) = 0x2 (flags O_RDWR)
connect(14, {sa_family=AF_FILE, path="/var/lib/mysql/mysql.sock"...}, 110) = 0
setsockopt(14, SOL_SOCKET, SO_RCVTIMEO, "\2003\341\1\0\0\0\0", 8) = 0
setsockopt(14, SOL_SOCKET, SO_SNDTIMEO, "\2003\341\1\0\0\0\0", 8) = 0
setsockopt(14, SOL_IP, IP_TOS, [8], 4) = -1 EOPNOTSUPP (Operation not supported)
setsockopt(14, SOL_SOCKET, SO_KEEPALIVE, [1], 4) = 0
Here is what I see from a mysql process:
futex(0x86fc9a4, FUTEX_WAIT_PRIVATE, 39, NULL) = 0
futex(0x86fc734, FUTEX_WAIT_PRIVATE, 2, NULL) = 0
futex(0x86fc734, FUTEX_WAKE_PRIVATE, 1) = 0
gettimeofday({1301465020, 141613}, NULL) = 0
clock_gettime(CLOCK_REALTIME, {1301465020, 141699633}) = 0
futex(0x8707a64, FUTEX_WAIT_PRIVATE, 1, {4, 999913367}) = 0
futex(0x8707a40, FUTEX_WAIT_PRIVATE, 2, NULL) = 0
futex(0x8707a40, FUTEX_WAKE_PRIVATE, 1) = 0
exit_group(0) = ?
