Frequent Kernel Panic on CentOS 6.5
Posted
by
Manuel Sopena Ballesteros
on Server Fault
See other posts from Server Fault
or by Manuel Sopena Ballesteros
Published on 2014-08-22T09:04:36Z
Indexed on
2014/08/23
10:23 UTC
Read the original article
Hit count: 257
I have a webserver with the configuration below:
VMWare ESXi environemt CPanel installed CentOS release 6.5 (Final) 4 CPUs 2G RAM 2x VM disks 100G each LVM system
My issue is I am getting kernel panic quite frequently. These is a list of some processes blocked I could see from the console:
- mysqld
- queueprocd
- httpd
- suphp
- vmtoolsd
- loop0
- auditd
this is my sar logs
Linux 2.6.32-431.3.1.el6.x86_64 (test01) 08/22/2014 _x86_64_ (4 CPU)
12:00:01 AM CPU %user %nice %system %iowait %steal %idle
12:10:01 AM all 26.86 0.01 0.98 0.57 0.00 71.57
12:20:01 AM all 1.78 0.02 1.03 0.08 0.00 97.09
12:30:01 AM all 26.34 0.02 0.85 0.05 0.00 72.74
12:40:01 AM all 27.12 0.01 1.11 1.22 0.00 70.54
12:50:01 AM all 1.59 0.02 0.94 0.13 0.00 97.32
01:00:01 AM all 26.10 0.01 0.77 0.04 0.00 73.07
01:10:01 AM all 27.51 0.01 1.16 0.14 0.00 71.18
01:20:01 AM all 1.80 0.07 1.06 0.08 0.00 96.99
01:30:01 AM all 26.19 0.01 0.78 0.05 0.00 72.96
01:40:01 AM all 26.62 0.02 0.87 0.05 0.00 72.45
01:50:02 AM all 1.35 0.01 0.87 0.02 0.00 97.75
02:00:01 AM all 26.11 0.02 0.69 0.02 0.00 73.17
02:10:01 AM all 26.73 0.02 0.89 0.14 0.00 72.21
02:20:01 AM all 1.45 0.01 0.92 0.04 0.00 97.58
02:30:01 AM all 26.59 0.01 1.06 0.03 0.00 72.31
02:40:01 AM all 26.27 0.01 0.72 0.05 0.00 72.95
02:50:01 AM all 0.86 0.01 0.50 0.09 0.00 98.53
03:00:01 AM all 25.61 0.02 0.39 0.03 0.00 73.96
03:10:01 AM all 26.30 0.08 0.66 0.14 0.00 72.82
03:20:01 AM all 0.81 0.01 0.51 0.04 0.00 98.63
03:30:02 AM all 26.15 0.02 0.53 0.07 0.00 73.24
03:40:01 AM all 26.06 0.01 0.47 0.04 0.00 73.42
03:50:01 AM all 0.96 0.02 0.51 0.03 0.00 98.48
Average: all 17.69 0.02 0.79 0.14 0.00 81.36
06:58:14 AM LINUX RESTART
07:00:01 AM CPU %user %nice %system %iowait %steal %idle
07:10:01 AM all 1.04 0.02 0.57 0.95 0.00 97.42
07:20:02 AM all 0.66 0.01 0.39 0.06 0.00 98.87
07:30:01 AM all 25.71 0.01 0.45 0.16 0.00 73.67
07:40:01 AM all 25.88 0.01 0.35 0.08 0.00 73.68
As you can see the server became unresponsive at 03.50 AM and I had to reset the VM at 06.58 AM to fix it.
sar -d
03:00:01 PM dev8-16 0.16 0.01 3.37 20.78 0.00 12.40 9.29 0.15
03:00:01 PM dev8-0 4.08 5.72 77.50 20.38 0.06 15.15 3.13 1.28
03:00:01 PM dev253-0 10.37 5.74 80.87 8.35 0.13 12.52 1.24 1.29
03:00:01 PM dev253-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
03:10:01 PM dev8-16 0.27 0.17 3.17 12.22 0.00 11.49 7.95 0.22
03:10:01 PM dev8-0 6.37 18.98 136.19 24.34 0.05 7.25 2.18 1.39
03:10:01 PM dev253-0 17.91 19.15 137.94 8.77 0.13 7.11 0.78 1.41
03:10:01 PM dev253-1 0.18 0.00 1.41 8.00 0.00 9.09 0.52 0.01
03:10:01 PM DEV tps rd_sec/s wr_sec/s avgrq-sz avgqu-sz await svctm %util
03:20:01 PM dev8-16 0.17 0.23 2.04 13.39 0.00 6.07 5.29 0.09
03:20:01 PM dev8-0 3.83 18.57 78.45 25.35 0.05 13.25 2.73 1.05
03:20:01 PM dev253-0 10.30 18.80 80.49 9.64 0.14 13.89 1.03 1.06
03:20:01 PM dev253-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
03:30:01 PM dev8-16 0.26 0.16 4.59 18.56 0.00 6.44 5.54 0.14
03:30:01 PM dev8-0 5.97 24.07 117.83 23.77 0.05 8.53 2.13 1.27
03:30:01 PM dev253-0 15.90 24.23 122.42 9.22 0.12 7.71 0.81 1.29
03:30:01 PM dev253-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
03:40:01 PM dev8-16 0.20 0.00 2.32 11.44 0.00 8.35 5.90 0.12
03:40:01 PM dev8-0 4.39 19.58 77.94 22.24 0.06 12.87 2.12 0.93
03:40:01 PM dev253-0 10.25 19.58 80.25 9.74 0.12 11.63 0.91 0.94
03:40:01 PM dev253-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
03:50:01 PM dev8-16 0.23 0.50 2.32 12.44 0.00 6.27 5.13 0.12
03:50:01 PM dev8-0 5.09 9.00 95.04 20.45 0.04 7.36 2.10 1.07
03:50:01 PM dev253-0 12.47 9.50 96.82 8.53 0.08 6.76 0.87 1.08
03:50:01 PM dev253-1 0.07 0.00 0.54 8.00 0.00 14.10 0.40 0.00
04:00:01 PM dev8-16 0.21 0.00 2.04 9.89 0.00 7.00 5.87 0.12
04:00:01 PM dev8-0 4.68 1.64 94.70 20.57 0.05 10.71 2.41 1.13
04:00:01 PM dev253-0 12.27 1.64 96.74 8.02 0.12 9.95 0.93 1.14
sar -q
01:00:01 AM 6 205 2.02 1.32 0.81
01:10:01 AM 3 187 0.08 0.72 0.86
01:20:01 AM 2 187 0.04 0.18 0.49
01:30:01 AM 4 205 2.04 1.34 0.82
01:40:01 AM 2 185 0.02 0.68 0.83
01:50:02 AM 1 185 0.08 0.15 0.45
02:00:01 AM 5 202 2.02 1.30 0.78
02:10:01 AM 4 185 0.11 0.72 0.84
02:20:01 AM 1 183 0.17 0.15 0.45
02:30:01 AM 5 206 2.03 1.32 0.79
02:40:01 AM 2 184 0.08 0.70 0.83
02:50:01 AM 1 183 0.00 0.10 0.43
03:00:01 AM 7 205 2.03 1.32 0.78
03:10:01 AM 2 194 0.34 0.73 0.83
03:20:01 AM 1 184 0.00 0.13 0.44
03:30:02 AM 4 201 2.04 1.32 0.78
03:40:01 AM 2 193 0.06 0.67 0.81
03:50:01 AM 1 183 0.06 0.12 0.43
Average: 3 192 0.68 0.70 0.69
06:58:14 AM LINUX RESTART
07:00:01 AM runq-sz plist-sz ldavg-1 ldavg-5 ldavg-15
07:10:01 AM 2 181 0.00 0.09 0.11
07:20:02 AM 1 179 0.00 0.00 0.04
07:30:01 AM 4 197 2.12 1.33 0.58
sar -r
Linux 2.6.32-431.3.1.el6.x86_64 (test01) 08/22/2014 _x86_64_ (4 CPU)
12:00:01 AM kbmemfree kbmemused %memused kbbuffers kbcached kbcommit %commit
12:10:01 AM 227484 1694468 88.16 117444 917004 635308 10.50
12:20:01 AM 219692 1702260 88.57 119556 920540 630940 10.43
12:30:01 AM 196248 1725704 89.79 121376 923592 695048 11.49
12:40:01 AM 127524 1794428 93.36 125004 1016196 633048 10.46
12:50:01 AM 127156 1794796 93.38 128212 1014536 624992 10.33
01:00:01 AM 110764 1811188 94.24 129964 1001608 700016 11.57
01:10:01 AM 160560 1761392 91.65 132260 973472 628640 10.39
01:20:01 AM 133076 1788876 93.08 134144 982608 655524 10.83
01:30:01 AM 121512 1800440 93.68 135548 985676 700500 11.58
01:40:01 AM 140640 1781312 92.68 137220 988576 628280 10.38
01:50:02 AM 139160 1782792 92.76 138688 990672 625224 10.33
02:00:01 AM 106112 1815840 94.48 139940 993976 700360 11.57
02:10:01 AM 155400 1766552 91.91 142112 971864 625656 10.34
02:20:01 AM 154056 1767896 91.98 143732 975556 621352 10.27
02:30:01 AM 110856 1811096 94.23 145032 978288 709360 11.72
02:40:01 AM 140200 1781752 92.71 146568 980656 624872 10.33
02:50:01 AM 137600 1784352 92.84 148940 984484 621948 10.28
03:00:01 AM 105032 1816920 94.54 150208 985736 706060 11.67
03:10:01 AM 168996 1752956 91.21 154708 941500 656312 10.85
03:20:01 AM 169408 1752544 91.19 156096 944100 621780 10.28
03:30:02 AM 132360 1789592 93.11 157724 951612 701296 11.59
03:40:01 AM 159012 1762940 91.73 158940 942560 656292 10.85
03:50:01 AM 163192 1758760 91.51 160312 944576 624544 10.32
Average: 148089 1773863 92.29 140162 969973 653363 10.80
06:58:14 AM LINUX RESTART
07:00:01 AM kbmemfree kbmemused %memused kbbuffers kbcached kbcommit %commit
07:10:01 AM 1016628 905324 47.10 85568 447556 600932 9.93
07:20:02 AM 1009996 911956 47.45 87616 451200 596156 9.85
07:30:01 AM 961128 960824 49.99 89164 464332 658912 10.89
07:40:01 AM 973376 948576 49.35 90880 473084 600176 9.92
dmesg does not show any relevant information.
I don't see any bottleneck in sar, any idea what can I check next?
thank you very much
© Server Fault or respective owner
