kernel panic after LVM setup
- by Manuel Sopena Ballesteros
I broke my webserver...
My setup is:
VMWare ESXi environemt
CPanel installed
CentOS release 6.5 (Final)
4 CPUs
2G RAM
2x VM disks 100G each
LVM system
This was my previous storage settings (the server was working fine at this time):
# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_test01-lv_root 95G 1.4G 88G 2% /
tmpfs 939M 0 939M 0% /dev/shm
/dev/sdb1 99G 188M 94G 1% /tmp
/dev/sda1 485M 54M 407M 12% /boot
My web developer asked me to merge /tmp and / disks so this is what I did:
Delete /dev/sdb1 partition using fdisk
Create a new partition as LVM on /dev/sdb1 using fdisk
Create a new physical volume -- pvcreate /dev/sdb1
Extend volume group -- vgextend /dev/sdb1 vg_test01
Extend logical volume -- lvextend -l +100%FREE /dev/vg_test01/lv_root
Resize filesystem -- resize2fs /dev/vg_test01/lv_root
This is the new configuration:
# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_test01-lv_root 213G 105G 97G 52% /
tmpfs 939M 0 939M 0% /dev/shm
/dev/sda1 485M 54M 407M 12% /boot
/usr/tmpDSK 4.0G 145M 3.6G 4% /tmp
Since I have the new settings my web server is throwing kernel panics quite often (around every 2 days).
The message says:
INFO: task <taskName>:<pid> blocked for more than 120 seconds.
The list of process affected that I can see from the console are:
mysqld
queueprocd
httpd
suphp
vmtoolsd
loop0
auditd
The only way I can fix this is reseting (cold reboot) the VM. I don't think it is a hardware issue as sar is not showing any bottleneck:
Linux 2.6.32-431.3.1.el6.x86_64 (test01) 08/22/2014 _x86_64_ (4 CPU)
12:00:01 AM CPU %user %nice %system %iowait %steal %idle
12:10:01 AM all 26.86 0.01 0.98 0.57 0.00 71.57
12:20:01 AM all 1.78 0.02 1.03 0.08 0.00 97.09
12:30:01 AM all 26.34 0.02 0.85 0.05 0.00 72.74
12:40:01 AM all 27.12 0.01 1.11 1.22 0.00 70.54
12:50:01 AM all 1.59 0.02 0.94 0.13 0.00 97.32
01:00:01 AM all 26.10 0.01 0.77 0.04 0.00 73.07
01:10:01 AM all 27.51 0.01 1.16 0.14 0.00 71.18
01:20:01 AM all 1.80 0.07 1.06 0.08 0.00 96.99
01:30:01 AM all 26.19 0.01 0.78 0.05 0.00 72.96
01:40:01 AM all 26.62 0.02 0.87 0.05 0.00 72.45
01:50:02 AM all 1.35 0.01 0.87 0.02 0.00 97.75
02:00:01 AM all 26.11 0.02 0.69 0.02 0.00 73.17
02:10:01 AM all 26.73 0.02 0.89 0.14 0.00 72.21
02:20:01 AM all 1.45 0.01 0.92 0.04 0.00 97.58
02:30:01 AM all 26.59 0.01 1.06 0.03 0.00 72.31
02:40:01 AM all 26.27 0.01 0.72 0.05 0.00 72.95
02:50:01 AM all 0.86 0.01 0.50 0.09 0.00 98.53
03:00:01 AM all 25.61 0.02 0.39 0.03 0.00 73.96
03:10:01 AM all 26.30 0.08 0.66 0.14 0.00 72.82
03:20:01 AM all 0.81 0.01 0.51 0.04 0.00 98.63
03:30:02 AM all 26.15 0.02 0.53 0.07 0.00 73.24
03:40:01 AM all 26.06 0.01 0.47 0.04 0.00 73.42
03:50:01 AM all 0.96 0.02 0.51 0.03 0.00 98.48
Average: all 17.69 0.02 0.79 0.14 0.00 81.36
06:58:14 AM LINUX RESTART
07:00:01 AM CPU %user %nice %system %iowait %steal %idle
07:10:01 AM all 1.04 0.02 0.57 0.95 0.00 97.42
07:20:02 AM all 0.66 0.01 0.39 0.06 0.00 98.87
07:30:01 AM all 25.71 0.01 0.45 0.16 0.00 73.67
07:40:01 AM all 25.88 0.01 0.35 0.08 0.00 73.68
07:50:01 AM all 1.13 0.02 0.55 0.11 0.00 98.19
As you can see the server became unresponsive at 03.50 AM and I had to reset the VM at 06.58 AM to bring the website up again.
I would appreciate any help/assistance to fix this issue.
thank you very much
