Did I lost my RAID again?
- by BarsMonster
Hi!
A little history: 2 years ago I was really excited to find out that mdadm is so powerful so it even can reshape arrays so you can start with a smaller array and the grow it as you need. I've bought 3x1Tb drives and made RAID-5. It was fine for a year.
Then I bought 2x more, and tried to reshape to RAID-6 out of 5 drives, and due to some mess with superblock versions, lost all content. Had to rebuild it from scratch, but 2Tb of data were gone.
Yesterday I bought 2 more drives, and this time I had everything: properly built array, UPS. I've disabled write intent map, added 2 new drives as a spare and run a command to grow array to 7-disk.
It started working, but speed was ridiculously slow, ~100kb/sec. AFter processing first 37Mb at such an amasing speed, one of old HDDs fails. I properly shutdown PC and disconnected failed drive. After bootup it appeared it recreated intent map as it was still in mdadm config, so I removed it from config and rebooted again.
Now all I see is that all mdadm processes deadlocks, and don't do anything.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1937 root 20 0 12992 608 444 D 0 0.1 0:00.00 mdadm
2283 root 20 0 12992 852 704 D 0 0.1 0:00.01 mdadm
2287 root 20 0 0 0 0 D 0 0.0 0:00.01 md0_reshape
2288 root 18 -2 12992 820 676 D 0 0.1 0:00.01 mdadm
And all I see in mdstat is:
$ cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid6 sdb1[1] sdg1[4] sdf1[7] sde1[6] sdd1[0] sdc1[5]
2929683456 blocks super 1.2 level 6, 1024k chunk, algorithm 2 [7/6] [UU_UUUU]
[>....................] reshape = 0.0% (37888/976561152) finish=567604147.2min speed=0K/sec
I've already tried mdadm 2.6.7, 3.1.4 and 3.2 - nothing helps. Did I lost my data again? Any suggestions how can I make it work? OS is Ubuntu Server 10.04.2...
PS. Needless to say that data is unaccessible - I cannot mount /dev/md0 as save the most valuable data.
You can see my disappointment - the very specific thing I was excited about failed twice taking 5Tb of my data with it.
