I have a software raid 10 with 6 x 2tb hard drives (raid 1 for /boot), ubuntu 10.04 is the os.
I had a raid controller failure that put 2 drives out of sync, crashed the system and initially the os didnt boot up and went into initramfs instead, saying that drives were busy but I eventually managed to bring the raid up by stopping and assembling the drives.
The os booted up and said that there were filesystem errors, I chose to ignore because it would remount the fs in read-only mode if there was a problem.
Everything seemed to be working fine and the 2 drives started to rebuild, I was sure that it was a sata controller failure because I had dma errors in my log files.
The os crashed soon after that with ext errors.
Now its not bringing up the raid, it says that there is no superblock on /dev/sda2.
I tried to reassemble manually with all the device names but it still would not bring up the raid 10 complaining about the missing superblock on sda2, and sda1 was also dropped from the raid 1.
When I did examine on the raid10 it says that 1 of the initially failed drives is a spare, the other is spare rebuilding and sda2 is removed.
It seems that sda decided to fail right when the system was vulnerable to it because when I boot up a live cd it spews out sda unrecoverable read failures.
I have been trying to fix this all week but I'm not sure where to go with this now, I ordered more hard drives because I didn't have a complete backup, but its too late for that now and the only thing I could do is mirror all the hard drives onto the new ones (I'm not sure whether sda was mirrored without errors).
On the internet I read that you can recover from this by recreating the array with the same options as when it was made, however because sda is failing I cant use it and I don't want to risk using its mirror instead, so I'm waiting to get another hard drive.
I'm also not sure whether to include the out of sync drives or if I can actually use those instead to recover the array.
Sorry if this is a mess to read but I've been trying to fix this all day and its late at night now, any thoughts on this would be greatly appreciated.
I also did a memtest and changed the motherboard in addition to everything else.
EDIT: This is my partition layout
Disk /dev/sdb: 2000.4 GB, 2000398934016 bytes
255 heads, 63 sectors/track, 243201 cylinders, total 3907029168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x0009c34a
Device Boot Start End Blocks Id System
/dev/sdb1 * 2048 511999 254976 83 Linux
/dev/sdb2 512000 3904980991 1952234496 83 Linux
/dev/sdb3 3904980992 3907028991 1024000 82 Linux swap / Solaris
