Linux software raid fails to include one device for one RAID1 array
- by user1389890
One of my four Linux software raid arrays drops one of its two devices when I reboot my system. The other three arrays work fine. I am running RAID1 on kernel version 2.6.32-5-amd64. Every time I reboot, /dev/md2 comes up with only one device. I can manually add the device by saying $ sudo mdadm /dev/md2 --add /dev/sdc1. This works fine, and mdadm confirms that the device has been re-added as follows:
mdadm: re-added /dev/sdc1
After adding the device and and allowing the array time to resynch, this is what the output of $ cat /proc/mdstat looks like:
Personalities : [raid1] 
md3 : active raid1 sda4[0] sdb4[1]
      244186840 blocks super 1.2 [2/2] [UU]
md2 : active raid1 sdc1[0] sdd1[1]
      732574464 blocks [2/2] [UU]
md1 : active raid1 sda3[0] sdb3[1]
      722804416 blocks [2/2] [UU]
md0 : active raid1 sda1[0] sdb1[1]
      6835520 blocks [2/2] [UU]
unused devices: <none>
Then after I reboot, this is what the output of $ cat /proc/mdstat looks like:
Personalities : [raid1] 
md3 : active raid1 sda4[0] sdb4[1]
      244186840 blocks super 1.2 [2/2] [UU]
md2 : active raid1 sdd1[1]
      732574464 blocks [2/1] [_U]
md1 : active raid1 sda3[0] sdb3[1]
      722804416 blocks [2/2] [UU]
md0 : active raid1 sda1[0] sdb1[1]
      6835520 blocks [2/2] [UU]
unused devices: <none>
During reboot, here is the output of $ sudo cat /var/log/syslog | grep mdadm :
Jun 22 19:00:08 rook mdadm[1709]: RebuildFinished event detected on md device /dev/md2
Jun 22 19:00:08 rook mdadm[1709]: SpareActive event detected on md device /dev/md2, component device /dev/sdc1
Jun 22 19:00:20 rook kernel: [ 7819.446412] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:00:20 rook kernel: [ 7819.446415] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:00:20 rook kernel: [ 7819.446782] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:00:20 rook kernel: [ 7819.446785] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:00:20 rook kernel: [ 7819.515844] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:00:20 rook kernel: [ 7819.515847] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:00:20 rook kernel: [ 7819.606829] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:00:20 rook kernel: [ 7819.606832] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:03:48 rook kernel: [ 8027.855616] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:03:48 rook kernel: [ 8027.855620] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:03:48 rook kernel: [ 8027.855950] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:03:48 rook kernel: [ 8027.855952] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:03:49 rook kernel: [ 8027.962169] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:03:49 rook kernel: [ 8027.962171] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:03:49 rook kernel: [ 8028.054365] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:03:49 rook kernel: [ 8028.054368] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:10:23 rook kernel: [    9.588662] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:10:23 rook kernel: [    9.588664] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:10:23 rook kernel: [    9.601990] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:10:23 rook kernel: [    9.601991] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:10:23 rook kernel: [    9.602693] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:10:23 rook kernel: [    9.602695] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:10:23 rook kernel: [    9.605981] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:10:23 rook kernel: [    9.605983] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:10:23 rook kernel: [    9.606138] mdadm: sending ioctl 800c0910 to a partition!
Jun 22 19:10:23 rook kernel: [    9.606139] mdadm: sending ioctl 800c0910 to a partition!
Jun 22 19:10:48 rook mdadm[1737]: DegradedArray event detected on md device /dev/md2
Here is the mdadm.conf file:
ARRAY /dev/md0 metadata=0.90 UUID=92121d42:37f46b82:926983e9:7d8aad9b
ARRAY /dev/md1 metadata=0.90 UUID=9c1bafc3:1762d51d:c1ae3c29:66348110
ARRAY /dev/md2 metadata=0.90 UUID=98cea6ca:25b5f305:49e8ec88:e84bc7f0
ARRAY /dev/md3 metadata=1.2 name=rook:3 UUID=ca3fce37:95d49a09:badd0ddc:b63a4792
I also ran $ sudo smartctl -t long /dev/sdc and no hardware issues were detected. As long as I do not reboot, /dev/md2 seems to work fine. Does anyone have any suggestions?
Here is the output of $ sudo mdadm -E /dev/sdc1 after re-adding the device and letting it resync:
/dev/sdc1:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : 98cea6ca:25b5f305:49e8ec88:e84bc7f0 (local to host rook)
  Creation Time : Sun Jul 13 08:05:55 2008
     Raid Level : raid1
  Used Dev Size : 732574464 (698.64 GiB 750.16 GB)
     Array Size : 732574464 (698.64 GiB 750.16 GB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 2
    Update Time : Mon Jun 24 07:42:49 2013
          State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0
       Checksum : 5fd6cc13 - correct
         Events : 180998
      Number   Major   Minor   RaidDevice State
this     0       8       33        0      active sync   /dev/sdc1
   0     0       8       33        0      active sync   /dev/sdc1
   1     1       8       49        1      active sync   /dev/sdd1
Here is the output of $ sudo mdadm -D /dev/md2 after re-adding the device and letting it resync:
/dev/md2:
        Version : 0.90
  Creation Time : Sun Jul 13 08:05:55 2008
     Raid Level : raid1
     Array Size : 732574464 (698.64 GiB 750.16 GB)
  Used Dev Size : 732574464 (698.64 GiB 750.16 GB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 2
    Persistence : Superblock is persistent
    Update Time : Mon Jun 24 07:42:49 2013
          State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0
           UUID : 98cea6ca:25b5f305:49e8ec88:e84bc7f0 (local to host rook)
         Events : 0.180998
    Number   Major   Minor   RaidDevice State
       0       8       33        0      active sync   /dev/sdc1
       1       8       49        1      active sync   /dev/sdd1