Safer RAID5 rebuilds using partially failed disks?
- by arcticmac
There have been lots of articles posted recently about how RAID5 is dangerous because of long resilver times, and in particular because of increasing chances of encountering a URE during the resilver. Obviously this is a significant concern. However, it seems that in many cases of interest (as long as you're keeping some kind of eye on your disks), when it comes time to rebuild the array, the disk that I'm replacing is still mostly readable.
If you try to explain this predicament to the average layperson, they are typically very confused as to why you have two almost completely functional disks but can't produce one working array. It seems to me that there ought to be some way to take advantage of this to make rebuilds safer, as long as I'm willing to have the RAID5 be read-only for a couple of days while it rebuilds. Conceptually, what I have in mind looks something like this:
When a disk fails, immediately take the RAID5 offline or mount it read-only
Attach a new disk (either in a spare bay, or externally via eSATA) and begin rebuilding it to replace the failed one. If known, perhaps start with the stripes in which the failure occurred, to minimize the chances of losing those if another disk fails.
In the event that a second disk experiences a URE or other failure during the rebuild, try to source that data from the disk that is being replaced. Presumably if this happens, more rebuilding would be necessary.
When complete, shut down the server, swap the replacement drive into the original bay if desired, and bring the array back up.
Obviously such a process would not be appropriate for applications where uptime is critical or data loss cannot be tolerated, but it seems to me that this could help considerably to improve the reliability of RAID5.
I assume that there's not a good way to implement a recovery like this at present, given that I haven't seen any indication of tools that are designed to do this, and that it seems like it would be rather obtuse to work out manually. Are there also technical issues with it that I haven't thought of (I'm still fairly new to RAID stuff)? Any thoughts on how hard something like this would be to implement (e.g. in linux md raid)?
