Rebuilding RAIDframe mirror after crash/power failure

[sherekhan]

I am setting up a new computer with OpenBSD, and have decided to try RAIDframe on it. I have set up two identical 10 GB disks with minimal a-partitions containing a kernel with RAIDframe support and a boot.conf to enable serial console, and b-partitions set up as a RAIDframe mirror, which in turn contains my root, swap and everything else.

I am not running a UPS on this computer, and while power failures are rare in my home they may happen. There is also the slim possibility of a crash, although I think that in all the years I've been using OpenBSD (since 3.4) I can still count all the crash incidents without running out of fingers on my first hand. Still, I have forced a power failure on the computer to see how it coped with that, and I was less than impressed with the time it took to bring the computer back up. The initial mirroring of the drives when I first set them up (raidctl -vi) took like 10-15 minutes, but when rebooting with an out of sync mirror it takes closer to an hour to bring the mirror back up. And the computer is unusable in this time.

- Why does rebuilding take so long?
- Is there anything I can do to speed this process up?
- Is it possible to let the computer boot into multiuser with all services running even with a dirty mirror, and then let it rebuild in the background?

[ocicat]

Originally Posted by sherekhan

- Why does rebuilding take so long?
- Is there anything I can do to speed this process up?
- Is it possible to let the computer boot into multiuser with all services running even with a dirty mirror, and then let it rebuild in the background?

To be clear, I don't run RAIDFrame, however, the misc@ archives indicate that you are not the first to voice similar concerns:

http://marc.info/?l=openbsd-misc&w=2...frame+slow&q=b

My suggestion is to study the archives (including different search terms). You will see others' experiences & possibly find other ideas.

[jggimi]

The default parity rebuild occurs before fsck or mounting. You can ctrl-C, or modify /etc/rc, to have it run in the background.

The risk, as I understand it, for RAID1 is of a failure of one of the mirrors -during- the parity rebuild. If the array is not writeable during the failure, data integrity is assured. But, if the array is in active use, e.g. mounted r/w or in an fsck that is changing blocks -- you may be resorting to a backup for recovery.

I run RAIDframe,

[sherekhan]

Originally Posted by ocicat

To be clear, I don't run RAIDFrame, however, the misc@ archives indicate that you are not the first to voice similar concerns:



My suggestion is to study the archives (including different search terms). You will see others' experiences & possibly find other ideas.

I did find several interesting posts. Several suggested hacking /etc/rc to do raidctl -p instead of raidctl -P, and then do the raidctl -P at the end of /etc/rc.local. I may try this and see if it improves the situation.

[sherekhan]

Originally Posted by jggimi

The default parity rebuild occurs before fsck or mounting. You can ctrl-C, or modify /etc/rc, to have it run in the background.

This behaviour is different from several other OSes with RAIDframe or other software RAID implementations, as far as I understand.

Ctrl-C is not really an option, this is a headless machine. I may modify the startup to let the machine run fsck and all services before starting resync, though.

Originally Posted by jggimi

The risk, as I understand it, for RAID1 is of a failure of one of the mirrors -during- the parity rebuild. If the array is not writeable during the failure, data integrity is assured. But, if the array is in active use, e.g. mounted r/w or in an fsck that is changing blocks -- you may be resorting to a backup for recovery.

That sounds like an acceptable risk to me.

I do have daily backups, so for me the main reason for running RAIDframe is reducing downtime in the event of a disk failure. But having an extra hour or more of downtime every single time the power fails or the computer crashes kind of defeats that purpose.

[jggimi]

Are both of your disks on the same IDE channel, by any chance? That would be a reason for your performance problem.

My RAIDframe system does a parity check of a RAID 1 pair of around 50GB in just over 10 minutes.

[sherekhan]

Originally Posted by jggimi

Are both of your disks on the same IDE channel, by any chance? That would be a reason for your performance problem.

My RAIDframe system does a parity check of a RAID 1 pair of around 50GB in just over 10 minutes.

No, I put them on separate channels. And even then it would not explain why a resync after a hard boot takes so much longer than the initial sync.

[jggimi]

Look at your dmesg and any warnings about your drives that might appear in /var/log/message -- perhaps your drives are running in a low end DMA or PIO mode?

Or, perhaps you have multiple raid* devices defined on this pair of drives? Because the as-shipped /etc/rc runs "raidctl -P all", the parity-rewrite will be run simultaneously on raid0, raid1, raid2, etc. If these are on the same physical devices ... that would explain your horror story.