softraid: your experiences and suggestions?

[dogeoj]

Hi guys,

I was looking into softraid but I didn't find a lot of doc, especially about the management aspect. I am unclear about a few things:

I didn't exactly understand how to replace a failed drive in a mirror. It seems the first step to building the mirror is to disklabel the whole drives as RAID (not documented in disklabel manpage as far as I could see) and then once the softraid device is setup, you disklabel the softraid device all over again with your individual slices. Is this right? So then if a device fails you detach it and then attach a new one. But do you have to disklabel and set it to RAID before you attach, or does the attach do a bitwise copy of the remaining devices in the softraid device automagically.. or something else?

After you configure a softraid device is it possible to read the individual drives separately? I'm kinda concerned, given the combination of my lack of UNIX prowess over the past twenty years and the terse doc, that a situation could arise where I can't get my data...which would be directly opposite to the reason I want to do this.

Yes, RAID mirror is my backup strategy. I am worried about disk failure, not about doing something dumb.

If you have any comments or stories about softraid on recent OpenBSD versions please post your experiences!

Thanks,

Joe

[jggimi]

Hello and welcome!

Originally Posted by dogeoj

It seems the first step to building the mirror is to disklabel the whole drives as RAID (not documented in disklabel manpage as far as I could see)...

Not directly, no. Guidance for the disklabel fstype is in the softraid(4) man page, where it describes the RAID fstype and its requirements for use as a softraid "chunk."

As for the disklabel program, there are two man pages, disklabel(8) and disklabel(5). The first describes the disklabel utility, but doesn't discuss individual fstypes. The latter discusses the on-disk and in-core disklabel and system calls for managing a disklabel programmatically. Neither man page discusses fstypes directly.

On our systems is the header file /usr/include/sys/disklabel.h, which lists all the valid DKTYPENAMES (in disklabel terminology, "fstypes"). But this doesn't necessarily help with use cases.

...and then once the softraid device is setup, you disklabel the softraid device all over again with your individual slices. Is this right?

Yes. The use of bioctl(8) creates a virtual softraid disk drive. If you're using RAID 1 (mirroring) with real drives sd0 and sd1, after you've issued the bioctl() command, you'll have a new drive sd2. The virtual drive can then be partitioned with up to 15 separate disklabel partitions (a-b, d-p), etc.

So then if a device fails you detach it and then attach a new one. But do you have to disklabel and set it to RAID before you attach, or does the attach do a bitwise copy of the remaining devices in the softraid device automagically.. or something else?

You must create a RAID disklabel partition on the new (or replacement) drive before rebuilding the mirror. See the -R operand of bioctl(8). If the RAID group is intended to be bootable, you would also need to run installboot(8) manually to reinstall boot blocks.

After you configure a softraid device is it possible to read the individual drives separately?

No. The data is read and written through the virtual drive only.

I'm kinda concerned, given the combination of my lack of UNIX prowess over the past twenty years and the terse doc, that a situation could arise where I can't get my data...which would be directly opposite to the reason I want to do this.

In the event you want to read one of these two mirrors from a "bare metal" system you would need to boot OpenBSD. The mirror can then be used in degraded mode.

If you're nervous, PRACTICE this. Use a pair of USB sticks, or play with it in a virtual machine. You should know how it works normally as well as when there is a loss of one of the drives, and how to recover from drive loss, BEFORE YOU PUT THIS INTO PRODUCTION.

Yes, RAID mirror is my backup strategy. I am worried about disk failure, not about doing something dumb.

I'm sorry, you won't want to read this, but unfortunately, RAID mirroring is not a backup strategy. It's not even a backup. Ignoring human error, any application error may still destroy your data. On both mirrors at the same moment.

If you have any comments or stories about softraid on recent OpenBSD versions please post your experiences!

I was a RAIDframe user before softraid() existed, and even had a "how to" for Root-on-RAIDframe published in the OpenBSD Journal. I've been a softraid() user since its inception, and I'm using it right now as I type this reply.

At one point in my career I was a business continuity and disaster recovery consultant for one of the world's largest infrastructure vendors. When it comes to RAID as a "backup strategy," I know of what I speak.

[dogeoj]

Hi Josh,

How great to see you and so many guys on the forum after so many years! Thanks for a superb, helpful answer as always

Thank you for the clarifications. I think I understand everything you wrote but I have a couple of questions.

Originally Posted by jggimi

In the event you want to read one of these two mirrors from a "bare metal" system you would need to boot OpenBSD. The mirror can then be used in degraded mode.

How do I do that? Do I have to bring up the system in single user mode, should I boot the installer, etc.? Then what? Mount the broken softraid device partitions on /mnt ?

Second question is on rebuilding a mirror. I understood from your post first you have to disklabel the replacement drive as RAID before you can attach it. But do you also have to disklabel the replacement drive as it should be reflected in the softraid device or is that handled automagically?

Thanks!

Joe

[jggimi]

Originally Posted by dogeoj

Do I have to bring up the system in single user mode, should I boot the installer, etc.? Then what? Mount the broken softraid device partitions on /mnt ?

All you need to do is attach the device to the test/recovery system, and boot OpenBSD. Any kernel, including the RAMDISK kernel, will scan the disklabel, discover the RAID partition, read the softraid metadata, and will attach it as a virtual sd(4) device. If the mirrored drive is bootable, you could even boot it directly -- the bootloader is integrated with softraid(4), for root-on-RAID and for FDE with the CRYPTO discipline.

Once you modify a single mirror in the RAID group -- such as merely mounting it r/w -- you have split the mirror and will need to rebuild the RAID group in order to resync mirrors.

Second question is on rebuilding a mirror. I understood from your post first you have to disklabel the replacement drive as RAID before you can attach it. But do you also have to disklabel the replacement drive as it should be reflected in the softraid device or is that handled automagically?

That's a good question, Joe. Something for you to test before going production. To my recollection, you can add a new, never-used chunk when rebuilding a degraded group. But you cannot increase the number of chunks that make up the group.

Keep in mind, especially for testing and familiarizing yourself with softraid(): the chunks do not have to be the size of a full drive, and for testing don't need to be on different drives.. The chunks don't even have to be all the same size -- the RAID group is set to the size of the smallest chunk.

[dogeoj]

Originally Posted by jggimi

All you need to do is attach the device to the test/recovery system, and boot OpenBSD. Any kernel, including the RAMDISK kernel, will scan the disklabel, discover the RAID partition, read the softraid metadata, and will attach it as a virtual sd(4) device. If the mirrored drive is bootable, you could even boot it directly -- the bootloader is integrated with softraid(4), for root-on-RAID and for FDE with the CRYPTO discipline.

Thanks again for tons of great info. I wish all your great posts could be captured in one OpenBSD book...

I have been playing around a bit but not much since I don't have much time lately. I installed 6.4 under Virtual Box. I went through the FAQ steps and created a softraid device and installed. System runs fine of course.

I may be missing something but when I boot the installer, I can't see anything but RAID when I fdisk the first drive in the softraid device. And disklabel borks when I specify the second drive. I just realized I want to check something. Too bad I can't save this as a draft. I'll try a few things and ask better questions. Thanks for your help!

[jggimi]

I'll make some guesses.

1. The RAMDISK kernel you boot to install has very limited "disk space" in RAM. To leave some room for devices we might need to add ourselves, there are a very limited number of device nodes that pre-exist in the /dev directory by default. For hard drives, only wd0 and sd0 device nodes exist when you finish booting and request to enter the shell.
2. To be able to address non-zero device numbers, or any other device missing from /dev you might need, you must run the MAKEDEV(8) script manually to have the device nodes created.
   

   # cd /dev
   # sh MAKEDEV sd1 sd2 sd3
   # fdisk ...
   # fdisk ....
   # disklabel ....
   # disklabel ....
   # bioctl ...
   # install

3. The install / upgrade script creates the device nodes for you, but that doesn't help when you need to run bioctl(8) first.

Thanks again for tons of great info. I wish all your great posts could be captured in one OpenBSD book...

Gee. Aside from the vast majority of them all being well out-of-date and by now most could be described as, "Wrong. Just stupidly wrong" -- I'd like to leave the writing of books to those who excel at it. As an example, I recommend Michael W. Lucas's Absolute OpenBSD and any of his Mastery books. He's got a writing style for tech stuff which makes complicated things understandable and fun. His fiction's excellent, too.