Can gmirror be used for this?

[PeterSteele]

We have FreeBSD boxes that consist of four physical drives. We want to divide each of these drives into two slices, where slice s1 will be used strictly for the OS (partitioned with /, /var, swap, etc) and slice s2 (partitioned as one large partition) will be used strictly for data that is managed by our application.

We need to provide a level of HA into our system and what we'd like to do is mirror the OS slice across all four drives (or at least two of them) so that if the main boot drive goes down the system will still boot with the other three drives. We do not want to mirror the second slice, since each drive will contain different information in this slice.

Can we setup this kind of configuration?

[phoenix]

Yes, quite easily. gmirror can be used to create mirrors across any number of partitions, slices, or whole disks.

I ran my home server with 3x 200 GB harddrives, partitioned as:
s1 10 GB for /
s2 1 GB for swap
s3 rest for ZFS

s1 was gmirror'd across all three drives.
s3 was added to a raidz zpool

There are a handful of different how-tos available online for using gmirror for the / partition/slice. This is the one I used on my home servers, and two systems at work.

[PeterSteele]

I see you have a separate slice for the swap space. That makes sense.

I looked at this link and I noticed the comment:

"The following is a step-by-step command list for remotely converting a production FreeBSD system from a single-disk/single-slice (ad0s1) to a two-disk/single-slice (ad0s1 & ad1s1) GEOM mirror (gm0)"

I'm not entirely clear how to adapt these instructions for 3-slice systems? I setup a VM with two single slice drives and followed the first set of instructions, successfully creating a mirrored, single slice system. Now, what about a three slice system?

What about the second approach described here? It seems a lot simpler, but I'm not sure what this means:

dd if=/dev/zero of=/dev/${d1} bs=512 count=79

This implies my main drive would be wiped, so I must be missing something.

[phoenix]

Originally Posted by PeterSteele

I see you have a separate slice for the swap space. That makes sense.

The VM system will stripe access across the disks when it needs to use swap space. Ideally, you wouldn't need swap, but this makes it faster for when you do access it.

I looked at this link and I noticed the comment:

"The following is a step-by-step command list for remotely converting a production FreeBSD system from a single-disk/single-slice (ad0s1) to a two-disk/single-slice (ad0s1 & ad1s1) GEOM mirror (gm0)"

I'm not entirely clear how to adapt these instructions for 3-slice systems? I setup a VM with two single slice drives and followed the first set of instructions, successfully creating a mirrored, single slice system. Now, what about a three slice system?

Just do the second step twice, but change the "second" disk in the script to point to the third slice.

I'm not sure what this means:

dd if=/dev/zero of=/dev/${d1} bs=512 count=79

This implies my main drive would be wiped, so I must be missing something

The first 79 blocks, or ~40 bytes, of the drive will be wiped. This clears out the boot blocks and the partition table, effectively making the drive appear to be unformatted.

[PeterSteele]

I should clarify--I don't actually want to mirror S3, but rather leave it untouched. We just want the S1 (OS) slice mirrored since our application uses the S3 slice for its own purposes. I guess we wouldn't mirror the swap slice either, but just let the OS stripe it as it normally would.

Can you explain what are the advantages the two approaches described in this link. The second one is considerably less involved, so why wouldn't I want to do it that way?

[phoenix]

No, no, I meant, use the scripts provided on this page, but run the second script twice. The first time, using ad0s1 and ad2s1, the second time using ad2s1 and ad4s1. That way, it creates the mirror using the first slice of the first two disks, then it adds the first slice of the third disk. To create the 3-way mirror. That's how I did it at home, and it worked fine.

Basically, the system setup becomes:
do the install to 1 disk, setting up the slices/partitions as you want them to be on all three disks
boot into OS
slice/partition the rest of the disks as needed, using ad0 as the template
run the first script using ad0s1 and ad2s1 for the variable names
reboot using gm0
run the second script using ad0s1 and ad2s1 for the variable names, to sync the two
reboot using gm0
run the second script using ad2s1 and ad4s1 for the variable names, to add the third

At that point, you'll have gm0 made up of ad0s1, ad2s1, and ad4s1, with s3 on all of them available for whatever you want them to be used for.

(Obviously, the ad device numbers will depend on how your devices are attached, I used 0, 2, and 4 just to put numbers in.)

[PeterSteele]

Okay, I think I got it. Think I'll rebuild my VM and start from scratch. Thanks!

[PeterSteele]

One other question: When I run the second script, the dd command complains:

dd if=/dev/zero of=/dev/${d1} bs=512 count=79
dd: /dev/ad0: Operation not permitted

The script is still running (doing the synchronize step), but what does this dd failure imply, as far as the final outcome is concerned?

I also noticed some fdisk errors from both the first script and the second script:

fdisk: conflict with open slices
fdisk: failed to write sector zero

The first script ran to completion and these errors appeared to be benign, but I'm curious if this is something I should be looking into?

[phoenix]

Did you set the sysctl kern.geom.debugflags=16 before starting?

Does your user (if not running as root) have write access to /dev/adX?

The dd command is not strictly needed. All it does is wipe out the boot blocks and partition table to make the disk appear as unformatted. It makes the initial fdisk/bsdlabel steps look neater, as you don't have to delete existing info. But it's not crucial to do.

I don't think I've seen those fdisk errors, but it's been several months since I did this at home using slices (the ones I've done at work have been using whole disks, USB and CF). I don't think it's a problem. But you may want to compare the output of fdisk /dev/ad0 and fdisk /dev/ad1 to make sure they are identical, before rebooting.

[PeterSteele]

No, I did not set debug mode. I just did now, but it doesn't seem to be giving any additional information. And I'm running as root so there shouldn't be any problem accessing /dev/adX.

I've progressed past these errors though. Not sure yet if they mean anything. I successfully completed the first script on ad0/ad1 and ran the second script on ad0/ad1. Other than these errors I mentioned everything seemed to work okay.

How do I undo the mirroring? I'm getting errors when trying to add the third disk to the mix and I'd like to start from scratch, without having to rebuild my VM.

These are the errors I'm seeing:

79+0 records in
79+0 records out
40448 bytes transferred in 0.053001 secs (763156 bytes/sec)
fdisk: invalid fdisk partition table found
fdisk: conflict with open slices
******* Working on device /dev/ad1 *******
parameters extracted from in-core disklabel are:
cylinders=33288 heads=15 sectors/track=63 (945 blks/cyl)

Figures below won't work with BIOS for partitions not in cyl 1
parameters to be used for BIOS calculations are:
cylinders=33288 heads=15 sectors/track=63 (945 blks/cyl)

Information from DOS bootblock is:
1: sysid 165 (0xa5),(FreeBSD/NetBSD/386BSD)
start 63, size 20971377 (10239 Meg), flag 80 (active)
beg: cyl 0/ head 1/ sector 1;
end: cyl 687/ head 14/ sector 63
2: <UNUSED>
3: <UNUSED>
4: <UNUSED>
gmirror: Provider /dev/ad1s1 already inserted.

[phoenix]

Hmm, maybe you can't use the second script to do it automatically.

If the third disk is already partitioned correctly, you can just manually add the slice to the existing mirror: gmirror insert gm0 ad2s1

[PeterSteele]

Okay, I'm still working on this, with varying degrees of success. One question I have is that in your earlier post you recommended to slice/partition all of the disks as I want them, using the first disk as a template. I did that, ending up with the three disks having three slices. Slice s1 is the OS, slice s2 is swap and slice s3 is reserved for our purposes:

g c33288 h15 s63
p 1 0xa5 63 20971377
a 1
p 2 0xa5 20971440 2096955
p 3 0xa5 23068395 8388765

However, one of the steps performed by the rse script is this:

(echo "p 1 165 63 $size"; echo "a 1") | fdisk -v -f- -i /dev/${d2}

This destroys the slices I created on disk 2, leaving a single slice, so there doesn't seem to be any purpose in slicing/partitioning the second and third drives until the whole process is done. Please correct me if I'm missing something.

Ignoring that issue, the script seems to work without any obvious problems. After the dump/restore operation is done (and before the reboot), gmirror shows this:

# gmirror list gm0
Geom name: gm0
State: COMPLETE
Components: 2
Balance: round-robin
Slice: 4096
Flags: NONE
GenID: 0
SyncID: 1
ID: 4032291634
Providers:
1. Name: mirror/gm0
Mediasize: 10736860672 (10G)
Sectorsize: 512
Mode: r2w2e3
Consumers:
1. Name: ad1s1
Mediasize: 10736861184 (10G)
Sectorsize: 512
Mode: r1w1e1
State: ACTIVE
Priority: 0
Flags: DIRTY
GenID: 0
SyncID: 1
ID: 1062144502
2. Name: ad0s1
Mediasize: 10737345024 (10G)
Sectorsize: 512
Mode: r1w1e1
State: ACTIVE
Priority: 1
Flags: DIRTY
GenID: 0
SyncID: 1
ID: 613987541

The thing that catches my eye here is that all three media size values listed here are different. I would think at least that the media sizes for ad0s1 and ad1s1 should be the same?

After rebooting, I run the second script and reboot. When the system comes up, the mirror is shown as degraded:

# gmirror list gm0
Geom name: gm0
State: DEGRADED
Components: 2
Balance: round-robin
Slice: 4096
Flags: NONE
GenID: 0
SyncID: 2
ID: 4032291634
Providers:
1. Name: mirror/gm0
Mediasize: 10736860672 (10G)
Sectorsize: 512
Mode: r2w2e3
Consumers:
1. Name: ad1s1
Mediasize: 10736861184 (10G)
Sectorsize: 512
Mode: r1w1e1
State: ACTIVE
Priority: 0
Flags: NONE
GenID: 0
SyncID: 2
ID: 1062144502

As you can see ad0s1 has dropped out. The slice/partition table has been modified as well, resulting from this step when the second script was run:

(echo "p 1 165 63 $size"; echo "a 1") | fdisk -v -B -f- -i /dev/${d1}

Again, this step implies that I shouldn't bother with setting up any more than the first slice on my disks, since these fdisk commands destroy the slices.

At this point, I cannot add the third disk to the mirror:

# gmirror insert gm0 ad3s1
gmirror: Not all disks connected.

I'm not sure what I'm doing wrong, but apparently I must be missing something. Any suggestions?

[PeterSteele]

I just had a successful run through. This time I created a single partition only on the original disk, leaving the disks that would be added to the mirror untouched. I now have a three way mirror up and running, with single slice on all three disks. I assume I can now create the extra, non-mirrored partitions as a post operation to this exercise...