DaemonForums  

Go Back   DaemonForums > FreeBSD > FreeBSD General

FreeBSD General Other questions regarding FreeBSD which do not fit in any of the categories below.

Reply
 
Thread Tools Display Modes
  #1   (View Single Post)  
Old 28th September 2009
DNAeon DNAeon is offline
Shell Scout
 
Join Date: Sep 2008
Location: Bulgaria
Posts: 138
Thanked 6 Times in 6 Posts
Default FreeBSD 7.2 and gvinum issues

Hi,

I'm intended to set up my root file system under gvinum and use mirroring.

Installed a fresh 7.2 system under Virtualbox to make some tests first, everything was working fine and then I did a fresh install again, but soon I found out that /dev/gvinum is not existing anymore.

kldload-ing manually the geom_vinum.ko module or even putting geom_vinum_load="YES" to /boot/loader.conf doesn't create the /dev/gvinum structure.

I can't figure out what I'm missing

Thanks for any advice!
__________________
"I never think of the future. It comes soon enough." - A.E

Useful links: FreeBSD Handbook | FreeBSD Developer's Handbook | The Porter's Handbook | PF User's Guide | unix-heaven.org
Reply With Quote
  #2   (View Single Post)  
Old 29th September 2009
DNAeon DNAeon is offline
Shell Scout
 
Join Date: Sep 2008
Location: Bulgaria
Posts: 138
Thanked 6 Times in 6 Posts
Default

Hi, again

Just a quick update on my gvinum issues - this time installed a 8.0-RC1 system under Virtualbox - just to mention there's a lot work being done on gvinum Now it includes some of the original vinum commands as well as some new like:
  • attach
  • concat
  • detach
  • grow
  • mirror
  • raid5
  • stripe

This time I had some progress settings up the root filesystem on gvinum.
I've also used this documentation, section "Install FreeBSD on Vinum":
http://www.vinumvm.org/cfbsd/vinum.pdf

However not all of the steps were correct, for example:

1. Editing with bsdlabel is being done in a different manner

I had to first mount / and /usr then make a copy of the current labels (bsdlabel ad0s1 > /tmp/bsdlabel.orig). Then edit it as described in the document and save it in /tmp/bsdlabel.new. Then a reboot in single-user mode is required again in order to write the new labels, otherwise you get an error like this:

Code:
bsdlabel: Class not found
re-edit the label? [y]:
/tmp/bsdlabel.orig
Code:
# /dev/ad0s1:
8 partitions:
#        size   offset    fstype   [fsize bsize bps/cpg]
  a:  2097152  2097152    4.2BSD        0     0     0 
  b:  2097152        0      swap                    
  c: 20964762        0    unused        0     0         # "raw" part, don't edit
  d:  2097152  4194304    4.2BSD        0     0     0 
  e: 14673306  6291456    4.2BSD        0     0     0
/tmp/bsdlabel.new (changes in red)
Code:
# /dev/ad0s1:
8 partitions:
#        size   offset    fstype   [fsize bsize bps/cpg]
  a:  2097152  2097152    4.2BSD        0     0     0 
  b:  2096871      281      swap                    
  c: 20964762        0    unused        0     0         # "raw" part, don't edit
  d:  2097152  4194304    4.2BSD        0     0     0 
  e: 14673306  6291456    4.2BSD        0     0     0 
  h: 20964746       16     vinum
Write the new labels after booting in single-user mode again:
Code:
bsdlabel -R ad0s1 /tmp/bsdlabel.new
This is where bsdlabel is showing up some errors, although the new labels are written.
Code:
vnode_pager_getpages: I/O read error
vm_fault: pager read error, pid 21 (bsdlabel)
pid 21 (bsdlabel), uid 0: exited on signal 1
Segmentation fault
Then I had to power-off the system since, it was not operating anymore and go to single-user mode again.

After that you can create your gvinum volumes/drives/subdisks/plexes and make an fsck against the created volumes - everything fine so far.

Added 'geom_vinum_load="YES"' to /boot/loader.conf and rebooted once again.

However upon boot the system is not able to find the root filesystem.

Screenshot here:
http://unix-heaven.org/gfx/gvinum-fbsd-root-fs.png

Even settings the vfs.root.mountfrom* variables from the loader prompt doesn't help much. I had to enter ufs:/dev/ad0s1a and the root filesystem is mounted again. However when I check the mounted filesystems - /usr, /var/, etc.. are all mounted using the gvinum volumes, except for root.

Seems that during boot the root filesystem cannot be found when it is used under gvinum.

Is anyone running a root filesystem under gvinum? Any ideas where to look for from now on?

Thanks!
__________________
"I never think of the future. It comes soon enough." - A.E

Useful links: FreeBSD Handbook | FreeBSD Developer's Handbook | The Porter's Handbook | PF User's Guide | unix-heaven.org
Reply With Quote
  #3   (View Single Post)  
Old 30th September 2009
DNAeon DNAeon is offline
Shell Scout
 
Join Date: Sep 2008
Location: Bulgaria
Posts: 138
Thanked 6 Times in 6 Posts
Default HOWTO: Making your partitions mirrored :)

Hello,

During the last two days I was trying to find a way to make my system and all of it's partitions work under gvinum. Everything works, but the root partition is just not able to boot when it is under gvinum.

Instead I used a different approach - used gvinum and gmirror to successfully mirror my partitions.

I'm sending this to you as well with the hope that someone may find it useful when he/she comes across this issue.

Part I: Making a mirror for the root filesystem


1. Install a fresh FreeBSD system - minimal install

For this setup to work you need at least two disks - I used one 10g and one 15g disks within Virtualbox.

ad0 - 10gb
ad4 - 15gb

When you enter FDISK create one slice in the first disk.

For the second disk create two slices - one which will store the root mirror, and the other we'll use with gvinum later. The first slice of the second disk make to be the size of the root partition which you will use + 5-10MB more - just in case FDISK won't shrink it and make gmirror fail later.

The disk layout that I chose is: 1gb for root, /tmp and swap, 2gb for /var and the rest for /usr

Now this is important - we need to make the first partition in the first slice of the first disk to be the swap area. Do not create any partitions on the second disk.

At the end you will have this:

NOTE: The swap partition comes first, so later we'll use it with gvinum. If you can't wait till later - read this http://www.vinumvm.org/cfbsd/vinum.pdf

First disk (ad0):
Code:
ad0s1b  swap	1024MB	SWAP
ad0s1a  /	1024MB	UFS2
ad0s1d  /tmp	1024MB	UFS2+S
ad0s1e  /var	2048MB	UFS2+S
ad0s1f  /usr	5119MB	UFS2+S
Install the system.

2. When the boot prompt appears enter the loader prompt

Code:
OK load geom_mirror
OK boot -s
3. Now you are in single-user mode

Change to a shell with history (not really required, though )
# csh

Install the bootstrap code on the second disk as well, in case of power failure and the first one is broken FreeBSD will still be able to boot from the second disk.
Code:
# fdisk -BI /dev/ad4s1
# bsdlabel -wB /dev/ad4s1
Create a UFS2 filesystem on the first and second slice on the second disk.

/dev/ad4s1 - this is where we will store the mirror for the root filesystem.

Code:
# newfs -O2 /dev/ad4s1
# newfs -U -O2 /dev/ad4s2
The root filesystem needs to be the first component in the new mirror, otherwise you will wipe out your root partition.

# gmirror label -vb round-robin gm0 /dev/ad0s1a

Now add the second component to the mirror and check the status of the mirror:
Code:
gmirror insert gm0 /dev/ad4s1
GEOM_MIRROR: Device gm0: rebuilding provider ad4s1
# gmirror status
	Name	Status	  Components
mirror/gm0	DEGRADED  ad0s1
			  ad4s1 (14%)
Now the root filesystem is being replicated on the first slice of the second disk.

When it finishes you will see this message:

GEOM_MIRROR: Device gm0: rebuilding provider ad4s1 finished.

You can verify it using the `gmirror status` command.

4. Reboot the system in single-user mode again.
A reboot is required, since gmirror is using the root filesystem and you will not be able to re-mount it in read-write mode.

Mount / and /usr in read-write mode and edit /etc/fstab and /boot/loader.conf, so we use our mirror for the root filesystem now.
Code:
# mount -u /
# mount /usr
# echo 'geom_mirror_load="YES"' >> /boot/loader.conf
# vi /etc/fstab
The line to edit is this:
/dev/ad0s1a / ufs rw 1 1

Change it to this:
/dev/mirror/gm0 / ufs rw 1 1

Then reboot. After FreeBSD boots up you should be running on a mirrored root filesystem.

NOTE: Only 8.0-RC1 related.

During boot under 8.0-RC1 the system complains with this:

Code:
pid 63 (swapon), uid 0: exited on signal 11
....
pid 65: (fsck), uid 0: exited on signal 11
Segmentation fault
Unknown error; help!
ERROR: ABORTING BOOT (sending SIGTERM to parent)!
When that happens just enter single-user mode and mount / and /usr in read-write mode:

Code:
# mount -u /
# mount /usr
Don't know why, but after reboot /etc/fstab looks strange and /boot/loader.conf is empty. So, just repair them and make a reboot again. This won't happen again - I promise

Code:
# echo 'geom_mirror_load="YES"' >> /boot/loader.conf
# vi /etc/fstab
And then remove all the '^@^@^@^@' strange characters from the end of the file.

NOTE: I've made some tests and tried using gvinum as a root filesystem but right after reboot the new root filesystem cannot be mounted when it is under gvinum. That's why for the root filesystem we are using gmirror instead.

PART II: Migrating the other partitions under gvinum

BIG_NOTE: The setup we are going to implement using gmirror and gvinum, currently works only under 8.0-RC1. FreeBSD 7.2 fails when creating the gvinum volumes and prints this error "GEOM_VINUM: g_access failed on drive <gvinum_drive>, errno 1". Continue further only if you are running 8.0-RC1 or later!


Now we can setup the rest of the partitions using gvinum, so we can mirror them, extend them, etc..

1. Migrating the other partitions to gvinum

Remember that we created first the swap partitions on the first slice on the first disks? That's where we are going to use this

The way we are going to migrate the partitions to gvinum is to shrink the swap partition with 281 sectors (265 for gvinum configration and 16 for the bootstrap).

NOTE: Read this http://www.vinumvm.org/cfbsd/vinum.pdf - it contains much more detailed information about why we need to shrink the partitions than this little-howto

We need to use bsdlabel for that purpose.
# bsdlabel ad0s1

Code:
# /dev/ad0s1:
8 partitions:
#        size   offset    fstype   [fsize bsize bps/cpg]
  a:  2097152  2097152	  4.2BSD		0	  0		0
  b:  2097152	      0	    swap		0	  0		0
  c: 20964762	      0	  unused		0	  0		   # "raw" part, don't edit
  d:  2097152  4194304	  4.2BSD		0	  0		0
  e:  4194304  6291456	  4.2BSD		0	  0		0
  f: 10479002 10485760	  4.2BSD		0	  0		0
Save that output somewhere in /tmp and make a copy of it.
Code:
# bsdlabel ad0s1 > /tmp/bsdlabel.orig
# cp /tmp/bsdlabel.orig /tmp/bsdlabel.new
# vi /tmp/bsdlabel.new
Now edit the /tmp/bsdlabel.new file and make a volume of type vinum that starts after the bootstrap and shrink the swap partition with 281 sectors. The result will be this:

-- /tmp/bsdlabel.new --

Code:
# /dev/ad0s1:
8 partitions:
#        size   offset    fstype   [fsize bsize bps/cpg]
  a:  2097152  2097152	  4.2BSD		0	  0		0
  b:  2096871	    281     swap		0	  0		0
  c: 20964762	      0	  unused		0	  0		   # "raw" part, don't edit
  d:  2097152  4194304	  4.2BSD		0	  0		0
  e:  4194304  6291456	  4.2BSD		0	  0		0
  f: 10479002 10485760	  4.2BSD		0	  0		0
  h: 20964746	     16	   vinum

2. Now boot into single-user mode.

Edit the labels using bsdlabel again
Code:
# mount /tmp
# bsdlabel -R ad0s1 /tmp/bsdlabel.new
NOTE_1: Under 8.0-RC1 bsdlabel prints error "Class not found" when trying to edit the labels when geom_mirror.ko is loaded. In order to avoid this under 8.0-RC1 you need to change /etc/fstab to use the old root partition (ad0s1a) and comment the geom_mirror_load="YES" line in /boot/loader.conf.

NOTE_2: After executing the above bsdlabel command you will see some error messages popping-up - check this for more information - http://lists.freebsd.org/pipermail/f...er/003736.html

If you did the steps from NOTE_1 you need now to bring back the mirror partition and enable geom_mirror.ko module upon boot.

In any case at this moment under 8.0-RC1 a reboot is required.

3. Create the gvinum objects - drives, volumes, etc..

For the subdisks you can use the values from bsdlabel, but you need to pay attention to the fact that gvinum starts after the bootstrap and we need to subtract 16 from the offset.

NOTE: Again - check this for more information about this here - http://www.vinumvm.org/cfbsd/vinum.pdf

# gvinum create

This is the config file that I created:

Code:
drive disk_ad0 device /dev/ad0s1h

volume swap 
  plex org concat
	sd length 2096871s driveoffset 265s drive disk_ad0

volume tmp
  plex org concat
	sd length 2097152s driveoffset 4184288s drive disk_ad0

volume var
  plex org concat
	sd length 4194304s driveoffset 6291440s drive disk_ad0

volume usr
  plex org concat
	sd length 10479002s driveoffset 10485744s drive disk_ad0
After gvinum processes this config file the result from `gvinum list` is:

Code:
1 drive:
D disk_ad0              State: up	/dev/ad0s1h	A: 1024/10236 MB (10%)

4 volumes:
V swap                  State: up	Plexes:       1	Size:       1023 MB
V tmp                   State: up	Plexes:       1	Size:       1024 MB
V var                   State: up	Plexes:       1	Size:       2048 MB
V usr                   State: up	Plexes:       1	Size:       5116 MB

4 plexes:
P swap.p0             C State: up	Subdisks:     1	Size:       1023 MB
P tmp.p0              C State: up	Subdisks:     1	Size:       1024 MB
P var.p0              C State: up	Subdisks:     1	Size:       2048 MB
P usr.p0              C State: up	Subdisks:     1	Size:       5116 MB

4 subdisks:
S swap.p0.s0            State: up	D: disk_ad0     Size:       1023 MB
S tmp.p0.s0             State: up	D: disk_ad0     Size:       1024 MB
S var.p0.s0             State: up	D: disk_ad0     Size:       2048 MB
S usr.p0.s0             State: up	D: disk_ad0     Size:       5116 MB
Note the 10% available space that is left on /dev/ad0s1h - this is the root filesystem which we are already mirroring using gmirror.

Check the gvinum volumes (/tmp, /var, /usr) we created using fsck.
# fsck -n -t ufs /dev/gvinum/<volume>

Edit /etc/fstab and use the gvinum volumes now for those partitions. After editing mine looks like this:

Code:
# Device		Mountpoint	FStype	Options		Dump	Pass#
/dev/gvinum/swap	none		swap	sw		0	0
/dev/mirror/gm0		/		ufs	rw		1	1
/dev/gvinum/tmp		/tmp		ufs	rw		2	2
/dev/gvinum/usr		/usr		ufs	rw		2	2
/dev/gvinum/var		/var		ufs	rw		2	2
/dev/acd0		/cdrom		cd9660	ro,noauto	0	0
Load geom_vinum.ko module during boot with this command:

# echo 'geom_vinum_load="YES"' >> /boot/loader.conf

And then reboot. After reboot all partitions should be mounted using gvinum, except for the root filesystem, which is gmirror-ed Now you can mirror the other partitions, extend them, etc. using gvinum.

And that's all

For more information, please check:
Any feedback on how this setup can be approved more is greatly appreciated!

And sorry for the long post if I bored you from reading, and sorry for all those NOTEs as well - I just wanted to make sure you'll notice them

Thanks!

P.S. Looking forward for 8.0-RELEASE to be ready, so I can migrate all of my systems with gmirror and gvinum

EDIT: Added the commands needed to install the bootstrap code on the second disk. This is usefull (and actually needed) in case of power failure and the first disk is broken, then FreeBSD will be able to boot from the second disk.
__________________
"I never think of the future. It comes soon enough." - A.E

Useful links: FreeBSD Handbook | FreeBSD Developer's Handbook | The Porter's Handbook | PF User's Guide | unix-heaven.org

Last edited by DNAeon; 3rd October 2009 at 03:17 PM.
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Encrypting gvinum raid5 oxy FreeBSD General 1 13th December 2008 05:59 PM
FreeBSD Server issues Crypt FreeBSD General 4 10th October 2008 07:44 PM
Weird NAT issues EvanED FreeBSD General 3 11th July 2008 11:02 PM
KVM issues lil_elvis2000 FreeBSD General 5 9th June 2008 07:55 PM
Strange FreeBSD Issues (USB/Nvidia Drivers) bjstick FreeBSD Installation and Upgrading 7 13th May 2008 10:20 AM


All times are GMT. The time now is 05:30 AM.


Powered by vBulletin® Version 3.8.4
Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content copyright © 2007-2010, the authors
Daemon image copyright ©1988, Marshall Kirk McKusick