|
||||
HOWTO: FreeBSD ZFS Madness
0. This is SPARTA!
Some time ago I found a good, reliable way of using and installing FreeBSD and described it in my Modern FreeBSD Install [1] [2] HOWTO. Now, more then a year later I come back with my experiences about that setup and a proposal of newer and probably better way of doing it. 1. Introduction Same as year ago, I assume that You would want to create fresh installation of FreeBSD using one or more hard disks, but also with (laptops) and without GELI based full disk encryption. This guide was written when FreeBSD 9.0 and 8.3 were available and definitely works for 9.0, but I did not try all this on the older 8.3, if You find some issues on 8.3, let me know I will try to address them in this guide. Earlier, I was not that confident about booting from the ZFS pool, but there is some very neat feature that made me think ZFS boot is now mandatory. If You just smiled, You know that I am thinking about Boot Environments feature from Illumos/Solaris systems. In case You are not familiar with the Boot Environments feature, check the Managing Boot Environments with Solaris 11 Express PDF white paper [3]. Illumos/Solaris has the beadm(1M) [4] utility and while Philipp Wuensche wrote the manageBE script as replacement [5], it uses older style used at times when OpenSolaris (and SUN) were still having a great time. I last couple of days writing an up-to-date replacement for FreeBSD compatible beadm utility, and with some tweaks from today I just made it available at SourceForge [6] if You wish to test it. Currently its about 200 lines long, so it should be pretty simple to take a look at it. I tried to make it as compatible as possible with the 'upstream' version, along with some small improvements, it currently supports basic functions like list, create, destroy and activate. Code:
# beadm usage: beadm subcommand cmd_options subcommands: beadm activate beName beadm create [-e nonActiveBe | beName@snapshot] beName beadm create beName@snapshot beadm destroy beName beadm destroy beName@snapshot beadm list My implementation is also style compatible with current Illumos/Solaris beadm(1M) which is like the example below. Code:
# beadm create -e default upgrade-test Created successfully # beadm list BE Active Mountpoint Space Policy Created default N / 1.06M static 2012-02-03 15:08 upgrade-test R - 560M static 2012-04-24 22:22 new - - 8K static 2012-04-24 23:40 # zfs list -r sys/ROOT NAME USED AVAIL REFER MOUNTPOINT sys/ROOT 562M 8.15G 144K none sys/ROOT/default 1.48M 8.15G 558M legacy sys/ROOT/new 8K 8.15G 558M none sys/ROOT/upgrade-test 560M 8.15G 558M none # beadm activate default Activated successfully # beadm list BE Active Mountpoint Space Policy Created default NR / 1.06M static 2012-02-03 15:08 upgrade-test - - 560M static 2012-04-24 22:22 new - - 8K static 2012-04-24 23:40 2. Now You're Thinking with Portals The main purpose of the Boot Environments concept is to make all risky tasks harmless, to provide an easy way back from possible troubles. Think about upgrading the system to newer version, an update of 30+ installed packages to latest versions, testing software or various solutions before taking the final decision, and much more. All these tasks are now harmless thanks to the Boot Environments, but this is just the tip of the iceberg. You can now move desired boot environment to other machine, physical or virtual and check how it will behave there, check hardware support on the other hardware for example or make a painless hardware upgrade. You may also clone Your desired boot environment and ... start it as a Jail for some more experiments or move Your old physical server install into FreeBSD Jail because its not that heavily used anymore but it still have to be available. Other good example may be just created server on Your laptop inside VirtualBox virtual machine. After you finish the creation process and tests, You may move this boot environment to the real server and put it into production. Or even move it into VMware ESX/vSphere virtual machine and use it there. As You see the possibilities with Boot Environments are unlimited. 3. The Install Process I created 3 possible schemes which should cover most demands, choose one and continue to the next step. 3.1. Server with Two Disks I assume that this server has 2 disks and we will create ZFS mirror across them, so if any of them will be gone the system will still work as usual. I also assume that these disks are ada0 and ada1. If You have SCSI/SAS drives there, they may be named da0 and da1 accordingly. The procedures below will wipe all data on these disks, You have been warned. Code:
1. Boot from the FreeBSD USB/DVD. 2. Select the 'Live CD' option. 3. login: root 4. # sh 5. # DISKS="ada0 ada1" 6. # for I in ${DISKS}; do > NUMBER=$( echo ${I} | tr -c -d '0-9' ) > gpart destroy -F ${I} > gpart create -s GPT ${I} > gpart add -t freebsd-boot -l bootcode${NUMBER} -s 128k ${I} > gpart add -t freebsd-zfs -l sys${NUMBER} ${I} > gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ${I} > done 7. # zpool create -f -o cachefile=/tmp/zpool.cache sys mirror /dev/gpt/sys* 8. # zfs set mountpoint=none sys 9. # zfs set checksum=fletcher4 sys 10. # zfs set atime=off sys 11. # zfs create sys/ROOT 12. # zfs create -o mountpoint=/mnt sys/ROOT/default 13. # zpool set bootfs=sys/ROOT/default sys 14. # cd /usr/freebsd-dist/ 15. # for I in base.txz kernel.txz; do > tar --unlink -xvpJf ${I} -C /mnt > done 16. # cp /tmp/zpool.cache /mnt/boot/zfs/ 17. # cat << EOF >> /mnt/boot/loader.conf > zfs_load=YES > vfs.root.mountfrom="zfs:sys/ROOT/default" > EOF 18. # cat << EOF >> /mnt/etc/rc.conf > zfs_enable=YES > EOF 19. # :> /mnt/etc/fstab 20. # zfs umount -a 21. # zfs set mountpoint=legacy sys/ROOT/default 22. # reboot Code:
# gpart show => 34 1048509 ada0 GPT (512M) 34 256 1 freebsd-boot (128k) 290 1048253 2 freebsd-zfs (511M) => 34 1048509 ada1 GPT (512M) 34 256 1 freebsd-boot (128k) 290 1048253 2 freebsd-zfs (511M) # gpart list | grep label label: bootcode0 label: sys0 label: bootcode1 label: sys1 # zpool status pool: sys state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM sys ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 gpt/sys0 ONLINE 0 0 0 gpt/sys1 ONLINE 0 0 0 errors: No known data errors If Your server configuration has only one disk, lets assume its ada0, then You need different points 5. and 7. to make, use these instead of the ones above. Code:
5. # DISKS="ada0" 7. # zpool create -f -o cachefile=/tmp/zpool.cache sys /dev/gpt/sys* 3.3. Road Warrior Laptop The procedure is quite different for Laptop because we will use the full disk encryption mechanism provided by GELI and then setup the ZFS pool. Its not currently possible to boot off from the ZFS pool on top of encrypted GELI provider, so we will use setup similar to the Server with ... one but with additional local pool for /home and /root partitions. It will be password based and You will be asked to type-in that password at every boot. The install process is generally the same with new instructions added for the GELI encrypted local pool, I put them with different color to make the difference more visible. Code:
1. Boot from the FreeBSD USB/DVD. 2. Select the 'Live CD' option. 3. login: root 4. # sh 5. # DISKS="ada0" 6. # for I in ${DISKS}; do > NUMBER=$( echo ${I} | tr -c -d '0-9' ) > gpart destroy -F ${I} > gpart create -s GPT ${I} > gpart add -t freebsd-boot -l bootcode${NUMBER} -s 128k ${I} > gpart add -t freebsd-zfs -l sys${NUMBER} -s 10G ${I} > gpart add -t freebsd-zfs -l local${NUMBER} ${I} > gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ${I} > done 7. # zpool create -f -o cachefile=/tmp/zpool.cache sys /dev/gpt/sys0 8. # zfs set mountpoint=none sys 9. # zfs set checksum=fletcher4 sys 10. # zfs set atime=off sys 11. # zfs create sys/ROOT 12. # zfs create -o mountpoint=/mnt sys/ROOT/default 13. # zpool set bootfs=sys/ROOT/default sys 14. # geli init -b -s 4096 -e AES-CBC -l 128 /dev/gpt/local0 15. # geli attach /dev/gpt/local0 16. # zpool create -f -o cachefile=/tmp/zpool.cache local /dev/gpt/local0.eli 17. # zfs set mountpoint=none local 18. # zfs set checksum=fletcher4 local 19. # zfs set atime=off local 20. # zfs create local/home 21. # zfs create -o mountpoint=/mnt/root local/root 22. # cd /usr/freebsd-dist/ 23. # for I in base.txz kernel.txz; do > tar --unlink -xvpJf ${I} -C /mnt > done 24. # cp /tmp/zpool.cache /mnt/boot/zfs/ 25. # cat << EOF >> /mnt/boot/loader.conf > zfs_load=YES > geom_eli_load=YES > vfs.root.mountfrom="zfs:sys/ROOT/default" > EOF 26. # cat << EOF >> /mnt/etc/rc.conf > zfs_enable=YES > EOF 27. # :> /mnt/etc/fstab 28. # zfs umount -a 29. # zfs set mountpoint=legacy sys/ROOT/default 30. # zfs set mountpoint=/home local/home 31. # zfs set mountpoint=/root local/root 32. # reboot Code:
# gpart show => 34 8388541 ada0 GPT (4.0G) 34 256 1 freebsd-boot (128k) 290 2097152 2 freebsd-zfs (1.0G) 2097442 6291133 3 freebsd-zfs (3G) # gpart list | grep label label: bootcode0 label: sys0 label: local0 # zpool status pool: local state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM sys ONLINE 0 0 0 gpt/local0.eli ONLINE 0 0 0 errors: No known data errors pool: sys state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM sys ONLINE 0 0 0 gpt/sys0 ONLINE 0 0 0 errors: No known data errors 1. Login as root with empty password. login: root password: [ENTER] 2. Create initial snapshot after install. # zfs snapshot -r sys/ROOT/default@install 3. Set new root password. # passwd 4. Set machine's hostname. # echo hostname=hostname.domain.com >> /etc/rc.conf 5. Set proper timezone. # tzsetup 6. Add some swap space. If You used the Server with ... type, then use this to add swap. Code:
# zfs create -V 1G -o org.freebsd:swap=on \ -o checksum=off \ -o sync=disabled \ -o primarycache=none \ -o secondarycache=none sys/swap # swapon /dev/zvol/sys/swap Code:
# zfs create -V 1G -o org.freebsd:swap=on \ -o checksum=off \ -o sync=disabled \ -o primarycache=none \ -o secondarycache=none local/swap # swapon /dev/zvol/local/swap After You configured Your fresh FreeBSD system, added needed packages and services, create snapshot called configured or production so if You mess something, You can always go back in time to bring working configuration back. mess something. # zfs snapshot -r sys/ROOT/default@configured 5. Enable Boot Environments Here are some simple instructions on how to download and enable the beadm command line utility for easy Boot Environments administration. Code:
# fetch https://downloads.sourceforge.net/project/beadm/beadm -o /usr/sbin/beadm # chmod +x /usr/sbin/beadm # rehash # beadm list BE Active Mountpoint Space Policy Created default NR / 592M static 2012-04-25 02:03 Now we have a working ZFS only FreeBSD system, I will put some example here about what You now can do with this type of installation and of course the Boot Environments feature. 6.1. Create New Boot Environment Before Upgrade 1. Create new environment from the current one. # beadm create upgrade Created successfully 2. Activate it. # beadm activate upgrade Activated successfully 3. Reboot into it. # shutdown -r now 4. Mess with it. You are now free to do anything You like fo or the upgrade process, but even if You break everything, You still have a working default working environment. 6.2. Perform Upgrade within a Jail This concept is about creating new boot environment from the desired one, lets call it jailed, then start that new environment inside a FreeBSD Jail and perform upgrade there. After You have finished all tasks related to this upgrade and You are satisfied with the achieved results, shutdown that Jail, set the boot environment into that just upgraded Jail called jailed and reboot into just upgraded system without any risks. 1. Create new boot environment called jailed. # beadm create -e default jailed Created successfully 2. Create /usr/jails directory. # mkdir /usr/jails 3. Set mount point of new boot environment to /usr/jails/jailed dir. # zfs set mountpoint=/usr/jails/jailed sys/ROOT/jailed 3.1. Make new Jail dataset mountable. # zfs set canmount=noauto sys/ROOT/jailed 3.2. Mount new Jail dataset. # zfs mount sys/ROOT/jailed 4. Enable FreeBSD Jails mechanism and the jailed Jail in /etc/rc.conf file. # cat << EOF >> /etc/rc.conf > jail_enable=YES > jail_list="jailed" > jail_jailed_rootdir="/usr/jails/jailed" > jail_jailed_hostname="jailed" > jail_jailed_ip="10.20.30.40" > jail_jailed_devfs_enable="YES" > EOF 5. Start the Jails mechanism. # /etc/rc.d/jail start Configuring jails:. Starting jails: jailed. 6. Check if the jailed Jail started. Code:
# jls JID IP Address Hostname Path 1 10.20.30.40 jailed /usr/jails/jailed # jexec 1 tcsh 8. PERFORM ACTUAL UPGRADE. 9. Stop the jailed Jail. # /etc/rc.d/jail stop Stopping jails: jailed. 10. Disable Jails mechanism in /etc/rc.conf file. # sed -i '' -E s/"^jail_enable.*$"/"jail_enable=NO"/g /etc/rc.conf 11. Activate just upgraded jailed boot environment. # beadm activate jailed Activated successfully 12. Reboot into upgraded system. 6.3. Import Boot Environment from Other Machine Lets assume, that You need to upgrade or do some major modification to some of Your servers, You will then create new boot environment from the default one, move it to other 'free' machine, perform these tasks there and after everything is done, move the modified boot environment to the production without any risks. You may as well transport that environment into You laptop/workstation and upgrade it in a Jail like in step 6.2 of this guide. 1. Create new environment on the production server. # beadm create upgrade Created successfully. 2. Send the upgrade environment to test server. # zfs send sys/ROOT/upgrade | ssh TEST zfs recv -u sys/ROOT/upgrade 3. Activate the upgrade environment on the test server. # beadm activate upgrade Activated successfully. 4. Reboot into the upgrade environment on the test server. # shutdown -r now 5. PERFORM ACTUAL UPGRADE AFTER REBOOT. 6. Sent the upgraded upgrade environment onto production server. # zfs send sys/ROOT/upgrade | ssh PRODUCTION zfs recv -u sys/ROOT/upgrade 7. Activate upgraded upgrade environment on the production server. # beadm activate upgrade Activated successfully. 8. Reboot into the upgrade environment on the production server. # shutdown -r nowCourier New 7. References [1] http://forums.freebsd.org/showthread.php?t=10334 [2] http://forums.freebsd.org/showthread.php?t=12082 [3] http://docs.oracle.com/cd/E19963-01/pdf/820-6565.pdf [4] http://docs.oracle.com/cd/E19963-01/.../beadm-1m.html [5] http://anonsvn.h3q.com/projects/free.../wiki/manageBE [6] https://sourceforge.net/projects/beadm/ The last part of the HOWTO remains the same as Year ago ... You can now add your users, services and packages as usual on any FreeBSD system, have fun ;)
__________________
religions, worst damnation of mankind "If 386BSD had been available when I started on Linux, Linux would probably never had happened." Linus Torvalds Linux is not UNIX! Face it! It is not an insult. It is fact: GNU is a recursive acronym for “GNU's Not UNIX”. vermaden's: links resources deviantart spreadbsd Last edited by vermaden; 20th June 2012 at 12:21 PM. |
|
||||
The problematic PR about ZFS canmount property has been fixed (thanks to Bryan Drewery) and merged to HEAD (with MFC: 1 week): http://freebsd.org/cgi/query-pr.cgi?pr=167905
So now beadm is fully functional on FreeBSD HEAD and will be in 9-STABLE in less then a week, or You may apply the patch Yourself from here: http://freshbsd.org/commit/freebsd/r237119 With these instructions: # cd /usr/src/cddl
__________________
religions, worst damnation of mankind "If 386BSD had been available when I started on Linux, Linux would probably never had happened." Linus Torvalds Linux is not UNIX! Face it! It is not an insult. It is fact: GNU is a recursive acronym for “GNU's Not UNIX”. vermaden's: links resources deviantart spreadbsd |
|
||||
Little ERRATA, thanks to srivo:
Quote:
__________________
religions, worst damnation of mankind "If 386BSD had been available when I started on Linux, Linux would probably never had happened." Linus Torvalds Linux is not UNIX! Face it! It is not an insult. It is fact: GNU is a recursive acronym for “GNU's Not UNIX”. vermaden's: links resources deviantart spreadbsd |
|
||||
Updates to the beadm utility:
- minor fixes and clean - added -F switch for destroy option - does not need confirmation upon destroy - implemented umount option with -f switch for umount -f (force) - implemented mount option with several variants of usage, examples: Code:
# beadm usage: beadm subcommand cmd_options subcommands: beadm activate beName beadm create [-e nonActiveBe | -e beName@snapshot] beName beadm create beName@snapshot beadm destroy [-F] beName | beName@snapshot beadm list beadm mount beadm mount beName [mountpoint] beadm umount [-f] beName beadm rename origBeName newBeName # beadm mount update sys/ROOT/update / # beadm mount test /test Mounted successfully on '/test' # beadm mount default Mounted successfully on '/tmp/tmp.KhAtHe' # beadm mount default sys/ROOT/default /tmp/tmp.KhAtHe test sys/ROOT/test /test sys/ROOT/test/SOMETHING /test/test update sys/ROOT/update / # beadm umount test Unmounted successfully # beadm umount -f default Unmounted successfully
__________________
religions, worst damnation of mankind "If 386BSD had been available when I started on Linux, Linux would probably never had happened." Linus Torvalds Linux is not UNIX! Face it! It is not an insult. It is fact: GNU is a recursive acronym for “GNU's Not UNIX”. vermaden's: links resources deviantart spreadbsd |
|
||||
Welcome. Fell free to report any issues.
__________________
religions, worst damnation of mankind "If 386BSD had been available when I started on Linux, Linux would probably never had happened." Linus Torvalds Linux is not UNIX! Face it! It is not an insult. It is fact: GNU is a recursive acronym for “GNU's Not UNIX”. vermaden's: links resources deviantart spreadbsd |
|
||||
The beadm 0.8 has just been commited to the Ports tree:
http://freshports.org/sysutils/beadm Changelog: Code:
-- Introduce proper space calculation by each boot environment in *beadm list* -- Rework the *beadm destroy* command so no orphans are left after destroying boot environment. -- Fix the *beadm mount* and *beadm umount* commands error handling. -- Rework consistency of all error and informational messages. -- Simplify and cleanup code where possible. -- Fix *beadm destroy* for 'static' (not cloned) boot environments received by *zfs receive* command. -- Use mktemp(1) where possible. -- Implement *beadm list -a* option to list all datasets and snapshots of boot environments. -- Add proper mountpoint listing to the *beadm list* command. % beadm list BE Active Mountpoint Space Created default NR / 11.0G 2012-07-28 00:01 test1 - /tmp/tmp.IUQuFO 41.2M 2012-08-27 21:20 test2 - - 56.6M 2012-08-27 21:20 -- Change snapshot format to the one used by original *beadm* command (%Y-%m-%d-%H:%M:%S). % zfs list -t snapshot -o name -r sys/ROOT/default NAME sys/ROOT/default@2012-08-27-21:20:00 sys/ROOT/default@2012-08-27-21:20:18 -- Implement *beadm list -D* option to display space that would be consumed by single boot environment if all other boot environments will be destroyed. % beadm list -D BE Active Mountpoint Space Created default NR / 9.4G 2012-07-28 00:01 test1 - /tmp/tmp.IUQuFO 8.7G 2012-08-27 21:20 test2 - 8.7G 2012-08-27 21:20 -- Add an option to BEADM DESTROY command to not destroy manually created snapshots used for boot environment. # beadm destroy test1 Are you sure you want to destroy 'test1'? This action cannot be undone (y/[n]): y Boot environment 'test1' was created from existing snapshot Destroy 'default@test1' snapshot? (y/[n]): y Destroyed successfully # beadm destroy test1 Are you sure you want to destroy 'test1'? This action cannot be undone (y/[n]): y Boot environment 'test1' was created from existing snapshot Destroy 'default@test1' snapshot? (y/[n]): n Origin snapshot 'default@test1' will be preserved Destroyed successfully
__________________
religions, worst damnation of mankind "If 386BSD had been available when I started on Linux, Linux would probably never had happened." Linus Torvalds Linux is not UNIX! Face it! It is not an insult. It is fact: GNU is a recursive acronym for “GNU's Not UNIX”. vermaden's: links resources deviantart spreadbsd |
|
|||
boot from usb to 100% encrypted laptop zfs over geli
hi here i describe the procedure to encrypt everything including the freebsd system that i use on my laptop and use a usb key with the bootcode, kernel and keys, you can detach the usb media after system boots (btw you'll have to enter two passphrases).
boot from freebsd 9.x usb live disk da0, hard drive is ada0, Code:
# gpart destroy -F ada0 # gpart create -s GPT ada0 # gpart add -t freebsd-boot -l bootcode -s 128k ada0 # gpart add -t freebsd-swap -l swap -s 8G ada0 # gpart add -t freebsd-zfs -l system -s 10G ada0 # gpart add -t freebsd-zfs -l local ada0 # gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0 # glabel label -v system /dev/ada0p3 # glabel label -v local /dev/ada0p4 # dd if=/dev/zero of=/dev/da1 # gpart destroy -F da1 # gpart create -s GPT da1 # gpart add -t freebsd-boot -l bootcode -s 128k da1 # gpart add -t freebsd-ufs -l cryptokey da1 # gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 da1 # newfs /dev/da1p2 # glabel label -v cryptokey /dev/da1p2 # mkdir /media # mount /dev/label/cryptokey /media # mkdir -m 700-p /media/boot/keys /media/backups # mkdir /media/etc # dd if=/dev/random of=/media/boot/keys/system.key bs=64 count=1 # dd if=/dev/random of=/media/boot/keys/local.key bs=64 count=1 # chmod 600 /media/boot/keys/*.key # geli init -e aes -l 128 -K /media/boot/keys/system.key -b -s 4096 -B /media/backups/system.eli.meta /dev/label/system # geli init -e aes -l 128 -K /media/boot/keys/local.key -b -s 4096 -B /media/backups/local.eli.meta /dev/label/local # geli attach -k /media/boot/keys/system.key /dev/label/system # geli attach -k /media/boot/keys/local.key /dev/label/local # zpool create -f -o cachefile=/tmp/zpool.cache system /dev/label/system.eli # zfs set mountpoint=none system # zfs set checksum=fletcher4 system # zfs set atime=off system # zfs create system/ROOT # zfs create -o mountpoint=/mnt system/ROOT/default # zpool set bootfs=system/ROOT/default system # zpool create -f -o cachefile=/tmp/zpool.cache local /dev/label/local.eli # zfs set mountpoint=none local # zfs set checksum=fletcher4 local # zfs set atime=off local # zfs create local/home # zfs create -o mountpoint=/mnt/root local/root # cd /usr/freebsd-dist/ # for T in base.txz kernel.txz; do > tar --unlink -xvpJf ${T} -C /mnt > done # cp /tmp/zpool.cache /mnt/boot/zfs/ # cat << EOF >> /mnt/boot/loader.conf > zfs_load=YES > ahci_load=YES > geom_eli_load=YES > geli_label_system_keyfile0_load=YES > geli_label_system_keyfile0_type="label/system:geli_keyfile0" > geli_label_system_keyfile0_name="/boot/keys/system.key" > geli_label_local_keyfile0_load=YES > geli_label_local_keyfile0_type="label/local:geli_keyfile0" > geli_label_local_keyfile0_name="/boot/keys/local.key" > vfs.root.mountfrom="zfs:system/ROOT/default" > EOF # cat << EOF >> /mnt/etc/rc.conf > zfs_enable=YES > EOF # echo /dev/ada0p2.eli none swap sw 0 0 > /mnt/etc/fstab # cp -Rp /mnt/boot /media/ # cp /mnt/etc/rc.conf /media/etc/ # cp /mnt/etc/fstab /media/etc/ # zfs umount -a # zfs set mountpoint=legacy system/ROOT/default # zfs set mountpoint=/home local/home # zfs set mountpoint=/root local/root # reboot |
|
||||
@silex
Its pointless to create both sys and local pools as You encrypt both of them, just create the encrypted sys pool. With USB thumbs that size below, its even aplyable to laptops: ... but the main question is: How it beadm working with it? (as this is this tutorial all about)
__________________
religions, worst damnation of mankind "If 386BSD had been available when I started on Linux, Linux would probably never had happened." Linus Torvalds Linux is not UNIX! Face it! It is not an insult. It is fact: GNU is a recursive acronym for “GNU's Not UNIX”. vermaden's: links resources deviantart spreadbsd |
|
|||
hi vermaden, what prompted me to share this is not beadm but the fact that your howto for zfs is the best around the interwebs and believe me i looked at so many. so all in all my addition was about encrypting the whole system and using a usb bootkey. on a side note i'm using the config above for a nas, didn't check beadm as of yet, so do you imply that it won't work with the system pool encrypted?. now back to your question i'm having 2 pools on the os disk because i feel it's easier to backup, recover the system should any upgrade go bad. the local part has an important essential subset backup from the raidz, in other words i'm just using the free space left on the OS disk for extra backups. I was hesitant to use a USB disk OS and maybe i'm wrong. other than that i'm using a small ssd for the zil. Well i think that's all about my zfs experience and thank you for sharing this as I said it's probably the best online .
|
|
||||
Quote:
Quote:
The MAIN problem, is that FreeBSD Bootloader is not able to boot FreeBSD from ZFS which is on encrypted GELI drive, so we have to do it other way. One of the things that beadm does is it changes bootfs property of ZFS pool and vfs.root.mountfrom line in the /boot/loader.conf, so beadm will have to be modified to do that on the separate / or /boot or separate pool. Its not impossible, its just pain in the ass Quote:
IMHO nothing wrong with that.
__________________
religions, worst damnation of mankind "If 386BSD had been available when I started on Linux, Linux would probably never had happened." Linus Torvalds Linux is not UNIX! Face it! It is not an insult. It is fact: GNU is a recursive acronym for “GNU's Not UNIX”. vermaden's: links resources deviantart spreadbsd |
|
|||
Quote:
Here's the implementation detail Code:
# gpart destroy -F ada1 ... # gpart destroy -F ada4 # gpart create -s GPT ada1 ... # gpart create -s GPT ada4 # gpart add -t freebsd-zfs -l zdisk1 -b 2048 -a 4k ada1 ... # gpart add -t freebsd-zfs -l zdisk4 -b 2048 -a 4k ada4 # glabel label -v znas1 /dev/ada1p1 ... # glabel label -v znas4 /dev/ada4p1 # geli init -e aes -l 128 -K /media/boot/keys/znas1.key -b -s 4096 -B /media/backups/znas1.eli.meta /dev/label/znas1 ... # geli init -e aes -l 128 -K /media/boot/keys/znas4.key -b -s 4096 -B /media/backups/znas4.eli.meta /dev/label/znas4 # geli attach -k /media/boot/keys/znas1.key /dev/label/znas1 ... # geli attach -k /media/boot/keys/znas4.key /dev/label/znas4 # zpool create nas raidz /dev/label/znas1.eli /dev/label/znas2.eli /dev/label/znas3.eli /dev/label/znas4.eli # zdb nas | grep ashift # zpool export nas # zpool import nas # zpool status Last edited by silex; 17th December 2012 at 02:14 PM. |
|
||||
Quote:
Quote:
__________________
religions, worst damnation of mankind "If 386BSD had been available when I started on Linux, Linux would probably never had happened." Linus Torvalds Linux is not UNIX! Face it! It is not an insult. It is fact: GNU is a recursive acronym for “GNU's Not UNIX”. vermaden's: links resources deviantart spreadbsd |
|
|||
Vermaden, something extra we forgot to mention on ZFS and disk alignment: I've seen a noticeable improvement when the ZIL on the SSD is properly aligned, in that case i've used Gnop to 4K-align a mounted memory drive then instructed ZFS to mirror log on the SSD with the properly aligned memory drive, I then deleted the MD and the Gnop device yet ZFS keeps a 12 ashift on the log disk and it's what we want. [strike]I'll add the instructions later.[/strike]
ZIL / Log Device /dev/ada5 for proper alignement calculate dd seek with this formula: device media size / 1024000 -1 Code:
# diskinfo -v ada5 #### media size: 128035676160 # echo "128035676160 / 1024000 - 1" | bc #### seek: 125033 # dd if=/dev/zero of=tmpdsk0 bs=1024000 count=1 seek=125033 # mdconfig -a -t vnode -f tmpdsk0 # gnop create -S 4096 md0 # gpart create -s gpt ada5 # gpart add -t freebsd-zfs -l zlog -b 2048 -a 4k ada5 # zpool add znas log mirror md0.nop gpt/zlog # zpool detach znas md0.nop # gnop destroy md0.nop # mdconfig -d -u md0 # zdb znas | grep ashift Code:
# zpool remove znas zlog ps. I'm posting here because this thread is probably one of the most complete ZFS installation instructions found online and it's good to keep it alive. Last edited by silex; 28th December 2012 at 09:26 PM. Reason: ZFS ZIL Correct Alignment Instructions |
|
||||
@silex
I have read some about 4K (ashift=12) align on L2ARC and ZIL devices, post these instructions here as many people will find them useful.
__________________
religions, worst damnation of mankind "If 386BSD had been available when I started on Linux, Linux would probably never had happened." Linus Torvalds Linux is not UNIX! Face it! It is not an insult. It is fact: GNU is a recursive acronym for “GNU's Not UNIX”. vermaden's: links resources deviantart spreadbsd |
|
|||
Done thing, my next step is on enhancing security with two factor authentication using Yubikey but that has more to do with GELI than ZFS per se;
Last edited by silex; 3rd January 2013 at 04:18 PM. Reason: removed off topic stuff |
|
||||
I do not own any of these devices and I do not plan to get one. My Mini-ITX storage box based on mobile Intel Core 2 Duo is more then enough for me.
__________________
religions, worst damnation of mankind "If 386BSD had been available when I started on Linux, Linux would probably never had happened." Linus Torvalds Linux is not UNIX! Face it! It is not an insult. It is fact: GNU is a recursive acronym for “GNU's Not UNIX”. vermaden's: links resources deviantart spreadbsd |
|
|||
EuroBSDCON 2012 presentation on ZFS tuning for database, web and file servers, http://www.youtube.com/watch?v=PIpI7Ub6yjo
|
|
||||
Quote:
Its quite long, so getting it for offline view could be handy: Code:
% cclive -f fmt22_720p "http://www.youtube.com/watch?v=PIpI7Ub6yjo" Checking ... ........ ........ ......done. Tuning ZFS on FreeBSD Martin Matuska EuroBSDcon 2012.mp4 468.43M [video/mp4] [##############-----------------------------------------------------------------------------------------------------------------] 11% 54.9M 1.0M/s 00:06:40
__________________
religions, worst damnation of mankind "If 386BSD had been available when I started on Linux, Linux would probably never had happened." Linus Torvalds Linux is not UNIX! Face it! It is not an insult. It is fact: GNU is a recursive acronym for “GNU's Not UNIX”. vermaden's: links resources deviantart spreadbsd |
|
|||
Two weeks ago I created a Makefile to automate the ZFS setup for 2 disks as described here, but adapted the procedure to align to 4K sectors disks.
A few hours ago I tested this with 9.2-BETA2. Some details ...... Code:
FreeBSD 9.2-BETA2 FreeBSD 9.2-BETA2 #0 r253698: Sat Jul 27 18:22:20 UTC 2013 root@bake.isc.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64 Code:
Code:
Code:
ada1 at ata3 bus 0 scbus1 target 0 lun 0 ada1: <ST2000DM001-1CH164 CC24> ATA-8 SATA 3.x device ada1: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes) ada1: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C) ada1: quirks=0x1<4K> ada1: Previously was known as ad6 ada2 at ata3 bus 0 scbus1 target 1 lun 0 ada2: <ST2000DM001-1CH164 CC24> ATA-8 SATA 3.x device ada2: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes) ada2: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C) ada2: quirks=0x1<4K> ada2: Previously was known as ad7
__________________
You don't need to be a genius to debug a pf.conf firewall ruleset, you just need the guts to run tcpdump Last edited by J65nko; 1st August 2013 at 10:53 PM. |
|
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
FreeBSD GPT howto | graudeejs | Guides | 10 | 21st December 2010 12:24 AM |
HOWTO: FreeBSD CPU Scaling with cpufreq.ko | vermaden | Guides | 10 | 27th October 2010 07:58 AM |
interrupt storm and irq madness | siffland | FreeBSD General | 5 | 23rd October 2009 05:16 AM |
HOWTO: QEMU on FreeBSD | vermaden | Guides | 10 | 9th March 2009 07:10 PM |
HOWTO: FreeBSD with CCACHE | vermaden | Guides | 10 | 9th July 2008 06:14 PM |