Everything you want to know about Installing FreeBSD on a USB stick

[FBSD]

Everything you want to know about
Installing FreeBSD on a USB stick.
Release 8.0

In Release 8.0 the sysinstall process had a major overhaul as well as the kernels USB handling. So this document assumes the reader will be using a 8.0 system when trying things described here.

To get everyone on the same page with terminology USB memory stick, flash drive, key, stick, and pen all mean the same thing. I will be using USB stick in this article.

Flash Memory Technology:

The memory chip, with the controller chip, is the most important element of a memory card. The memory chip is based on the flash memory technology which is solid-state, non-volatile and rewritable. Solid-State means it contains no moving parts and therefore is immune to mechanical failures and damages from movements and vibrations. It also operates completely silent with a zero decibel noise level. Non-Volatile flash memory stores bits of information in memory cells made of silicon wafers which do not require power to retain information when power is turned off.

In general flash memory functions like RAM memory and a hard disk drive combined. It stores digital data in memory cells like the RAM memory, and stores information like a hard disk drive when the power is turned off. In comparison to other storage media flash memory offers superior features such small form factor, high degree of durability, high degree of reliability, low power consumption and high transfer speed. Based on that flash memory technology is ideal for use in USB sticks. The only disadvantage is the manufacturing cost, which is higher compared to hard disk drives, CDs and DVDs.

There are two different technologies of flash memory, NOR and NAND. NAND flash memory is ideal for memory cards because is less expensive and can accommodate more storage capacity in the same die size. Memory card manufactures are using different NAND technologies for either boasting the memory card’s performance or for decreasing the memory card’s manufacturing costs. The most common flash memory technologies are the Single-Level Cell, Multi-Level Cell, Multi-Bit Cell and Chip Stacking. The Single-Level Cell technology is used to boast the memory card’s performance while the rest are used for decreasing manufacturing costs. The most used technology of them all is the Multi-Level Cell.

A Single-Level Cell, SLC, memory card stores one bit in each cell, leading to faster transfer speeds, lower power consumption and higher cell endurance. The only disadvantage of Single-Level Cell is the manufacturing cost per MB. Based on that, the SLC flash technology is used in high-performance memory cards.

A Multi-Level Cell, MLC, memory card stores three or more bits in each cell. By storing more bits per cell, a Multi-Level Cell memory card will achieve slower transfer speeds, higher power consumption and lower cell endurance than a Single-Level Cell memory card. The advantage of Multi-Level Cell memory card is the lower manufacturing costs. The MLC flash technology is used mostly in standard memory cards. The Multi-Bit Cell, MBC, is a similar technology to the Multi-Level Cell but stores only two bits per cell.

Chip stacking technology is used by many manufactures to double the memory card’s capacity at considerable lower manufacturing costs. This is achieved by putting two chips together to form a single chip. For example, by stacking two 256 MB chips together they will form a single 512 MB chip. This technology is far less expensive alternative to the single-die chips or even called monolithic chips.

USB sticks are most commonly manufactured using MLC (multi-level cell) or SLC (single-level cell) technology. MLC is a lot less expensive to manufacture, but is a lot slower than SLC. MLC have ‘write’ lifetimes in the 10,000 to 100,000 range.

SLC is a bit more expensive, but is very fast, and has ‘write’ lifetimes in the 100,000-millions range.

Which chip used in the manufacture of the USB stick is the most important thing that affects throughput and longevity.

What really sucks, is how hard it is to find out which type of chips are used in the USB stick before purchasing it. Some USB sticks come packaged with the vendors website listed. Check out their website looking for email address for tech support or sales dept. Email them and ask what chip type is used in the model of USB stick you are interested in purchasing.

Another point of interest is does your PC BIOS have option to boot from USB stick or even if the BIOS will recognize the USB stick media as bootable? This is a motherboard BIOS problem not un-common for PC’s manufactured before 2008. Testing different vendor USB sticks on your PC maybe the only way to find one that will boot. Size as 2GB, 4GB, 8GB, what ever has no baring on this BIOS problem. Using a USB stick for storage only has never been a problem.

Installing FreeBSD on a USB stick to create a bootable USB stick is a lot of work just to test if the USB stick is recognized by your PC BIOS. A quick way is to dd an floppy image to your USB stick and try booting that.
Here are some commands that will help

dd if=/dev/da0 count=2 | od –c          # to display the USB stick MBR
dd if=/dev/zero of=/dev/da0 count=2   # to zero out the USB stick MBR
dd if=/path/floppy.img of=/dev/da0      # to write the floppy bootable image

The download of floppy.img is located at the end of this article.

Another way to do the same thing.
Use this windows image writer to write the bootable floppy.img file to the USB stick. You can download your own copy from
https://launchpad.net/win32-image-writer/+download

The objective is to purchase a USB stick containing SLC (single-level chip) then you have no need to take special effort to limit writes like you would have to using a USB stick containing MLC (multi-level chip).




Types of USB stick Installs.


1. Putting the disc1.iso on a USB stick so it can be used to install FreeBSD on a target in the same way it’s commonly done from the disc1.iso burned to CD. In this usage the USB stick using MLC or SLC makes no difference as after the original writes to population it, there are only reads from that point on. Assuming you don't have any SCSI disks occupying the dax device range or USB external hard drive your USB stick will show up on /dev/da0
You can download this script from the end of this article.

The following script will do this for you.

#!/bin/sh
# Purpose = Use to transfer the FreeBSD install disc1.iso files to 
# a bootable USB stick drive so it can be used to install from. 
# First fetch the FreeBSD 8.0-RELEASE-i386-disc1.iso to your
# hard drive /usr. Then execute this script from the command line
# fbsdiso2usb 
 
# NOTE: This script has to be run from root and your USB stick drive
# has to be plugged in before running this script. 
 
echo ' '
echo '****** Prepare disc1.iso for usage'
echo ' '
cd /usr
mkdir dis
mdconfig -a -f /usr/8.0-RELEASE-i386-disc1.iso md0
mount -v -t cd9660 /dev/md0 /usr/dis
echo ' '
echo '****** Prepare target usb stick'
echo ' '
dd if=/dev/zero of=/dev/da0 count=2
fdisk -vBI /dev/da0
bsdlabel -B -w da0s1
newfs -O 1 /dev/da0s1a
mount -v /dev/da0s1a /mnt
echo ' '
echo ' '
echo '****** Copy all the disc1.iso files onto the usb stick'
cd /usr/dis
find . -print -depth | cpio -dump /mnt
echo ' '
echo 'Finished do clean up now'
cd /usr
umount -v /mnt
umount -v /usr/dis
mdconfig -d -u md0
rmdir dis 
echo ' '
echo ' '
echo "### Script finished ###"

2. With Release 8.0 an new USB stick livefs/sysinstall image is released. This memstick.img file is three times larger than the disc1.iso. It’s intended you dd this image to your USB stick. You can use it in fixit mode (it’s a complete running system) or in sysinstall mode. In this usage the USB stick using MLC or SLC makes no difference as after the original writes to population it, there are only reads from that point on. Assuming you don't have any SCSI disks occupying the dax device range or USB external hard drive your USB stick will show up on /dev/da0
Use this command # dd if=8.0-RELEASE-i386-memstick.img of=/dev/da0 bs=10240
To write the memstick.img to your USB stick.



3. Installing FreeBSD on a USB stick containing SLC (single-level chip) from a CD burned from the disc1.iso results in a complete base release system which you can configure the way you like and install ports depending on the GB size of your USB stick. Assuming you don't have any SCSI disks occupying the dax device range or USB external hard drive your USB stick will show up on /dev/da0



4. Installing FreeBSD on a USB stick containing SLC (single-level chip) from another USB stick created by method 1 or 2 above, results in a complete base release system which you can configure the way you like and install ports depending on the GB size of your USB stick. Assuming you don't have any SCSI disks occupying the dax device range or USB external hard drive. You do have a USB stick containing the sysinstall world and a USB stick desired to be the target both plugged in before booting. In sysinstall you have option to select the source of the install files and later on what is the device to be used as the target.

The secret is you have to know which device points to the USB stick containing the sysinstall World and which device points to the target USB stick. Sysinstall will allow you to fdisk the device pointing to the USB stick containing the sysinstall world when you really want the target device. So be very careful and pay close attention to what device you tell sysinstall to use.

One way to know them apart is for the target USB stick to be larger in size. That way in fdisk you will see the size shown on top of the screen.

Another way is to only plug in the USB stick containing the sysinstall World and boot. It will be assigned /dev/da0. When you get to the sysinstall main menu. Plug in the target USB stick and choose the standard install/options item. Use the arrow keys to highlight the "re-scan devices (*)" item and hit the space bar to do the re-scan. The re-scan is so fast you will not notice anything. The target will be assigned /dev/da1. Now you should have no problems selecting the correct device of the source install files and later on for the device to be used as the target.

Take note: Since the target is assigned da1 that will also be what is auto coded in the /etc/fstab file. After the sysinstall is complete and before rebooting you have to manually edit /etc/fstab file and change all references from da1 to da0.

This can be done while the sysinstall main menu is showing. Hit the ALT key and the F4 key at the same time to open the shell console. Issue

ee /etc/fstab and make your changes.

Then ALT key and the F1 key at the same time to return to the sysinstall main menu. Select exit and yes to reboot. Unplug the USB stick containing the sysinstall World and the reboot will boot your new target stick.

If you try to boot the target USB stick without making the /etc/fstab changes, You will be in a world of hurt. The system will not boot because the target USB stick is now the only one plugged in and will be assigned da0 while it's fstab saying its da1.



5. Custom install of FreeBSD on USB stick containing MLC (multi-level chip). Using this type of USB stick, effort is taken to minimize the amount of writes done to the USB stick while running the operating system from it. This same procedure can also be used on a USB stick containing SLC (single-level chip). Assuming you don't have any SCSI disks occupying the dax device range or USB external hard drive. The install source can come from the disc1.iso file or the source can be ftp downloaded. This custom install procedure uses GEOM disk labels so we don't care where the USB stick appears in the device tree. This custom install will be a (minimal install) just the basic running system, no doc, no man, no info, no root password, no nothing else.

There are two ways to obtain the necessary source files. You can download the disc1.iso file which gives you everything, or just ftp the base and kernel files which is all that is needed for this minimal install. Both ways are explained.





Obtaining the disc1.iso

Here is a script that will download the sysinstall world disc1.iso. You can download this script from the end of this article.

#! /bin/sh
echo ' '
echo 'There is not enough disk space on the FBSD slice to hold the'
echo 'disc1.iso-image of the new FBSD version.'
echo 'cd /usr before starting this script' 
echo ' '
 
path="pub/FreeBSD/releases/i386/ISO-IMAGES"
name="8.0/8.0-RELEASE-i386-disc1.iso"
name2="8.0/CHECKSUM.MD5"
 
fetch -avrpAFU ftp://ftp.FreeBSD.org/$path/$name
fetch -avrpAFU ftp://ftp.FreeBSD.org/$path/$name2
 
echo ' '
echo ' '
echo 'Run these steps to verify download is good'
echo 'ls -l to verify file sizes'
echo 'md5 8.0-RELEASE-i386-disc1.iso >> CHECKSUM.MD5' 
echo ' to create value & append to end of file' 
echo 'ee CHECKSUM.MD5 last line = download value'
echo 'check it to disc1 value in the CHECKSUM.MD5' 
echo 'if they do not match download again'
echo ' '
echo 'Script completed.'

Obtaining the release files using FTP.

Here is how to use FTP to download only the base and kernel files needed.
Put the follow code into /root/.netrc that is (DOTnetrc)
You can download this script from the end of this article.

machine ftp2.jp.FreeBSD.org
login anonymous
password FBSD@home.com
macdef init
prompt off
cd /pub/FreeBSD/releases/i386/8.0-RELEASE
epsv4 off
#mreget ERRATA.HTM ERRATA.TXT HARDWARE.HTM HARDWARE.TXT README.HTM
#mreget README.TXT RELNOTES.HTM RELNOTES.TXT cdrom.inf docbook.css
#$ getdir base catpages dict doc games info kernels manpages ports proflibs src
$ getdir base kernels
quit
 
macdef getdir
! mkdir $i
mreget $i/*

Issue this command on the user root console command line for the first execution and all restarts.

#cd /usr
#mkdir dist
#cd dist
#ftp -v ftp2.jp.FreeBSD.org

Executing ftp while in directory /usr/dist will result in ftp putting the downloaded files in that Directory. When accessing one of the FreeBSD FTP servers it is very common to get timed out before downloading of all the files is completed. The mreget command will cause FTP to check the local host for the file and only download it if it's missing from the local host or it's date or size is different. This way you can use the same .netrc for the first execution and to restart downloading files where your task was forced to end because the remote host timed you out.

I am using the mirror ftp2.jp.FreeBSD.org site. I tried a few other mirror FTP sites and was timeout after 3 to 5 minutes. This mirror has very little traffic and I got 80% done before timing out. Only one restart was necessary to complete the download. Recommend you try different mirrors located near your geographical location until you find one with low traffic.

FTP defaults to using extended passive mode. I could not get FTP to function through my ipfilter firewall until I forced FTP to use native passive mode by adding the "epsv4 off" which turns off extended passive mode.

You can see the commented out statements in the .netrc file. These statements would download the complete release source. Basically the same stuff contained on the disc1.iso




Custom process description.

You can download this as a script from the end of this article.

If using the disc.iso as install source do this.
Make a mount point, put the disc1-iso into a memory disk
and then mount it in cdrom format.

# mkdir /dist
# mdconfig -a -f /usr/8.0-RELEASE-i386-disc1.iso md0
# mount -t cd9660 /dev/md0 /dist

Plug the TARGET USB stick in.
They come preformatted from the manufacture with a FAT32 partition on it.
This command will destroy all existing data on the USB stick.

Zero out the MBR

#dd if=/dev/zero of=/dev/da0 count=2

Do fdisk with a new MBR.

# fdisk -BI /dev/da0

The –B means Reinitialize the boot code contained is sector 0 of the disk.
******Default from /boot/mbr
The ‘I’ means initialize sector 0 slice table for one slice covering the entire disk. You will get a message saying ‘Class not found’ disregard it.

Label the USB stick:

# bsdlabel -B -w da0s1

The –B means bootstrap code will be read from the file /boot/boot and written to the disk. The –w means write a standard label.

Allocate the file system.
In order to reduce the number of writes to the USB stick, and as common practice, use the -U flag to enable soft updates. Additionally, so that we can find the filesystem easily no matter where the USB stick appears in the device tree, we will label the filesystem as FBSDonUSB:

# newfs -U -L FBSDonUSB /dev/da0s1a

Mount the USB stick

# mount /dev/da0s1a /mnt

Install the base files the same way sysinstall does by running the same script

# cd /dist/8.0-RELEASE/base  # This command if using the disc1.iso
# cd /usr/dist/base               # This command if using the Ftp source
# env –iv DESTDIR=/mnt ./install.sh

The following prompt is shown to you and you have to enter y
You are about to extract the base distribution into /mnt - are you SURE
you want to do this over your installed system (y/n)? y

Install the generic kernel

# cd /8.0-RELEASE /kernels   # This command if using the disc1.iso
# cd /usr/dist/kernels           # This command if using the Ftp source
# env –iv DESTDIR=/mnt ./install.sh generic
# rmdir /mnt/boot/kernel
# mv /mnt/boot/GENERIC /mnt/boot/kernel

Putting the base and the kernel on the USB stick does not create a fstab file. Create an /etc/fstab file on the USB stick. This one puts the logs on to memory storage (to minimize writes). We also null mount /var/tmp on /tmp, which makes it non-persistent:

# cat >> /etc/fstab << EOF
Device           Mountpoint   FStype   Options                 Dump Pass#
/dev/ufs/FBSDonUSB  /           ufs      rw,noatime               1    1
md                  /tmp        mfs      rw,-s16M,nosuid,noatime  0    0
md                  /var/run    mfs      rw,-s4M,nosuid,noatime   0    0
md                  /var/log    mfs      rw,-s16M,nosuid,noatime  0    0
/tmp                /var/tmp    nullfs   rw                       0    0
/dev/acd0           /cdrom      cd9660   ro,noauto,nosuid         0    0
EOF

Since we're using the UFS label to define the root filesystem,
we must force the GEOM label class to be loaded early:

# cat >> /mnt/boot/loader.conf << EOF
geom_label_load="YES"
EOF

The vi edit program needs a /var/tmp/vi.recover file.

# mkdir -p /mnt/usr/local/etc/rc.d/
# cd /mnt/usr/local/etc/rc.d/
# cat >> mkvirecover << EOF
#!/bin/sh
# PROVIDE: mkvirecover
# REQUIRE: mountcritremote
# BEFORE: DAEMON virecover
 
. /etc/rc.subr
 
name="mkvirecover"
stop_cmd=":"
start_cmd="mkvirecover_start"
 
mkvirecover_start()
{
[ -d /var/tmp/vi.recover ] || mkdir -m 1777 /var/tmp/vi.recover
echo '.'
}
 
load_rc_config $name
run_rc_command "$1"
EOF 
 
# chmod 555 mkvirecover

/var/log is now on memory disk. The newsyslog.conf files needs adjustment. All logs now have count of one, they rotate when their size fills 100KB. And the log file should be created if it does not already exit.

# cat >> /etc/newsyslog.conf << EOF
# logfilename         mode count size when flags 
/var/log/all.log      600    1   100   *   JC
/var/log/amd.log      644    1   100   *   JC
/var/log/auth.log     600    1   100   *   JC
/var/log/console.log  600    1   100   *   JC
/var/log/cron         600    1   100   *   JC
/var/log/daily.log    640    1   100   *   JNC
/var/log/debug.log    600    1   100   *   JC
/var/log/kerberos.log 600    1   100   *   JC
/var/log/lpd-errs     644    1   100   *   JC
/var/log/maillog      640    1   100   *   JC
/var/log/messages     644    1   100   *   JC
/var/log/monthly.log  640    1   100   *   JNC
/var/log/security     600    1   100   *   JC
/var/log/sendmail.st  640    1   100   *   BJC
/var/log/weekly.log   640    1   100   *   JNC
/var/log/wtmp         644    1   10    *   BC
/var/log/xferlog      600    1   100   *   JC
EOF

Set the interfaces to configure themselves over DHCP.

# cat >> /etc/rc.conf << EOF
ifconfig_DEFAULT="DHCP"
EOF

Turn off rebuilding locate database:

# cat >> /etc/periodic.conf << EOF
weekly_locate_enable="NO"
weekly_whatis_enable="NO"
EOF

Set the command line prompt format for root

#echo ‘set prompt = “# %/ >”’ >> /mnt/root/.cshrc

End do clean up.

#cd /root
#umount -v /mnt
#umount –v /usr/dis 
#mdconfig –d –u md0
#rm –vrf /usr/dis

6. The mfsBSD remote installation of the FreeBSD operating system when the console
of the remote system is unavailable. This article documents its purpose and use.
http://www.freebsd.org/doc/en/articl...all/intro.html

This Paper on mfsBSD was presented by Martin Matuska at the EuroBSDCon 2009
conference held September 18-20 2009 in Cambridge, UK.
http://people.freebsd.org/~mm/mfsbsd/mfsbsd.pdf

The mfsbsd product can be downloaded from http://mfsbsd.vx.sk/ It's a set of
custom scripts that generates a bootable image (and/or ISO file), that creates
a minimal installation of FreeBSD that is completely loaded into memory.
Documentation on what is happening internally is not available. This mfsBSD
project is based on the ideas from the depenguinator project found here.
http://www.daemonology.net/depenguinator/




All the scripts discussed can be downloaded from here
All downloaded files have .zip suffix.
Do not try to open, just do save.
Just remove the suffix to use.