DaemonForums  

Go Back   DaemonForums > Miscellaneous > Guides

Guides All Guides and HOWTO's.

Reply
 
Thread Tools Display Modes
  #1   (View Single Post)  
Old 8th February 2009
carpman carpman is offline
Shell Scout
 
Join Date: Jul 2008
Posts: 94
Default Remote FreeBSD server upgrade - Guide!

Hello, ok following my recent experience of updating a remote freebsd server for this first time i have come with following method for future updates/upgrades.

This is a work in progress and will try and keep it upto date from feedback.

This was put together after help from:

TerryP
robbak
DutchDaemon
phoenix

Version 2
Updated 18 Feb 09

If there are any points that still need clarification and if you spot anything incorrect or that could be added i would appreciate it

Those marked with * have info in notes section which you should read.

Code:
1. Turn OFF kern_securelevel=2 in rc.conf

2. Reboot

3. cd /boot

4. cp -Rp kernel kernel.works *

5. Edit stable-supfile file to required version

6. run: csup stable-supfile

7. cd /usr/src

8. less UPDATE - check for important notes

9.  make -j3 buildworld

10.  make -j3 buildkernel KERNCONF=MYKERNEL *

11.  make KODIR=/boot/kernel.new KERNCONF=MYKERNEL installkernel *

12. nextboot -k kernel.new  *

13.  reboot

14. rename /boot/kernel.new to /boot/kernel

15. mergemaster -p *

16. make installworld

17.  mergemaster -viU *

18.  reboot *

19. Turn ON kern_securelevel= in rc.conf

20. reboot

Notes:

1.
I never found any mention of this in docs or guides as they assume you will be in single user mode, if you don't comment out kern_securelevel= and reboot the make installworld will fail.

4.
(name it after the kernconf file name, version number, or something that is easy to remember). This isn't as big a deal when doing remote upgrades as you can't get a loader prompt (unless using a serial console), but it's a good habit to get into and you may get datacenter to boot for you, as you can then you can use
Code:
load /boot/kernel.works/kernel
load /boot/kernel.works/acpi.ko
to load the known-good kernel.

If you use 11. & 12. this is less of an issue.

9. & 10.
The standard recommendation is to use (NUMCPUS + 1). Using too high of a -j setting will slow things down immensely. You judge this on server load, use low setting if doing it when server has high load or higher setting when server not busy, it is case of balancing 'get it done quick with high load on server' or 'take longer but don' load server as much'.
You may want to do a few test runs to find the optimum setting for your systems. Run the following with various -j settings:
Code:
# /usr/bin/time -h make -jX buildworld
11.
This will install the kernel as /boot/kernel.new and leave the working kernel as /boot/kernel. (recommended)

If you wish you can also have entry in make.conf that will build generic and custom kernel but only use custom kernel.

KERNCONF= is a space separated list of kernels to build/install, in the order to build/install them. So you can use things like:

Code:
# make KERNCONF="MYKERNEL GENERIC" buildkernel
# make KERNCONF=GENERIC KODIR=/boot/kernel.generic installkernel
# make KERNCONF=MYKERNEL KODIR=/boot/kernel.custom installkernel
to build the two kernels one after the other, then install them individually. You can also use the following, which will install the two kernels one after the other, which has the effect of making /boot/kernel==GENERIC and /boot/kernel.old==MYKERNEL:
# make KERNCONF="MYKERNEL GENERIC" installkernel

12.
This boots into new kernel, if it all goes foobar up then a remote reboot by datacenter should get you back into old kernel.

13.
To make sure the kernel works. You can run an old world on a new kernel, but you can't run a new world on an old kernel. So if you reboot after installworld and things fail, you may be screwed, requiring the use of a fixit CD.


15 & 17
During mergemaster the question about deleting /tmp/temproot kept coming up and was not sure if i should or not, here is explanation given:
Quote:
Not deleting temproot just saves a little bit of time when you run mergemaster again. If you delete it, mergemaster will simply repopulate it. Usually just delete temproot after mergemaster -p and mergemaster -U/ai/whatever.
18.
This just to make sure all is ok with new kernel and world before setting securelevel.

Single user mode.
In handbook and guides tell you to enter single user mode before doing make installworld, trouble is on remote server this is not possible so you have bypass this step. I am told there is risk but with so many remote servers it does appear to work ok.

Quote:
Single use mode is used to ensure that there is no processes that will mess up as system files change. It is rare, and should never cause problems that won't be fixed by the next restart.
Quote:
It's safest to use single-user mode, as nothing is loaded into memory except the kernel, and no users can login. However, if you manually stop as many running processes as possible (except sshd of course), and know that no users will be logging in, you can run installworld without dropping to single-user mode.
Hope this is of use to others in same boat as me in regards to remote update / upgrade.

Last edited by carpman; 18th February 2009 at 06:23 PM.
Reply With Quote
  #2   (View Single Post)  
Old 8th February 2009
phoenix's Avatar
phoenix phoenix is offline
Risen from the ashes
 
Join Date: May 2008
Posts: 696
Default

Note: please use [ code ] tags and not [ quote ] tags, as using quotes will break replies for people, as the forum doesn't do multi-level quoting in replies.

Code:
1. Turn OFF kern_securelevel=2 in rc.conf

2. Reboot

3. cd /boot

4. tar cf KERNELCONF-NAME.tar kernel
Don't use tar, as you then can't select it from the loader menu/prompt. Instead, use
# cd /boot
# cp -Rp kernel kernel.works
(name it after the kernconf file name, version number, or something that is easy to remember). This isn't as big a deal when doing remote upgrades as you can't get a loader prompt (unless using a serial console), but it's a good habit to get into. Then you can use
load /boot/kernel.works/kernel
load /boot/kernel.works/acpi.ko

to load the known-good kernel.

Code:
5. Edit stable-supfile file to required version

6. run: cvsup stable-supfile
You can uninstall cvsup and all the modula-3 packages. csup comes with the base OS now, and can be used in place of cvsup.

Code:
7. cd /usr/src

8. less UPDATE - check for important notes

9.  make -j12 buildworld

10.  make -j12 buildkernel KERNCONF=MYKERNEL *
-j12 is extreme overkill. The standard recommendation is to use (NUMCPUS + 1). Using too high of a -j setting will slow things down immensely. You may want to do a few test runs to find the optimum setting for your systems. Run the following with various -j settings:
# /usr/bin/time -h make -jX buildworld

Code:
11.  make installkernel
Reboot after installing the new kernel, to make sure the kernel works. You can run an old world on a new kernel, but you can't run a new world on an old kernel. So if you reboot after installworld and things fail, you may be screwed, requiring the use of a fixit CD.

You should also get in the habit of using:
# make KODIR=/boot/kernel.new KERNCONF=whatever installkernel

This will install the kernel as /boot/kernel.new and leave the working kernel as /boot/kernel. Then you can use:
# nextboot -k kernel.new

to boot using the new kernel for the next reboot only. If things fail, just reboot again, and you are back to using the old kernel. If the boot succeeds, then you can just rename /boot/kernel.new to /boot/kernel.

Code:
12. mergemaster -p *

13. make installworld

14.  mergemaster -viU *
You should do a reboot here as well, to make sure the system will come up properly when using both the new kernel and the new world. Before setting securelevel. Just to be safe.

Quote:
11. I use custom kernel so need to declare what kernel, i have read somewhere that you can have entry in make.conf that will build generic and custom kernel but only use custom kernel, but have not been able to find it?
KERNCONF= is a space separated list of kernels to build/install, in the order to build/install them. So you can use things like:
# make KERNCONF="MYKERNEL GENERIC" buildkernel
# make KERNCONF=GENERIC KODIR=/boot/kernel.generic installkernel
# make KERNCONF=MYKERNEL KODIR=/boot/kernel.custom installkernel

to build the two kernels one after the other, then install them individually. You can also use the following, which will install the two kernel one after the other, which has the effect of making /boot/kernel==GENERIC and /boot/kernel.old==MYKERNEL:
# make KERNCONF="MYKERNEL GENERIC" installkernel

Quote:
12 & 14
During mergemaster the question about deleting /tmp/temproot kept coming up and was not sure if i should or not, here is explanation given:
Delete it. Unless you are running mergemaster on a daily basis, there's no point wasting the space to save a few extra seconds.

Quote:
Single user mode.
In handbook and guides tell you to enter single user mode before doing make installworld, trouble is on remote server this is not possible so you have bypass this step. I am told there is risk but with so many remote servers it does appear to work ok.
It's safest to use single-user mode, as nothing is loaded into memory except the kernel, and no users can login. However, if you manually stop as many running processes as possible (except sshd of course), and know that no users will be logging in, you can run installworld without dropping to single-user mode.
__________________
Freddie

Help for FreeBSD: Handbook, FAQ, man pages, mailing lists.
Reply With Quote
  #3   (View Single Post)  
Old 9th February 2009
cajunman4life cajunman4life is offline
Real Name: Aaron Graves
Package Pilot
 
Join Date: May 2008
Location: Coolidge, Arizona
Posts: 203
Default

A serial connection may come in handy too. Most colo shops can offer you a remote console connection, which would make single user mode possible.
__________________
I just saved a bunch of money on my car insurance by fleeing the scene of the accident!
Reply With Quote
  #4   (View Single Post)  
Old 9th February 2009
carpman carpman is offline
Shell Scout
 
Join Date: Jul 2008
Posts: 94
Default

Freddie, many thanks for your informative post, i will edit my OP to reflect it when have time.
Reply With Quote
  #5   (View Single Post)  
Old 18th February 2009
carpman carpman is offline
Shell Scout
 
Join Date: Jul 2008
Posts: 94
Default

Updated guide, if find anything incorrect or could be better let me know.
Reply With Quote
  #6   (View Single Post)  
Old 19th February 2009
vermaden's Avatar
vermaden vermaden is offline
Administrator
 
Join Date: Apr 2008
Location: pl_PL.lodz
Posts: 1,056
Default

Quote:
3. cd /boot
4. cp -Rp kernel kernel.works *
Dunno it that is necessery, you do installkernel with KODIR option, so original /boot/kernel will stay untouched, and even if you forgot to add KODIR option, it will be automatically renamed to /boot/kernel.old while new one will be placed as /boot/kernel

Quote:
10. make -j3 buildkernel KERNCONF=MYKERNEL *
11. make KODIR=/boot/kernel.new KERNCONF=MYKERNEL installkernel *
These tow can be merged into one:
10. make -j3 kernel KERNCONF=MYKERNEL KODIR=/boot/kernel.new
__________________
religions, worst damnation of mankind
"If 386BSD had been available when I started on Linux, Linux would probably never had happened." Linus Torvalds

Linux is not UNIX! Face it! It is not an insult. It is fact: GNU is a recursive acronym for “GNU's Not UNIX”.
vermaden's: links resources deviantart spreadbsd
Reply With Quote
  #7   (View Single Post)  
Old 19th February 2009
Carpetsmoker's Avatar
Carpetsmoker Carpetsmoker is offline
Real Name: Martin
Tcpdump Spy
 
Join Date: Apr 2008
Location: Netherlands
Posts: 2,243
Default

As a sidenote, make -jn kernel often fails for me, while make -j[i]n[/n] always works. I experienced this on several machines.
__________________
UNIX was not designed to stop you from doing stupid things, because that would also stop you from doing clever things.
Reply With Quote
  #8   (View Single Post)  
Old 5th April 2009
ohauer ohauer is offline
Port Guard
 
Join Date: May 2008
Location: germany
Posts: 32
Default

Maybe you find this way dangerous, but I update my 25+ FreeBSD machines mostly this way.
With a little work before build new ports ... at another machine update takes normaly including ports ~25min

Before you try this with a machine on the other side of the world test it with a spare machine or a VM


1. generate a kernel template that works on 'every machine' regardless if I can get a little more speed at machine X if module Z is removed.
To have a better overview betweeen the kernel versions for example 7.0 and 7.1 I configure my kernels this way.
Include GENERIC and unconfigure parts I don't need with the no prefix

Code:
#!/bin/sh
#==========================================================
# use this script to generate a new kernel config
# usage:
#  $> mkdir SAVE_PLACE/kernconf
#  $> cd /usr/src/sys/${arch}/conf
#  $> mk_kern_template.sh > SAVE_PLACE/kernconf/MYKERNEL
#  $> ln -s SAVE_PLACE/kernconf/MYKERNEL MYKERNEL
#  $> cd /usr/src/
#  $> make buildkernel KERNCONF=MYKERNEL
#  $> make installkernel KERNCONF=MYKERNEL KODIR=/boot/MYKERNEL
#
# in MYKERNEL: include the following line before the cpu options
#
#       include GENERIC             <- include the GENERIC kernel config,
#                                      allready done by this script, but check
#                                      for duplicates
#       # nocpu         I486_CPU
#       # nocpu         I586_CPU
#       # nocpu         I686_CPU
#       ident           MYKERNEL    <- replace MYKERNEL with your kernel name
#
#  Now the easy part, uncomment the nocpu, nooptions, nodevice you don't want
#  to have in the kernel.
#  Keep this file at a save place, and you can easily create a diff between
#  FreeBSD releases to adjust your kernel config
#
#  2006-08-07 olli hauer
#==========================================================


REL=$(sysctl -n kern.osrelease)
cat << _EOF
#==========================================================
#  \$Id\$
#
#  Kernel Config for FreeBSD ${REL}
#    This config overrides the default settings from GENERIC
#    with the 'no' prefix, so we can track differences of
#    the kernel config between FreeBSD versions easier
#
include GENERIC

_EOF
Now I build this kernel at a fast machine, we will rsync it later to the remote machines


2. collect all the files we need for the OS update
Code:
#> mdconfig -a -t vnode -f ./7.1-RELEASE-i386-disc1.iso
md0
#> mount_cd9660 /dev/md0 /cdrom/
#> mkdir /space/7.1-RELEASE
#> rsync -a --exclude=install.sh /cdrom/7.1-RELEASE/base /space/7.1-RELEASE/
#> rsync -a /cdrom/7.1-RELEASE/src /space/7.1-RELEASE/
#> rsync -a /cdrom/7.1-RELEASE/manpages /space/7.1-RELEASE/
#> umount /cdrom
#> mdconfig -d -u 0
Note:
exclude install.sh from base we will replace it with this script to exclude the /etc directory
-> save it as update.sh in the /space/7.1-RELEASE/base directory
Code:
#!/bin/sh
#
# $FreeBSD: src/release/scripts/base-install.sh,v 1.6.32.1 2008/11/25 02:59:29 kensmith Exp $
#
# This is a modified script to update quickly a already installed system
# but excludes the /etc directory.
# to update /etc extract first the sources to /usr/src
# and do first a mergemaster -p
# then a mergemaster

if [ "`id -u`" != "0" ]; then
        echo "Sorry, this must be done as root."
        exit 1
fi

#DESTDIR=/backup/testupdate
DESTDIR=/

# remove schg flags from files and dircetories
SCHGDIRS="bin lib libexec sbin usr/bin usr/lib usr/libexec usr/sbin var/empty"

echo "You are about to extract the base distribution into ${DESTDIR:-/} - are you SURE"
echo ""
echo "*******************************************"
echo " host: $(hostname) "
echo "*******************************************"
echo ""
echo -n "you want to do this over your installed system (y/n)? "
read ans
if [ "$ans" = "y" ]; then
    echo "remove schg flag from"
    for DIR in ${SCHGDIRS}; do
        find /${DIR}/ -flags schg -maxdepth 1 | xargs chflags -v noschg
    done
    cat base.?? | tar --unlink -xpzf - --exclude ./etc -C ${DESTDIR:-/}
fi
3. Now it is time to rsync all the parts wee need to the remote machines
Code:
#> rsync -a /boot/kernel/ remote1:/boot/kernel.71/
#> rsync -a /space/7.1-RELEASE/ remote1:/space/7.1-RELEASE/
4. if you like to save some infos from the remote machine before you continue
Code:
#> mkdir -p /space/upgrade_save/remote1/local
#> cd /space/upgrade_save/remote1/
#> rsync -a remote1:/etc .
#> rsync -a remote1:/usr/local/etc local/
#> ssh remote1 pkg_info -qoa > pkg_info.remote1
5. Update the remote machine
!Attention! shutdown -k kicks off everyone except root, do not use if you logged in remote via other account and the su!

Code:
#> ssh remote1
#> shutdown -k now "system update in progress"
#> for i in `find /usr/local/etc/rc.d/ -type f`; do $i stop; done
#> mv /usr/src /usr/src.old
#> cd /space/7.1-RELEASE/src/ && ./install.sh
#> cd /usr/src/
#> mergemaster -p
#> cd /space/7.1-RELEASE/manpages/ && ./install.sh
#> nextboot -k kernel.71
#> cd /space/7.1-RELEASE/base/ && ./update.sh
6. reboot the machine (cross the fingers

7. login again to the machine
Code:
#> shutdown -k now "system update in progress"
#> cd /boot
#> mv kernel kernel.oldrelease
#> mv kernel.71 kernel
#> cd /usr/src
#> mergemaster -i
#> rm /var/run/nologin
#> shutdown -r now
8. system update with freebsd-update
Code:
#> mv /var/db/freebsd-update /var/db/freebsd-update.oldrelease
#> freebsd-update fetch
#> freebsd-update install
9. do some checks for your specific setup ...
Maybe you want to for sshd the new 1024 bit keys instead the old 768 bit then it is time to throw the old one away and restart sshd.
To cleanup old libs...
Hint:
A line with the following 'WITHOUT_KERBEROS=true' will remove also all kerberos parts from the system, so to be save rename /etc/make.conf
#> cd /usr/src
#> make clean-old


If you want to replace all ports with fresh ports build at another machine
# remote machine
#> mv /usr/ports/packages/All /usr/ports/packages/.All.oldrelease
#> pkg_info -qoa | sort > /usr/ports/packages/.pkg_info.oldrelease

# from build machine
#> rsync -av /space/prebuildpackages/All remote1:/usr/ports/packages/

# remote machine
#> pkg_delete -a
#> cd /usr/ports/packages/All/
#> pkg_install ./packagename

to see what we are mybee missing
#> pkg_info -qoa | sort > /usr/ports/packages/.pkg_info.new
#> diff -u /usr/ports/packages/.pkg_info.oldrelease /usr/ports/packages/.pkg_info.new
Reply With Quote
  #9   (View Single Post)  
Old 5th April 2009
TerryP's Avatar
TerryP TerryP is offline
Arp Constable
 
Join Date: May 2008
Location: USofA
Posts: 1,547
Default

I've used the -j switch to make for years without any problems building world or kernel. Usually my desktop does a -j12 on a dual cored Pentium D; on my slower single CPU systems, usually -j4. If work loads are heavier then my abusive multi-tasking, I reduce the jobs (e.g. -j8). The only time I've ever gotten a bad kernel out of the deal is with a bad config, and I have never had any built kernel fail to boot and run stably.


Whenever I've forgotten to specify the -j switches like above, the builds take much longer to complete. Making a FreeBSD 7.0 release kernel under -j12 takes about 10-15 min on the main box here, without it, the kernel takes closer to 45 minutes to build.


of course, YMMV
__________________
My Journal

Thou shalt check the array bounds of all strings (indeed, all arrays), for surely where thou typest ``foo'' someone someday shall type ``supercalifragilisticexpialidocious''.
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Remote backups server using FreeBSD, ZFS, and Rsync phoenix Guides 1 5th March 2010 12:17 AM
upgrade xorg on freebsd ccc FreeBSD Ports and Packages 5 1st February 2009 04:17 PM
freebsd 7.1 upgrade buildworld error map7 FreeBSD Installation and Upgrading 9 30th October 2008 06:54 PM
A Comprehensive Guide to FreeBSD ijk Book reviews 1 29th July 2008 03:53 PM
Remote Access to File Server Oko OpenBSD Security 7 23rd June 2008 05:17 PM


All times are GMT. The time now is 02:05 AM.


Powered by vBulletin® Version 3.8.4
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Content copyright © 2007-2010, the authors
Daemon image copyright ©1988, Marshall Kirk McKusick