In other words, upgrading a FreeBSD box the dirty way.

This is very much a “works on my machine” approach but it should get you 90% of the way to dealing with your own. Expect annoying breakages and minor fixes along the way.

The scenario is that we have, for some reason, a system that can’t be upgraded the normal way - perhaps it’s FreeBSD 13.0-BETA1 where upgrades were broken, or you are planning to side-load a RELEASE version while your main system still runs CURRENT.

Instead of running the installer like a sensible person, we’ll use the bectl(8) features to provide a Boot Environment, and do a dirty upgrade inside that, out of the way of the current system.

This means we can re-use any zfs datasets across both “installs”, such as zroot/usr/home and zroot/var/db/... and FreeBSD’s zfs dataset layout will just Do The Right Thing in most cases. Brilliant!

There are a few tricks and traps, but the main one is that /etc and similar config dirs will not quite line up between the current setup and whatever we are sideloading.

scenario

Like any decent FreeBSD developer, I am eating my own delicious dog-food, and running FreeBSD current - today it’s 14.0-CURRENT. But I also need to test out changes for release versions of FreeBSD, so here the version I want to side-load is 13.0-BETA1 although by the time you read this, it will be BETA4 and soon RC1. It’s hard to keep up.

things we want to keep

• /boot/loader.conf(8)
• anything in our EFI partition
• /etc/passwd, /etc/group and related files
• /etc/shells
• /etc/rc.conf
• /var/unbound this is softlinked in /etc/unbound but because local-unbound(8) runs in a chroot, it can’t be present in /etc
• missing /etc/fstab mountpoints that aren’t part of default install
• a list of leaf packages via pkg prime-origins > /etc/packages.list
• a handful of other similar files
• /home softlink

stash our configs

I use git to keep track of any local changes to /etc and similar dirs. This isn’t necessary but you end up with more fine-grained history than just zfs snapshots, and you’ll see that later on, when we want to “merge in” the changes in /etc/rc.d and similar files, it makes this process very very easy.

export NOW=`date -u +%Y%m%d-%H%M`
export PAGER="/bin/cat -bu"
export RELEASE=`freebsd-version -ku | sort -r |head -1`-update
umask 027
cd /etc
test -d .git || git init .
git add -A
git commit --allow-empty -am ${RELEASE}
zfs snapshot -r zroot@${NOW}:${RELEASE}

download your txz files

• base.txz
• kernel.txz
• src.txz and -dbg as required

create a new boot environment

Normally, you’d use bectl(8) to create a new boot environment, and it would do so using another boot environment to reference as a snapshot, so you’re not starting from an empty system. In our case, we absolutely do not want to inherit all the garbage from the other BE, so we will make a new zfs dataset, with the correct mount-related settings, and have it beautifully pristine and empty:

zfs create -o canmount=noauto -o mountpoint=/ zroot/ROOT/vanilla
bectl mount vanilla /mnt

The key settings here are:

• canmount=noauto as the FreeBSD loader will mount only the active BE at startup. If you don’t set this, then zfs will, once you’re past the initial boot loader stage, happily load your different kernel/userland from a different boot environment over the top of your expected one. This is extremely frustrating to realise, and then fix!
• mountpoint=/ clearly the root filesystem needs to be mounted at / and we must not inherit some other mountpoint from elsewhere

unpack your tarballs

This is part of the dirty work - we need to exclude some of /etc as we’re definitely wanting to keep our existing users and passwords, but the rest of the /etc dir such as /etc/rc.d we definitely want the side-loaded version to be present.

So we exclude the critical bits, and then assume that our existing setup’s files are compatible. This is usually true, but not always, and if you’re reading this, it’s reasonable to expect you’re willing to clean your own mess up when something breaks - it’s always just a boot environment away!

# cd /path/to/your/stash
tar xvzpf ./kernel-dbg.txz -C /mnt/
tar xvzpf ./kernel.txz -C /mnt/
tar xvzpf ./base-dbg.txz -C /mnt/
tar xvzpf ./base.txz -C /mnt/ \
  --clear-nochange-fflags \
  --exclude crontab \
  --exclude dhclient.conf \
  --exclude group \
  --exclude pwd.db \
  --exclude spwd.db \
  --exclude hosts \
  --exclude master.passwd \
  --exclude passwd \
  --exclude shells \
  --exclude sshd_config \
  --exclude sysctl.conf
zfs snapshot -r zroot/ROOT/vanilla@tarballs

move in /etc and friends

Anything in /etc and anything that is a softlink in /etc needs to move:

mv /mnt/etc /mnt/etc.dist
cp -av /etc /mnt/
cp -av /usr/local/etc /mnt/usr/local/
cp -av /boot/loader.conf* /mnt/boot/
cp -av /var/unbound /mnt/var/
zfs snapshot -r zroot/ROOT/vanilla@configs

tidy up our dirty work

This is typically where things go wrong - missing mountpoints ensure that a system will not get past single-user mode, any video/storage/network drivers that are required at loader stage will stop earlier. Confirm you have console access!

This is the really dirty spot. In particular:

• the password file schema & default users & groups, could be different between FreeBSD versions
• we need to make sure all mountpoints are available inside the chroot; my dirty hack might not be sufficient for your needs
• the /etc dir we side-loaded is very likely to be different in key aspects between versions. You may need to polish, this is where the git repo of /etc comes in very handy

cd /mnt/etc
umask 027
git status
### make poor life choices now and edit away
git checkout -- want_other_version_of_this_file
git add -A
git commit -am 'post-13 sideload tweaks'

Now that we’ve done terrible things, we drop back into the chroot and rebuild our password file for the side-loaded FreeBSD version:

chroot /mnt /bin/sh
mkdir -p $(egrep -v 'none|^#' /etc/fstab | cut -wf 2 | sort | uniq)
uname -vm
    FreeBSD 14.0-CURRENT main-n244978-e1b88764b9c4 GENERIC  amd64
freebsd-version -kru
    13.0-BETA3    <---- our installed userland in the chroot
    14.0-CURRENT  <---- actual running kernel
    13.0-BETA3    <---- our installed kernel in the chroot
ln -s /usr/home /home
pwd_mkdb /etc/master.passwd
exit

Note that the new version 13.0-BETA3 is already showing up, even though we’ve not rebooted. This makes sense once you know how FreeBSD determines what version you’re running: freebsd-version uses on-disk information (from installed files) to determine what the installed kernel is, and that obviously can be different to the actual running version, during upgrades and boot environments.

install packages

From outside the chroot, run a final snapshot, and then we can install packages. Note that /etc/resolv.conf in the chroot must be usable!

At this point, we also need devfs and tmpfs to keep our chroot nice and tidy. If you don’t have a GB of ram free for a tmpfs, then consider nullfs mounting. Keeping the new BE clean is worth it.

zfs snapshot -r zroot/ROOT/vanilla@tidy`
mount -t devfs devfs /mnt/dev
mount -t tmpfs tmpfs /mnt/tmp
mount -t tmpfs tmpfs /mnt/var/cache/pkg
chroot /mnt /bin/sh
pkg install -r FreeBSD `cat /etc/packages.list`
exit
### outside the chroot again
zfs snapshot -r zroot/ROOT/vanilla@packages`

update boot blocks and/or EFI partitions

The final step is to ensure your boot blocks (UEFI or MBR) match. I’m leaving that as “exercise to the reader” but feel free to email me with any updates you figured out. Make sure to do this on all disks if you have zfs mirrors, or more than one disk.

activate and reboot

bectl activate vanilla
reboot

thanks

Guntbert Reiter replied with questions, and thus helped improve the article. Thanks for reading!