As part of getting FreeBSD running on OCI, we’ll need to teach the FreeBSD Release Engineering process and people how to build them, including our special tweaks and polishing.

Assume we have locally a clone FreeBSD src tree, either 13.x branch or main, and buildworld & buildkernel is already taken care of.

Releases are generated using a set of Makefiles in in /release, this is documented in release(7) and does a better overview than me. Note in particular the CLOUDWARE stuff, that’s what we need.

Running git log release shows you a lot of the previous tinkering, and reading those commits was largely what brought me up to speed. There are plentiful notes and solid commit messages, so finding the context is really easy.

Adding a Custom Makefile target

We can see there are 3 main files we need to touch:

• /release/Makefile.vm which introduces our new oci target
• /release/tools/oci.conf which specifies the image contents
• /release/Makefile.oci which will upload the images into OCI system

We’ll ignore the latter stage for the moment, and see what needs to be done for the first two.

Other image targets, such as GCE and EC2 are listed here, under CLOUDWARE?=, so let’s add an OCI set there:

Index: release/Makefile.vm
===================================================================
--- release/Makefile.vm
+++ release/Makefile.vm
@@ -19,6 +19,7 @@
 CLOUDWARE?=    BASIC-CI \
        EC2 \
        GCE \
+       OCI \
        VAGRANT-VIRTUALBOX \
        VAGRANT-VMWARE
AZURE_FORMAT=   vhdf
@@ -33,6 +34,9 @@
 GCE_FORMAT=    raw
 GCE_DESC=  Google Compute Engine image
 GCE_DISK=  disk.${GCE_FORMAT}
+OCI_FORMAT=    qcow2
+OCI_DESC=  Oracle Cloud Infrastructure image
+OCI_DISK=  disk.${OCI_FORMAT}
 OPENSTACK_FORMAT=qcow2
 OPENSTACK_DESC=    OpenStack platform image
 OPENSTACK_DISK=    ${OSRELEASE}.${OPENSTACK_FORMAT}
@@ -177,4 +181,6 @@
 .include "${.CURDIR}/Makefile.ec2"
 .include "${.CURDIR}/Makefile.azure"
 .include "${.CURDIR}/Makefile.gce"
+# TODO write one of these when we have figured out upload
+# .include "${.CURDIR}/Makefile.oci"
 .include "${.CURDIR}/Makefile.vagrant"

This small change allows building images like this:

# make -j64 buildworld -s
# make -j64 buildkernel KERNCONF=GENERIC -s

# cd release
# make clean
Building /usr/obj/usr/src/arm64.aarch64/release/beforeclean
rm -f packagesystem *.txz MANIFEST release  disc1.iso bootonly.iso memstick.img mini-memstick.img ec2ami cw-ec2-portinstall azure-check-depends  azure-do-upload gce-do-login vagrant-check-depends vagrant-do-upload-virtualbox vagrant-do-upload-vmware vagrant-do-package-virtualbox FreeBSD-13.1-RC1-arm64-aarch64.virtualbox.box vagrant-do-upload-virtualbox vagrant-do-package-vmware FreeBSD-13.1-RC1-arm64-aarch64.vmware.box vagrant-do-upload-vmware
rm -rf dist ftp disc1 bootonly dvd virtualbox vmware

# make -DNOPORTS -DNOSRC \
    KERNCONF=GENERIC \
    WITH_CLOUDWARE=yes \
    CLOUDWARE=OCI \
    -s \
    cloudware-release

make[3]: "/usr/obj/usr/src/arm64.aarch64/toolchain-metadata.mk" line 1: Using cached toolchain metadata from build at straylight.skunkwerks.at on Fri Apr  1 23:43:26 UTC 2022
--------------------------------------------------------------
>>> Install check world
...

# ls -AFGhls /usr/obj/usr/src/arm64.aarch64/release/oci.qcow2
671989 -rw-r--r--  1 root  wheel   2.4G Mar 26 10:37 oci.qcow2

That seems fine, but the image is very large - it looks awfully like we have ~ 0,4GiB of actual data and 2.0GiB of empty swap file?

The mkimg(1) tool that is used to create the images, seems not to do native qcow compression, so we will re-build our image using native qemu-img(1), which is from the emulators/qemu port:

$ qemu-img convert -S 1k -p -O raw -f qcow2 oci.qcow2 oci.raw
    (100.00/100%)
$ qemu-img convert -S 1k -p -O qcow2 -f raw -c oci.raw oci-compressed.qcow2
    (100.00/100%)
$  ls -AFGhlsS
total 5625784
2515616 -rw-r--r--  1 dch  wheel   6.0G Mar 26 10:38 oci.raw
2551104 -rw-r--r--  1 dch  wheel   2.4G Mar 26 10:37 oci.qcow2
 757768 -rw-r--r--  1 dch  wheel   740M Mar 26 10:41 oci-compressed.qcow2

That’s looks much better for uploading from my pathetically slow home office internet.

Customising the Image

So far, all we’ve done is produced the same image as existing systems, but we need some customisations:

• loader.conf to ensure we have serial console boot output
• rc.conf tweaks to start sshd, ntpd
• ensure packages and patches are applied at startup
• use cloud-init to fetch metadata & admin ssh keys at instance boot

For the purposes of this post, we’re not really interested in the exact changes, just in the mechanisms on how this is done. Let’s take a closer look at release/tools/oci.conf, in sections:

• define packages to install inside the image
• these are fetched from quarterly branch by default
• list which daemons will be auto-started in the image

#!/bin/sh
#
# $FreeBSD$
#

# # Set to a list of packages to install.
export VM_EXTRA_PACKAGES="
    ftp/curl
    net/cloud-init
    sysutils/firstboot-freebsd-update
    sysutils/firstboot-pkgs 
    sysutils/panicmail
    sysutils/tmux
"
# Should be enough for base image, image can be resized in needed
export VMSIZE=5g

# Set to a list of third-party software to enable in rc.conf(5).
export VM_RC_LIST="
    cloudinit
    firstboot_pkgs
    firstboot_freebsd_update
    growfs
    ntpd
    ntpd_sync_on_start
    sshd
    zfs"

heredocs

• we ensure that DHCP is available on all interfaces, with the magical ifconfig_DEFAULT trick that assigns settings to any unconfigured interfaces
• when will sendmail in base finally die?
• loader.conf tunables ensure that early stage output goes over the serial interface
• enable root ssh only via keys, with sshd_config & authorized_keys

vm_extra_pre_umount() {
    cat << EOF >> ${DESTDIR}/etc/rc.conf
dumpdev=AUTO
ifconfig_DEFAULT=SYNCDHCP
sendmail_enable=NONE
EOF

    cat << EOF >> ${DESTDIR}/boot/loader.conf
autoboot_delay="5"
beastie_disable="YES"
boot_serial="YES"
loader_logo="none"
# ensure disk devices are found by label not partition
# kern.geom.label.disk_ident.enable="0"
# kern.geom.label.gptid.enable="0"
# storage
cryptodev_load="YES"
opensolaris_load="YES"
xz_load="YES"
zfs_load="YES"
EOF

    test -d ${DESTDIR}/root/.ssh || mkdir -p -m 0700 ${DESTDIR}/root/.ssh
    cat <<EOF >> ${DESTDIR}/root/.ssh/authorized_keys
# dch
ecdsa-sha2-nistp384 AAAAE2VjZHNhLXNoYTItbmlzdHAzODQAAAAIbmlzdHAzODQAAABhBOK3AqefoNusxGu6Oo89aex0keBn7qAnZxXKsD77b0rp3UKJLKettY60Ox47XhIbt4Y50Xjnc0GRKhEX0jceZYO/CgZ53V/tLS5TmOOU2gZVc/7DSTy+gkQiDT9CNAiiEQ==
ecdsa-sha2-nistp384 AAAAE2VjZHNhLXNoYTItbmlzdHAzODQAAAAIbmlzdHAzODQAAABhBItIwUr8zhXOBtFH1B0YmNz2WJcY6w1ysRiTAIkI2CBenMb0f7H2pH1rFAa6ZF6dYS3SuLMng+igZUfkqhV/0Km+zus3lAjc37FFiawtATt+/nRj3hj/AaVz/cK7NnWdlg==
EOF
    chown -R root:wheel ${DESTDIR}/root/.ssh
    chmod 0600 ${DESTDIR}/root/.ssh/authorized_keys

    cat <<EOF >> ${DESTDIR}/etc/ssh/sshd_config
PermitRootLogin prohibit-password
PasswordAuthentication no
KbdInteractiveAuthentication no
PermitEmptyPasswords no
UsePAM no
UseDNS no
EOF

• finally, enable firstboot which provides automatic patching and pkg(8) upgrades on startup, via firstboot-freebsd-update and firstboot-pkgs respectively
• these are short self-explanatory shell scripts directly in ports tree

    touch ${DESTDIR}/firstboot

    return 0
}

Summary

The FreeBSD release procedure is well documented and pretty solid, after a look around, it seems most of the hard work is already done!

Our next stages are:

• upload these new images into OCI
• automate the upload process

Stay tuned.