Booting FreeBSD directly from firmware via HTTP
Tuesday, 2 Jun 2020
HTTP booting FreeBSD
If your UEFI firmware is later than 2.5 and you have a compatible NIC, it’s possible to for your computer’s boot sequence to hand off directly from the UEFI firmware to FreeBSD’s HTTP-enabled loader.efi!
If you don’t, you can use net/ipxe to achieve the same result, with a very small “virtual USB memstick”, or just a file in the EFI partition.
This requires a few moving parts, particularly you need to have either:
- a hard-coded HTTP URL in your UEFI firmware, with DHCP or static IP
- a DHCP server that can return an HTTP URL
- chain-loading iPXE (which provides the HTTP loading capabilities) from a less capable PXE boot client
Once that’s done, loader.efi will retrieve all additional files it needs
from the same HTTP path that it was itself loaded. This is significantly
faster than the slower DHCP->PXE->NFS chain that has been used in the past
for diskless clients, and it’s far more flexible than NFS as we all know
how to use proxy and web servers to redirect almost anything to anywhere:
for example, we can share the same kernel across multiple clients, but allow
different loader.conf strategies for each system.
The final boot stage is to switch from the kernel into a new mfs (ramdisk)
and pivot into that via reboot -r.
thanks
- bcran
- ocochard
- emaste
- …
Objectives
- Set up the webserver with appropriate files
Build using poudriere, a suitable “mini-root”
- this enables loading Just Enough FreeBSD to pivot into a memdisk
Create qemu and bhyve VMs to test with
- configure UEFI firmware to use HTTP boot
- set up your system to UEFI boot over HTTP
- or, use the iPXE system to handle that instead
- Boot to kernel over HTTP and $$PROFIT$$
Setting up the HTTP directory
Let’s assume you have /var/www/pub somewhere as our root for this, and
the usual tarballs of base.txz and kernel.txz handy. These will need to
be from 13.0-CURRENT at present to have the requisite functionality, the
same build as used for your poudriere install below.
# WWW=/var/www/pub/
# tar xzC ${WWW} -f base.txz boot/
# tar xzC ${WWW} -f kernel.txz boot/Do the chmod thing and ensure that this is accessible via HTTP, not HTTPS.
what is loaded
According to my HTTP server logs, the following files will be retrieved,
up to the point you have loaded the kernel and begun the rc.d script:
$ tail -qF /var/log/messages |rg -Po "GET /.+ (?=HTTP/1.1)"
/boot/loader.efi
/boot/defaults/loader.conf
/boot/lua/loader.lua
/boot/lua/cli.lua
/boot/lua/config.lua
/boot/lua/hook.lua
/boot/lua/core.lua
/boot/lua/color.lua
/boot/lua/password.lua
/boot/lua/screen.lua
/boot/lua/local.lua
boot/defaults/loader.conf
/boot/device.hints
/boot/loader.conf
/boot/loader.conf.local
/boot/nextboot.conf
/boot/lua/menu.lua
/boot/lua/drawer.lua
/boot/kernel/linker.hints
/boot/kernel/kernel
/boot/kernel/tmpfs.ko
/boot/mfs-miniroot <-------- we will create this later onIf it can’t find a given file, it will loop through a few reasonable options before giving up:
/boot/lua/local.lua/
/boot/lua/local.lua/
/boot/lua/local.lua.gz
/boot/lua/local.lua.gz/
/boot/lua/local.lua.bz2
/boot/lua/local.lua.bz2/ Make a miniroot memdisk to pivot into
NB this probably only works if you’re running -CURRENT already, but you can
fetch a recent base.txz and kernel.txz from snapshots, and seeing how that
works out if you’re game.
You’ll need ports-mgmt/poudriere-devel to gain the new image functionality, we remove a small broken make invocation from it, and fix a small awk bug:
https://github.com/freebsd/poudriere/pull/732
First you build a new jail using your tarball, in my case this is the one my system is already running, so everything Just Works.
This was lifted entirely from ocochard’s amazing BSDRP BSD Router Project
I’ll assume you have poudriere set up, and created a nice 13.0-CURRENT already.
Populate your /var/www/pub/boot/miniroot dir with any loader.conf parameters
you want in the image, and this rc script I lifted verbatim from ocochard’s
blog post from his BSDRP project.
# poudriere jail -c\
-j current_x64 \
-v 13.0-CURRENT \
-a amd64 \
-S /usr/src \
-K GENERIC \
-m tar=/downloads/FreeBSD/current/latest/base.txz
# fetch -o - https://git.io/JeXhP \
| patch -p4 /usr/local/share/poudriere/image.sh
# poudriere image -t tar \
-n mfs \
-j current_x64 \
-m /var/www/pub/boot/miniroot/ \
-c /var/www/pub/boot/miniroot/ \
-o /var/www/pub/boot/boot qemu via HTTP
You’ll need to drop into the EFI shell in qemu, and configure
the network device to boot via HTTP to your loader.efi URL.
# qemu-img create -f qcow2 /tmp/disk0.qcow 10G
# qemu-system-x86_64 \
-m 8192M \
-serial telnet::6000,server \
-drive if=pflash,format=raw,readonly,file=/usr/local/share/uefi-edk2-qemu/QEMU_UEFI_CODE-x86_64.fd \
-drive if=pflash,format=raw,file=/usr/local/share/uefi-edk2-qemu/QEMU_UEFI_VARS-x86_64.fd \
-nographic \
-vga none \
-drive file=/tmp/disk0.qcow,if=virtio \
-net nic -net tap,ifname=tap0 \
-object rng-random,id=rng0,filename=/dev/urandom -device virtio-rng-pci,rng=rng0 \
-rtc base=utcmake UEFI iPXE boot disk
For bhyve, there is no embedded PXE boot yet, but we can make a readonly memdisk and store it in a file just for this.
# truncate -s 4M /tmp/uefi.img
# mdconfig -a -t vnode -u 99 -f /tmp/uefi.img
# gpart create -s gpt /dev/md99
# gpart add -t efi /dev/md99
# newfs_msdos -F12 /dev/md99p1
# mount -t msdosfs -o longnames /dev/md99p1 /mnt
# mkdir -p /mnt/EFI/Boot/
# cp (pkg list net/ipxe |grep x86_64) /mnt/EFI/Boot/BootX64.efi
# umount /mnt
# mdconfig -du md99build arm64 UEFI iPXE image
My ampere server needs a more up-to-date UEFI boot image than what’s normally available. This was built on a boring ubuntu.
$ sudo apt-get -y --no-install-recommends install build-essential \
gcc-aarch64-linux-gnu git liblzma-dev
$ git clone https://github.com/ipxe/ipxe
$ git log --oneline HEAD -1
1192edf3 (HEAD -> master) [dhcp] Handle DHCPNAK by returning to discovery state
$ make CROSS_COMPILE=aarch64-linux-gnu- ARCH=arm64 bin-arm64-efi/snp.efireferences
- https://ipxe.org/
- https://ipxe.org/appnote/buildtargets
- https://forum.ipxe.org/showthread.php?tid=8526
boot bhyve from UEFI boot disk
# bhyvectl --vm=pixie --force-poweroff
# bhyvectl --vm=pixie --destroy
# /usr/sbin/bhyve -c 2 -m 6G -H -w \
-s 0,hostbridge \
-s 1,nvme,/var/www/freeside/pub/boot/uefi.img \
-s 2,virtio-net,tap0 \
-s 29,fbuf,tcp=0.0.0.0:5900 \
-s 30,xhci,tablet \
-s 31,lpc \
-l com1,stdio \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI_CODE-devel.fd \
pixie
# -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI_CODE-devel.fd \
# -s 4,nvme,/projects/groupon/iPXE/ipxe.efi.fat \
# -s 3,virtio-9p,root=/projects/groupon/iPXE \PCBIOS iPXE boot
It’s not possible to pivot from a BIOS boot to a UEFI firmware, and so if your iPXE boot shows that you’re booting in PC BIOS mode like this, you can’t use the UEFI HTTP boot functionality in FreeBSD. Sad Pandas.
net0: 0c:c4:7a:e5:46:7c using undionly on 0000:00:14.0 (open)
[Link:up, TX:0 TXE:1 RX:0 RXE:0]
[TXE: 1 x "Network unreachable (http://ipxe.org/28086011)"]
Configuring (net0 0c:c4:7a:e5:46:7c).................. ok
net0: 147.75.84.133/255.255.255.254 gw 147.75.84.132
net0: fe80::ec4:7aff:fee5:467c/64
Next server: 147.75.204.3
Filename: http://147.75.204.3/auto.ipxe
http://147.75.204.3/auto.ipxe... ok
auto.ipxe : 385 bytes [script]
Packet.net Baremetal - iPXE boot
http://147.75.204.3/phone-home...... ok
http://freeside.skunkwerks.at/pub/boot/ipxe.sh... ok
......... iPXE..................
mac..........0c:c4:7a:e5:46:7c
uuid.........00000000-0000-0000-0000-000000000000
ip...........147.75.84.133
filename.....http://147.75.204.3/auto.ipxe
default-url..
boot.mode....pcbios <---------------------------- bad ------->
http://freeside.skunkwerks.at/pub/ipxe/log... ok
http://freeside.skunkwerks.at/pub/boot/loader.efi... ok