Booting FreeBSD directly from firmware via HTTP

Tuesday, 2 Jun 2020 Tags: boot freebsd http

HTTP booting FreeBSD

If your UEFI firmware is later than 2.5 and you have a compatible NIC, it’s possible to for your computer’s boot sequence to hand off directly from the UEFI firmware to FreeBSD’s HTTP-enabled loader.efi!

If you don’t, you can use net/ipxe to achieve the same result, with a very small “virtual USB memstick”, or just a file in the EFI partition.

This requires a few moving parts, particularly you need to have either:

a hard-coded HTTP URL in your UEFI firmware, with DHCP or static IP
a DHCP server that can return an HTTP URL
chain-loading iPXE (which provides the HTTP loading capabilities) from a less capable PXE boot client

Once that’s done, loader.efi will retrieve all additional files it needs from the same HTTP path that it was itself loaded. This is significantly faster than the slower DHCP->PXE->NFS chain that has been used in the past for diskless clients, and it’s far more flexible than NFS as we all know how to use proxy and web servers to redirect almost anything to anywhere: for example, we can share the same kernel across multiple clients, but allow different loader.conf strategies for each system.

The final boot stage is to switch from the kernel into a new mfs (ramdisk) and pivot into that via reboot -r.

thanks

bcran
ocochard
emaste
…

Objectives

Set up the webserver with appropriate files
Build using poudriere, a suitable “mini-root”
- this enables loading Just Enough FreeBSD to pivot into a memdisk
Create qemu and bhyve VMs to test with
- configure UEFI firmware to use HTTP boot
- set up your system to UEFI boot over HTTP
- or, use the iPXE system to handle that instead
Boot to kernel over HTTP and $$PROFIT$$

Setting up the HTTP directory

Let’s assume you have /var/www/pub somewhere as our root for this, and the usual tarballs of base.txz and kernel.txz handy. These will need to be from 13.0-CURRENT at present to have the requisite functionality, the same build as used for your poudriere install below.

# WWW=/var/www/pub/
# tar xzC ${WWW} -f base.txz   boot/
# tar xzC ${WWW} -f kernel.txz boot/

Do the chmod thing and ensure that this is accessible via HTTP, not HTTPS.

what is loaded

According to my HTTP server logs, the following files will be retrieved, up to the point you have loaded the kernel and begun the rc.d script:

$ tail -qF /var/log/messages |rg -Po "GET /.+ (?=HTTP/1.1)"
/boot/loader.efi 
/boot/defaults/loader.conf 
/boot/lua/loader.lua 
/boot/lua/cli.lua 
/boot/lua/config.lua 
/boot/lua/hook.lua 
/boot/lua/core.lua 
/boot/lua/color.lua 
/boot/lua/password.lua 
/boot/lua/screen.lua 
/boot/lua/local.lua 
boot/defaults/loader.conf 
/boot/device.hints 
/boot/loader.conf 
/boot/loader.conf.local 
/boot/nextboot.conf 
/boot/lua/menu.lua 
/boot/lua/drawer.lua 
/boot/kernel/linker.hints 
/boot/kernel/kernel 
/boot/kernel/tmpfs.ko 
/boot/mfs-miniroot <-------- we will create this later on

If it can’t find a given file, it will loop through a few reasonable options before giving up:

/boot/lua/local.lua/
/boot/lua/local.lua/ 
/boot/lua/local.lua.gz 
/boot/lua/local.lua.gz/ 
/boot/lua/local.lua.bz2 
/boot/lua/local.lua.bz2/

Make a miniroot memdisk to pivot into

NB this probably only works if you’re running -CURRENT already, but you can fetch a recent base.txz and kernel.txz from snapshots, and seeing how that works out if you’re game.

You’ll need ports-mgmt/poudriere-devel to gain the new image functionality, we remove a small broken make invocation from it, and fix a small awk bug:

https://github.com/freebsd/poudriere/pull/732

First you build a new jail using your tarball, in my case this is the one my system is already running, so everything Just Works.

This was lifted entirely from ocochard‘s amazing BSDRP BSD Router Project

I’ll assume you have poudriere set up, and created a nice 13.0-CURRENT already.

Populate your /var/www/pub/boot/miniroot dir with any loader.conf parameters you want in the image, and this rc script I lifted verbatim from ocochard‘s blog post from his BSDRP project.

# poudriere jail -c\
   -j current_x64 \
   -v 13.0-CURRENT \
   -a amd64 \
   -S /usr/src \
   -K GENERIC \
   -m tar=/downloads/FreeBSD/current/latest/base.txz

# fetch -o - https://git.io/JeXhP \
    | patch -p4 /usr/local/share/poudriere/image.sh

# poudriere image -t tar \
    -n mfs \
    -j current_x64 \
    -m /var/www/pub/boot/miniroot/ \
    -c /var/www/pub/boot/miniroot/ \
    -o /var/www/pub/boot/

boot qemu via HTTP

You’ll need to drop into the EFI shell in qemu, and configure the network device to boot via HTTP to your loader.efi URL.

# qemu-img create -f qcow2 /tmp/disk0.qcow 10G
#  qemu-system-x86_64 \
	-m 8192M \
	-serial telnet::6000,server \
	-drive if=pflash,format=raw,readonly,file=/usr/local/share/uefi-edk2-qemu/QEMU_UEFI_CODE-x86_64.fd \
	-drive if=pflash,format=raw,file=/usr/local/share/uefi-edk2-qemu/QEMU_UEFI_VARS-x86_64.fd \
	-nographic \
	-vga none \
	-drive file=/tmp/disk0.qcow,if=virtio \
	-net nic -net tap,ifname=tap0 \
	-object rng-random,id=rng0,filename=/dev/urandom -device virtio-rng-pci,rng=rng0 \
	-rtc base=utc

make UEFI iPXE boot disk

For bhyve, there is no embedded PXE boot yet, but we can make a readonly memdisk and store it in a file just for this.

# truncate -s 4M /tmp/uefi.img
# mdconfig -a -t vnode -u 99 -f /tmp/uefi.img
# gpart create -s gpt /dev/md99
# gpart add -t efi /dev/md99
# newfs_msdos -F12 /dev/md99p1
# mount -t msdosfs -o longnames /dev/md99p1 /mnt
# mkdir -p /mnt/EFI/Boot/
# cp (pkg list net/ipxe |grep x86_64) /mnt/EFI/Boot/BootX64.efi
# umount /mnt
# mdconfig -du md99

build arm64 UEFI iPXE image

My ampere server needs a more up-to-date UEFI boot image than what’s normally available. This was built on a boring ubuntu.

$ sudo apt-get -y --no-install-recommends install build-essential \
  gcc-aarch64-linux-gnu git liblzma-dev
$ git clone https://github.com/ipxe/ipxe
$ git log --oneline HEAD -1
  1192edf3 (HEAD -> master) [dhcp] Handle DHCPNAK by returning to discovery state
$ make CROSS_COMPILE=aarch64-linux-gnu- ARCH=arm64 bin-arm64-efi/snp.efi

references

boot bhyve from UEFI boot disk

# bhyvectl --vm=pixie --force-poweroff
# bhyvectl --vm=pixie --destroy
# /usr/sbin/bhyve -c 2 -m 6G -H -w \
    -s 0,hostbridge \
    -s 1,nvme,/var/www/freeside/pub/boot/uefi.img \
    -s 2,virtio-net,tap0 \
    -s 29,fbuf,tcp=0.0.0.0:5900 \
    -s 30,xhci,tablet \
    -s 31,lpc \
    -l com1,stdio \
    -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI_CODE-devel.fd \
    pixie
    #    -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI_CODE-devel.fd \
    # -s 4,nvme,/projects/groupon/iPXE/ipxe.efi.fat \
    # -s 3,virtio-9p,root=/projects/groupon/iPXE \

PCBIOS iPXE boot

It’s not possible to pivot from a BIOS boot to a UEFI firmware, and so if your iPXE boot shows that you’re booting in PC BIOS mode like this, you can’t use the UEFI HTTP boot functionality in FreeBSD. Sad Pandas.

                                                                                
net0: 0c:c4:7a:e5:46:7c using undionly on 0000:00:14.0 (open)                   
  [Link:up, TX:0 TXE:1 RX:0 RXE:0]                                              
  [TXE: 1 x "Network unreachable (http://ipxe.org/28086011)"]                   
Configuring (net0 0c:c4:7a:e5:46:7c).................. ok                       
net0: 147.75.84.133/255.255.255.254 gw 147.75.84.132                            
net0: fe80::ec4:7aff:fee5:467c/64                                               
Next server: 147.75.204.3                                                       
Filename: http://147.75.204.3/auto.ipxe                                         
http://147.75.204.3/auto.ipxe... ok                                             
auto.ipxe : 385 bytes [script]                                                  
Packet.net Baremetal - iPXE boot                                                
http://147.75.204.3/phone-home...... ok                                         
http://freeside.skunkwerks.at/pub/boot/ipxe.sh... ok                            
......... iPXE..................                                                
mac..........0c:c4:7a:e5:46:7c                                                  
uuid.........00000000-0000-0000-0000-000000000000                               
ip...........147.75.84.133                                                      
filename.....http://147.75.204.3/auto.ipxe                                      
default-url..                                                                   
boot.mode....pcbios            <---------------------------- bad ------->                      
http://freeside.skunkwerks.at/pub/ipxe/log... ok                                
http://freeside.skunkwerks.at/pub/boot/loader.efi... ok

Random Musings