Random Musings

O for a muse of fire, that would ascend the brightest heaven of invention!


Booting FreeBSD on Ampere's Awesome Altra CPUs

Friday, 2 Jul 2021 Tags: amperefreebsdoci

This is a series of posts exploring how we can get FreeBSD to boot up in Oracle Cloud Infrastructure OCI, which is the commercially available source of the 160-core dual socket Altra CPUs from Ampere.

Geek Envy

Yes, 160 cores on a single system. These are not your scabby x86 Symmetric Hyperthreading cores, each one is a fully capable 3.3GHz behemoth. Here are the specs from the initial Altra announcement:

Altra® Multi Core Server Processors

Ampere server processors are a complete system on chip (SOC) solution built for Cloud Native server applications. The device supports up to 128 Arm 64 bit cores, 8 Channels of DDR4 and 128 Channels of PCIe Gen 4 interfaces.

  • Up to 128 cloud optimized 64 bit Armv8.2 cores
  • Up to 3.3GHz frequency of operation
  • 8×72 bit DDR4 3200, ECC Support, 4TB Capacity
  • 128 PCIe Gen4 lanes
  • Coherent multi socket support
  • I/O and Interrupt Virtualization
  • Enterprise server class RAS

There is some performance propaganda available too.

Support

The arm64 platform has had tier 1 support since FreeBSD 13.0-RELEASE, and I have been running it very happily on the “small” eMAG systems from Ampere for many months. When I say “small” it only has 32 cores, runs at a leisurely 3.0GHz and my install only has a single PCIe4 NVMe drive, with just 256GiB RAM… so only small in comparison to its newer, bigger twin brothers!

We’ve had access to the newer Altra system hardware already, and the functionality is identical.

The performance is excellent, all the nice FreeBSD bits like ZFS, jails, and firewalls just work, as you’d expect. Some functionality (full IPMI, various sensors, including cpu frequency & temperature sysctls), are not yet exposed but all the important stuff - networking, storage, remote console, is fantastic. Of particular note is that the NICs support HTTP boot, as well as full remote console (mounting images etc), so once you’ve physically plugged the box in, you’ll rarely need to go to it.

ipxe also works, so a simple recovery solution is to configure the UEFI firmware to load ipxe over the network, and from there to pull in the FreeBSD loader via HTTP boot.

OCI

Oracle has production Altra systems available, both as para-virtualised cloud VMs, and also as bare metal behemoths. There’s a bunch of propaganda docs with some useful general info:

We are interested in 2 things:

  • boot FreeBSD on PV mode
  • boot FreeBSD in BM mode

Hijacking Linux, aka de-penguinification

As a starting point, I tried “de-penguinating” an existing Linux instance. This was easy: drop loader.efi into the EFI partition, dd(1) the FreeBSD installer on top of the Linux swap partition, and reboot. But, on reboot, the EFI partition mysteriously returned to Oracle defaults. I am not sure if this is a loader issue, or an OCI feature.

# cd /root
# curl -#LO https://download.freebsd.org/ftp/releases/arm64/aarch64/13.0-RELEASE/kernel.txz
# curl -#LO https://download.freebsd.org/ftp/releases/arm64/aarch64/13.0-RELEASE/base.txz
# tar xvJf base.txz ./boot/loader_lua.efi
# mkdir -p /boot/efi/EFI/FreeBSD
# mv ./boot/loader_lua.efi /boot/efi/EFI/FreeBSD/13.efi
# efibootmgr -c -L FreeBSD  -l /boot/efi/EFI/FreeBSD/13.efi
    BootCurrent: 0001
    Timeout: 0 seconds
    BootOrder: 0005,0004,0000,0001,0002,0003
    Boot0000* UiApp
    Boot0001* UEFI ORACLE BlockVolume
    Boot0002* UEFI PXEv4 (MAC:020017009D5B)
    Boot0003* EFI Internal Shell
    Boot0004* Oracle Linux
    Boot0005* FreeBSD
# efibootmgr -n 5
    BootNext: 0005
    BootCurrent: 0001
    Timeout: 0 seconds
    BootOrder: 0005,0004,0000,0001,0002,0003
    Boot0000* UiApp
    Boot0001* UEFI ORACLE BlockVolume
    Boot0002* UEFI PXEv4 (MAC:020017009D5B)
    Boot0003* EFI Internal Shell
    Boot0004* Oracle Linux
    Boot0005* FreeBSD

I re-tried with refind, also no luck.

Bring Your Own Images

My second attempt was to upload various FreeBSD release installers into OCI Cloud Storage, then create new Bring-Your-Own bootable images from those uploaded files, and try to boot off those.

$ qemu-img convert -f raw -O qcow2 \
    FreeBSD-13.0-RELEASE-arm64-aarch64-memstick.img \
    FreeBSD-13.0-RELEASE-arm64-aarch64-memstick.qcow2

I also realised in hindsight that FreeBSD provides both vmdk and raw ready-to-roll images, but I did not get as far with those.

This process is a little muddly, I will update this post in future on exactly how to go about that. In a nutshell, qemu-img convert your desired setup, upload it into blob storage, create a new “cloud image”, tweak the supported shapes to only A1.Flex and A1.BM types (Para-virtualised and Bare Metal respectively), and guess randomly the bewildering array of options you’re presented with. I would hope that Native eventually works, but it’s not entirely clear what underlying hardware this actually is. I expect it’s some sort of KVM / QEMU combined with iSCSI for storage, and VFIO for networking.

When you create the image, you need to define its capabilities, and compatible shapes:

  • BM.Standard.A1.160
  • VM.Standard.A1.Flex
  • firmware: uefi_64
  • launch mode: emulated
  • nic attachment type: paravirtualised
  • boot: paravirtualised
  • boot volume type: paravirtualised
  • local data volume: paravirtualised
  • remote data volume: paravirtualised
  • paravirtualisation version: 2 (using v1 also fails)

I tried creating a native image type, which got quite a bit further. Unfortunately it’s not clear what the image source was, but I think it’s my qemu-img converted raw FreeBSD installer.

Image:                                  freebsd-13.0-arm64-native
Launch Mode:                            EMULATED
Shape:                                  VM.Standard.A1.Flex
OCPU Count:                             1
Network Bandwidth (Gbps):               1
Memory (GB):                            6
Local Disk:                             Block Storage Only
NIC Attachment Type:                    PARAVIRTUALIZED
Remote Data Volume:                     PARAVIRTUALIZED
Firmware:                               UEFI_64
Boot Volume Type:                       PARAVIRTUALIZED

After the instance has been created, add a console connection which is just a bunch of ssh wrapping whatever KVM provides us over serial port, and then force restart the instance, (its dead at this point anyway) and see the console garbage fly past.

$ ssh -o ProxyCommand='ssh -W %h:%p -p 443 ocid1.@oci.oraclecloud.com' \
ocid1.instance.oc1.eu-amsterdam-1....
The authenticity of host
'[instance-console.eu-amsterdam-1.oci.oraclecloud.com]:443
([140.204.36.211]:443)' can't be established. RSA key fingerprint is
SHA256:R4nCgf3DqENCAEJtA2e+UfkzG3dTUhgUqLvLak/Y8bk.
No matching host key fingerprint found in DNS.
This host key is known by the following other names/addresses:
    ~/.ssh/known_hosts:74: [instance-console.eu-amsterdam-1.oraclecloud.com]:443
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added
'[instance-console.eu-amsterdam-1.oci.oraclecloud.com]:443' (RSA) to the
list of known hosts.

=================================================
IMPORTANT: Use a console connection to troubleshoot a malfunctioning
instance. For normal operations, you should connect to the instance
using a Secure Shell (SSH) or Remote Desktop connection. For steps, see
https://docs.cloud.oracle.com/iaas/Content/Compute/Tasks/accessinginstance.htm

For more information about troubleshooting your instance using a console
connection, see the documentation:
https://docs.cloud.oracle.com/en-us/iaas/Content/Compute/References/serialconsole.htm#four
=================================================

The authenticity of host 'ocid1.instance.oc1.eu-amsterdam-1...
(<no hostip for proxy command>)' can't be established. RSA key
fingerprint is SHA256:cVF7Hw5kRv4gyaxZwj0yqAWUlvU2LwW5VXJ124LgxhI.
No matching host key fingerprint found in DNS.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added
'ocid1.instance.oc1.eu-amsterdam-1....'
(RSA) to the list of known hosts.

I could get to the loader prompt, as far as the infamous error 19:

mountroot: waiting for device /dev/ufs/FreeBSD_Install...
Mounting from ufs:/dev/ufs/FreeBSD_Install failed with error 19.

This presumably fail without appropriate iSCSI or similar drivers already available to the loader itself. This may require building a custom FreeBSD kernel with the drivers already embedded into loader.efi, if that’s even possible.

The full dmesg, as far as we get, follows:

   ______               ____   _____ _____
  |  ____|             |  _ \ / ____|  __ \
  | |___ _ __ ___  ___ | |_) | (___ | |  | |
  |  ___| '__/ _ \/ _ \|  _ < \___ \| |  | |
  | |   | | |  __/  __/| |_) |____) | |__| |
  | |   | | |    |    ||     |      |      |
  |_|   |_|  \___|\___||____/|_____/|_____/      ```                        `
                                                s` `.....---.......--.```   -/
 /-----------Welcome to FreeBSD------------\    +o   .--`         /y:`      +.
 |                                         |     yo`:.            :o      `+-
 |  1. Boot Multi user [Enter]             |      y/               -/`   -o/
 |  2. Boot Single user                    |     .-                  ::/sy+:.
 |  3. Escape to loader prompt             |     /                     `--  /
 |  4. Reboot                              |    `:                          :`
 |  5. Cons: Dual (Video primary)          |    `:                          :`
 |                                         |     /                          /
 |  Options:                               |     .-                        -.
 |  6. Kernel: default/kernel (1 of 1)     |      --                      -.
 |  7. Boot Options                        |       `:`                  `:`
 |                                         |         .--             `--.
 |                                         |            .---.....----.
 \-----------------------------------------/
   Autoboot in 0 seconds, hit [Enter] to boot or any other key to stop

Loading kernel...
/boot/kernel/kernel text=0x2a8 text=0x8a7b2c text=0x1e9934 data=0x1a31c8 data=0x0+0x37f000 syms=[0x8+0x117420+0x8+0x13cea2]
Loading configured modules...
can't find '/boot/entropy'
can't find '/etc/hostid'
No valid device tree blob found!
WARNING! Trying to fire up the kernel, but no device tree blob found!
EFI framebuffer information:
addr, size     0x0, 0x0
dimensions     800 x 600
stride         800
masks          0x000000ff, 0x0000ff00, 0x00ff0000, 0xff000000
---<<BOOT>>---
Copyright (c) 1992-2021 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 13.0-RELEASE #0 releng/13.0-n244733-ea31abc261f: Fri Apr  9 03:54:53 UTC 2021
    root@releng1.nyi.freebsd.org:/usr/obj/usr/src/arm64.aarch64/sys/GENERIC arm64
FreeBSD clang version 11.0.1 (git@github.com:llvm/llvm-project.git llvmorg-11.0.1-0-g43ff75f2c3fe)
VT: init without driver.
module firmware already present!
real memory  = 6305218560 (6013 MB)
avail memory = 6123745280 (5840 MB)
FreeBSD/SMP: Multiprocessor System Detected: 1 CPUs
arc4random: WARNING: initial seeding bypassed the cryptographic random device because it was not yet seeded and the knob 'bypass_before_seeding' was enabled.
random: entropy device external interface
MAP 1b06b0000 mode 2 pages 80
MAP 1b0700000 mode 2 pages 80
MAP 1b0750000 mode 2 pages 80
MAP 1b07a0000 mode 2 pages 80
MAP 1b07f0000 mode 2 pages 192
MAP 1b08b0000 mode 2 pages 96
MAP 1b0910000 mode 2 pages 160
MAP 1b09b0000 mode 2 pages 80
MAP 1b0a00000 mode 2 pages 80
MAP 1b0a50000 mode 2 pages 240
MAP 1b3e20000 mode 2 pages 144
MAP 1b3ec0000 mode 2 pages 288
MAP 4000000 mode 0 pages 16384
MAP 9010000 mode 0 pages 1
WARNING: Device "openfirm" is Giant locked and may be deleted before FreeBSD 14.0.
WARNING: Device "kbd" is Giant locked and may be deleted before FreeBSD 14.0.
kbd0 at kbdmux0
acpi0: <BOCHS BXPCFACP>
acpi0: Power Button (fixed)
acpi0: Sleep Button (fixed)
acpi0: Could not update all GPEs: AE_NOT_CONFIGURED
psci0: <ARM Power State Co-ordination Interface Driver> on acpi0
gic0: <ARM Generic Interrupt Controller v3.0> iomem 0x8000000-0x801ffff,0x80a0000-0x8ffffff on acpi0
its0: <ARM GIC Interrupt Translation Service> on gic0
generic_timer0: <ARM Generic Timer> irq 34,35,36 on acpi0
Timecounter "ARM MPCore Timecounter" frequency 25000000 Hz quality 1000
Event timer "ARM MPCore Eventtimer" frequency 25000000 Hz quality 1000
efirtc0: <EFI Realtime Clock>
efirtc0: registered as a time-of-day clock, resolution 1.000000s
cpu0: <ACPI CPU> on acpi0
uart0: <PrimeCell UART (PL011)> iomem 0x9000000-0x9000fff irq 0 on acpi0
uart0: console (9600,n,8,1)
pcib0: <Generic PCI host controller> on acpi0
pci0: <PCI bus> on pcib0
virtio_pci0: <VirtIO PCI (modern) GPU adapter> mem 0x10119000-0x10119fff,0x8000100000-0x8000103fff at device 1.0 on pci0
virtio_pci0: cannot alloc resource for COMMON_CFG
virtio_pci0: cannot map configs
device_attach: virtio_pci0 attach returned 22
xhci0: <XHCI (generic) USB 3.0 controller> mem 0x8000108000-0x800010bfff at device 2.0 on pci0
xhci0: 32 bytes context size, 64-bit DMA
usbus0 on xhci0
virtio_pci0: <VirtIO PCI (legacy) Network adapter> mem 0x10118000-0x10118fff,0x8000104000-0x8000107fff at device 3.0 on pci0
vtnet0: <VirtIO Networking Adapter> on virtio_pci0
vtnet0: Ethernet address: 02:00:17:00:32:6a
pcib1: <PCI-PCI bridge> mem 0x10117000-0x10117fff at device 16.0 on pci0
pci1: <PCI bus> on pcib1
pcib2: <PCI-PCI bridge> mem 0x10116000-0x10116fff at device 16.1 on pci0
pci2: <PCI bus> on pcib2
pcib3: <PCI-PCI bridge> mem 0x10115000-0x10115fff at device 16.2 on pci0
pci3: <PCI bus> on pcib3
pcib4: <PCI-PCI bridge> mem 0x10114000-0x10114fff at device 16.3 on pci0
pci4: <PCI bus> on pcib4
pcib5: <PCI-PCI bridge> mem 0x10113000-0x10113fff at device 16.4 on pci0
pci5: <PCI bus> on pcib5
pcib6: <PCI-PCI bridge> mem 0x10112000-0x10112fff at device 16.5 on pci0
pci6: <PCI bus> on pcib6
pcib7: <PCI-PCI bridge> mem 0x10111000-0x10111fff at device 16.6 on pci0
pci7: <PCI bus> on pcib7
pcib8: <PCI-PCI bridge> mem 0x10110000-0x10110fff at device 16.7 on pci0
pci8: <PCI bus> on pcib8
pcib9: <PCI-PCI bridge> mem 0x1010f000-0x1010ffff at device 17.0 on pci0
pci9: <PCI bus> on pcib9
pcib10: <PCI-PCI bridge> mem 0x1010e000-0x1010efff at device 17.1 on pci0
pci10: <PCI bus> on pcib10
pcib11: <PCI-PCI bridge> mem 0x1010d000-0x1010dfff at device 17.2 on pci0
pci11: <PCI bus> on pcib11
pcib12: <PCI-PCI bridge> mem 0x1010c000-0x1010cfff at device 17.3 on pci0
pci12: <PCI bus> on pcib12
pcib13: <PCI-PCI bridge> mem 0x1010b000-0x1010bfff at device 17.4 on pci0
pci13: <PCI bus> on pcib13
pcib14: <PCI-PCI bridge> mem 0x1010a000-0x1010afff at device 17.5 on pci0
pci14: <PCI bus> on pcib14
pcib15: <PCI-PCI bridge> mem 0x10109000-0x10109fff at device 17.6 on pci0
pci15: <PCI bus> on pcib15
pcib16: <PCI-PCI bridge> mem 0x10108000-0x10108fff at device 17.7 on pci0
pci16: <PCI bus> on pcib16
pcib17: <PCI-PCI bridge> mem 0x10107000-0x10107fff at device 18.0 on pci0
pci17: <PCI bus> on pcib17
pcib18: <PCI-PCI bridge> mem 0x10106000-0x10106fff at device 18.1 on pci0
pci18: <PCI bus> on pcib18
pcib19: <PCI-PCI bridge> mem 0x10105000-0x10105fff at device 18.2 on pci0
pci19: <PCI bus> on pcib19
pcib20: <PCI-PCI bridge> mem 0x10104000-0x10104fff at device 18.3 on pci0
pci20: <PCI bus> on pcib20
pcib21: <PCI-PCI bridge> mem 0x10103000-0x10103fff at device 18.4 on pci0
pci21: <PCI bus> on pcib21
pcib22: <PCI-PCI bridge> mem 0x10102000-0x10102fff at device 18.5 on pci0
pci22: <PCI bus> on pcib22
pcib23: <PCI-PCI bridge> mem 0x10101000-0x10101fff at device 18.6 on pci0
pci23: <PCI bus> on pcib23
pcib24: <PCI-PCI bridge> mem 0x10100000-0x10100fff at device 18.7 on pci0
pci24: <PCI bus> on pcib24
virtio_pci1: <VirtIO PCI (modern) SCSI adapter> mem 0x10000000-0x10000fff,0x8000000000-0x8000003fff at device 0.0 on pci24
virtio_pci1: cannot alloc resource for COMMON_CFG
virtio_pci1: cannot map configs
device_attach: virtio_pci1 attach returned 22
acpi_button0: <Power Button> on acpi0
cryptosoft0: <software crypto>
armv8crypto0: <AES-CBC,AES-XTS,AES-GCM>
Timecounters tick every 1.000 msec
CPU  0: ARM Neoverse-N1 r3p1 affinity:  0
                   Cache Type = <64 byte D-cacheline,64 byte I-cacheline,PIPT ICache,64 byte ERG,64 byte CWG,IDC>
 Instruction Set Attributes 0 = <DP,RDM,Atomic,CRC32,SHA2,SHA1,AES+PMULL>
 Instruction Set Attributes 1 = <RCPC-8.3,DCPoP>
         Processor Features 0 = <CSV3,CSV2,RASv1,GIC,AdvSIMD+HP,FP+HP,EL3,EL2,EL1,EL0 32>
         Processor Features 1 = <PSTATE.SSBS MSR>
      Memory Model Features 0 = <TGran4,TGran64,TGran16,SNSMem,BigEnd,16bit ASID,256TB PA>
      Memory Model Features 1 = <PAN+ATS1E1,LO,HPD+TTPBHA,VH,16bit VMID,HAF+DS>
      Memory Model Features 2 = <32bit CCIDX,48bit VA,UAO,CnP>
             Debug Features 0 = <SPE,2 CTX BKPTs,4 Watchpoints,6 Breakpoints,PMUv3+16 bit evtCount,Debugv8.2>
             Debug Features 1 = <>
         Auxiliary Features 0 = <>
         Auxiliary Features 1 = <>
usbus0: 5.0Gbps Super Speed USB v3.0
Trying to mount root from ufs:/dev/ufs/FreeBSD_Install [ro,noatime]...
Root mount waiting for:ugen0.1: <0x1b36 XHCI root HUB> at usbus0
uhub0 on usbus0
uhub0: <0x1b36 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
 usbus0 CAM
uhub0: 8 ports with 8 removable, self powered
Root mount waiting for: usbus0
ugen0.2: <QEMU QEMU USB Tablet> at usbus0
mountroot: waiting for device /dev/ufs/FreeBSD_Install...
Mounting from ufs:/dev/ufs/FreeBSD_Install failed with error 19.

Loader variables:
  vfs.root.mountfrom=ufs:/dev/ufs/FreeBSD_Install
  vfs.root.mountfrom.options=ro,noatime

Manual root filesystem specification:
  <fstype>:<device> [options]
      Mount <device> using filesystem <fstype>
      and with the specified (optional) option list.

    eg. ufs:/dev/da0s1a
        zfs:zroot/ROOT/default
        cd9660:/dev/cd0 ro
          (which is equivalent to: mount -t cd9660 -o ro /dev/cd0 /)

  ?               List valid disk boot devices
  .               Yield 1 second (for background tasks)
  <empty line>    Abort manual input

mountroot>

At this point, I fiddled with the loader, but made no progress. Whatever image we boot from, I can’t get past this point. Maybe the info below helps somebody else.

Type '?' for a list of commands, 'help' for more detailed help.
OK configuration
NumberOfTableEntries=9
  fc1bcdb0-7d31-49aa-936a-a4600d9dd083 at 0x633a37998
  DXE Table at 0x637528180
  HOB List Table at 0x633a35018
  MemoryTypeInformation at 0x637528c38
  Debug Image Info Table at 0x637529520
  a4ee0728-e5d7-4ac5-b21e-658ed857e834 at 0x633fdf698
  SMBIOS3 Table at 0x633ed0000
  dcfa911d-26eb-469f-a220-38b7dc461220 at 0x632ae7318
  ACPI 2.0 Table at 0x6305f0018
OK 

OK lsdev
disk devices:
    disk0:    97677312 X 512 blocks
      disk0p1: EFI
      disk0p2: FreeBSD UFS
http: (unknown)
net devices:

OK ls /
/
 d  lib
 d  boot
 d  root
 d  proc
 d  etc
 d  net
    .profile
 d  libexec
 d  media
 d  usr
 d  bin
 d  rescue
 d  var
 d  dev
 d  sbin
 d  mnt
    .cshrc
    COPYRIGHT
 d  tmp

OK lsefi
Handle 0x633a37f98
  loaded image
Handle 0x633a37a18
  decompress
Handle 0x633a2db18
  firmware volume2
  device path
  firmware volume block
Handle 0x633a2dd98
  firmware volume2
  device path
  firmware volume block
Handle 0x633a2a218
  fc1bcdb0-7d31-49aa-936a-a4600d9dd083
Handle 0x633017498
  loaded image device path
  loaded image
Handle 0x632fc6c18
  05c99a21-c70f-4ad2-8a5f-35df3343f51e
  device path to text
  0379be4e-d706-437d-b037-edb82fb772a4
Handle 0x632fc6b18
  loaded image device path
  loaded image
Handle 0x632fdeb18
  fd0f4478-0efd-461d-ba2d-e58c45fd5f5e
  5be40f57-fa68-4610-bbbf-e9c5fcdad365
  EFI PCD
  PCD
Handle 0x632fde818
  e11faca0-4710-4c8e-a7a2-01baa2591b4c
  loaded image device path
  loaded image
Handle 0x632fd0a98
  loaded image device path
  loaded image
Handle 0x632fd0398
  Runtime arch
Handle 0x632fd0518
  loaded image device path
  loaded image
Handle 0x632fcfe98
  Security arch
  Security2 arch
Handle 0x632fcfb18
  15853d7c-3ddf-43e0-a1cb-ebf85b8f872c
Handle 0x632fcf798
  loaded image device path
  loaded image
Handle 0x632fd3c18
  metronome arch
Handle 0x632fd3998
  loaded image device path
  loaded image
Handle 0x632fd3418
  1a1241e6-8f19-41a9-bc0e-e8ef39e06546
  HII image
  0a8badd5-03b8-4d19-b128-7b8f0edaa596
  HII config routing
  HII database
  HII string
  HII font
Handle 0x632fdd098
  loaded image device path
  loaded image
Handle 0x632fdd718
  device path
  serial io
Handle 0x632fdd418
  loaded image device path
  loaded image
Handle 0x632fdca18
... boring

OK memmap
                   Type     Physical      Virtual   #Pages Attr
     ConventionalMemory 000040000000 000000000000 005e83dd WB
             LoaderData 0006283dd000 000000000000 00008000 WB
             LoaderCode 0006303dd000 000000000000 00000134 WB
       BootServicesCode 000630511000 000000000000 000000df WB
      ACPIReclaimMemory 0006305f0000 000000000000 00000010 WB
     ConventionalMemory 000630600000 000000000000 00000006 WB
       BootServicesCode 000630606000 000000000000 000000aa WB
    RuntimeServicesData 0006306b0000 000000000000 00000050 WB
    RuntimeServicesCode 000630700000 000000000000 00000050 WB
    RuntimeServicesData 000630750000 000000000000 00000050 WB
    RuntimeServicesCode 0006307a0000 000000000000 00000050 WB
    RuntimeServicesData 0006307f0000 000000000000 000000c0 WB
    RuntimeServicesCode 0006308b0000 000000000000 00000060 WB
    RuntimeServicesData 000630910000 000000000000 000000a0 WB
    RuntimeServicesCode 0006309b0000 000000000000 00000050 WB
    RuntimeServicesData 000630a00000 000000000000 00000050 WB
    RuntimeServicesCode 000630a50000 000000000000 000000f0 WB
     ConventionalMemory 000630b40000 000000000000 00000fe1 WB
       BootServicesData 000631b21000 000000000000 00001f17 WB
     ConventionalMemory 000633a38000 000000000000 0000000a WB
       BootServicesCode 000633a42000 000000000000 000003de WB
    RuntimeServicesCode 000633e20000 000000000000 00000090 WB
     ConventionalMemory 000633eb0000 000000000000 00000010 WB
    RuntimeServicesData 000633ec0000 000000000000 00000120 WB
     ConventionalMemory 000633fe0000 000000000000 0000001f WB
       BootServicesData 000633fff000 000000000000 00000021 WB
     ConventionalMemory 000634020000 000000000000 0000349d WB
       BootServicesData 0006374bd000 000000000000 00000021 WB
       BootServicesCode 0006374de000 000000000000 00000069 WB
       BootServicesData 000637547000 000000000000 0000096f WB
       BootServicesCode 000637eb6000 000000000000 00000012 WB
       BootServicesData 000637ec8000 000000000000 00000101 WB
       BootServicesCode 000637fc9000 000000000000 00000032 WB
       BootServicesData 000637ffb000 000000000000 00000005 WB
         MemoryMappedIO 000004000000 000000000000 00004000 UC
         MemoryMappedIO 000009010000 000000000000 00000001 UC

OK efi-show
global BS,RS OsIndicationsSupported = 0x41
global BS,RS BootOptionSupport = 0x313
global BS,RS LangCodes = engfraengfra
global BS,RS PlatformLangCodes = en;fr;en-US;fr-FR
global BS,RS PlatformRecovery0000 = 
01 00 00 00  36 00 44 00  65 00 66 00  61 00 75 00  6c 00 74 00  
20 00 50 00  6c 00 61 00  74 00 66 00  6f 00 72 00  6d 00 52 00  
65 00 63 00  6f 00 76 00  65 00 72 00  79 00 00 00  04 04 32 00  
5c 00 45 00  46 00 49 00  5c 00 42 00  4f 00 4f 00  54 00 5c 00  
42 00 4f 00  4f 00 54 00  41 00 41 00  36 00 34 00  2e 00 45 00  
46 00 49 00  00 00 7f ff  04 00 
global BS,RS ConInDev = VenHw(D3987D4B-971A-435F-8CAF-4967EB627241)/Uart(38400,8,N,1)/VenPcAnsi(),/VenHw(D3987D4B-971A-435F-8CAF-4967EB627241)/Uart(38400,8,N,1)/VenVt100(),/VenHw(D3987D4B-971A-435F-8CAF-4967EB627241)/Uart(38400,8,N,1)/VenVt100Plus(),/VenHw(D3987D4B-971A-435F-8CAF-4967EB627241)/Uart(38400,8,N,1)/VenUtf8(),/VenHw(D3987D4B-971A-435F-8CAF-4967EB627241)/Uart(38400,8,N,1)/VenMsg(7D916D80-5BB1-458C-A48F-E25FDD51EF94),/VenHw(D3987D4B-971A-435F-8CAF-4967EB627241)/Uart(38400,8,N,1)/VenMsg(E4364A7F-F825-430E-9D3A-9C9BE6817CA5),/VenHw(D3987D4B-971A-435F-8CAF-4967EB627241)/Uart(38400,8,N,1)/VenMsg(FBFCA56B-BB36-4B78-AAAB-BE1B97EC7CCB),/VenHw(D3987D4B-971A-435F-8CAF-4967EB627241)/Uart(38400,8,N,1)/VenMsg(8E46DDDD-3D49-4A9D-B875-3C086F6AA2BD),/VenHw(D3987D4B-971A-435F-8CAF-4967EB627241)/Uart(38400,8,N,1)/VenMsg(FC7DD6E0-813C-434D-B4DA-3BD649E9E15A)
global BS,RS ConOutDev = PciRoot(0x0)/Pci(0x1,0x0)/AcpiAdr(0x80010300),/VenHw(D3987D4B-971A-435F-8CAF-4967EB627241)/Uart(38400,8,N,1)/VenPcAnsi(),/VenHw(D3987D4B-971A-435F-8CAF-4967EB627241)/Uart(38400,8,N,1)/VenVt100(),/VenHw(D3987D4B-971A-435F-8CAF-4967EB627241)/Uart(38400,8,N,1)/VenVt100Plus(),/VenHw(D3987D4B-971A-435F-8CAF-4967EB627241)/Uart(38400,8,N,1)/VenUtf8(),/VenHw(D3987D4B-971A-435F-8CAF-4967EB627241)/Uart(38400,8,N,1)/VenMsg(7D916D80-5BB1-458C-A48F-E25FDD51EF94),/VenHw(D3987D4B-971A-435F-8CAF-4967EB627241)/Uart(38400,8,N,1)/VenMsg(E4364A7F-F825-430E-9D3A-9C9BE6817CA5),/VenHw(D3987D4B-971A-435F-8CAF-4967EB627241)/Uart(38400,8,N,1)/VenMsg(FBFCA56B-BB36-4B78-AAAB-BE1B97EC7CCB),/VenHw(D3987D4B-971A-435F-8CAF-4967EB627241)/Uart(38400,8,N,1)/VenMsg(8E46DDDD-3D49-4A9D-B875-3C086F6AA2BD),/VenHw(D3987D4B-971A-435F-8CAF-4967EB627241)/Uart(38400,8,N,1)/VenMsg(FC7DD6E0-813C-434D-B4DA-3BD649E9E15A)
global BS,RS ErrOutDev = PciRoot(0x0)/Pci(0x1,0x0)/AcpiAdr(0x80010300),/VenHw(D3987D4B-971A-435F-8CAF-4967EB627241)/Uart(38400,8,N,1)/VenPcAnsi(),/VenHw(D3987D4B-971A-435F-8CAF-4967EB627241)/Uart(38400,8,N,1)/VenVt100(),/VenHw(D3987D4B-971A-435F-8CAF-4967EB627241)/Uart(38400,8,N,1)/VenVt100Plus(),/VenHw(D3987D4B-971A-435F-8CAF-4967EB627241)/Uart(38400,8,N,1)/VenUtf8(),/VenHw(D3987D4B-971A-435F-8CAF-4967EB627241)/Uart(38400,8,N,1)/VenMsg(7D916D80-5BB1-458C-A48F-E25FDD51EF94),/VenHw(D3987D4B-971A-435F-8CAF-4967EB627241)/Uart(38400,8,N,1)/VenMsg(E4364A7F-F825-430E-9D3A-9C9BE6817CA5),/VenHw(D3987D4B-971A-435F-8CAF-4967EB627241)/Uart(38400,8,N,1)/VenMsg(FBFCA56B-BB36-4B78-AAAB-BE1B97EC7CCB),/VenHw(D3987D4B-971A-435F-8CAF-4967EB627241)/Uart(38400,8,N,1)/VenMsg(8E46DDDD-3D49-4A9D-B875-3C086F6AA2BD),/VenHw(D3987D4B-971A-435F-8CAF-4967EB627241)/Uart(38400,8,N,1)/VenMsg(FC7DD6E0-813C-434D-B4DA-3BD649E9E15A)
global BS,RS BootCurrent = 2
freebsd BS,RS LoaderPath = \EFI\BOOT\BOOTAA64.EFI
freebsd BS,RS LoaderDev = PciRoot(0x0)/Pci(0x12,0x7)/Pci(0x0,0x0)/Scsi(0x0,0x1)/HD(1,GPT,FCE27ED8-9904-11EB-BF4C-002590EC5BF2,0x3,0x10418)
59324945-ec44-4c0d-b1cd-9db139df070c NV,BS Attempt 1 = 
00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00  00 01 00 00  
00 00 00 00  00 00 41 74  74 65 6d 70  74 20 31 00  00 00 00 00  
... boring

global NV,BS,RS Boot0000 = UiApp
        Fv(64074AFE-340A-4BE6-94BA-91B5B4D0F71E)/FvFile(462CAA21-7614-4503-836E-8AB6F4662331)
global NV,BS,RS Timeout = 3
global NV,BS,RS PlatformLang = en
global NV,BS,RS Lang = eng
04b37fe8-f6ae-480b-bdd5-37d98c5e89aa NV,BS,RS VarErrorFlag = 
ff 
global NV,BS,RS ConIn = UsbHID(0xFFFF,0xFFFF,0x1,0x1),/VenHw(D3987D4B-971A-435F-8CAF-4967EB627241)/Uart(38400,8,N,1)/VenVt100()
global NV,BS,RS ConOut = PciRoot(0x0)/Pci(0x1,0x0)/AcpiAdr(0x80010300),/VenHw(D3987D4B-971A-435F-8CAF-4967EB627241)/Uart(38400,8,N,1)/VenVt100()
global NV,BS,RS ErrOut = PciRoot(0x0)/Pci(0x1,0x0)/AcpiAdr(0x80010300),/VenHw(D3987D4B-971A-435F-8CAF-4967EB627241)/Uart(38400,8,N,1)/VenVt100()
global NV,BS,RS Key0000 = 
00 00 00 40  5d 4d 80 dd  00 00 0c 00  00 00 
global NV,BS,RS Key0001 = 
00 00 00 40  5d 4d 80 dd  00 00 17 00  00 00 
5b446ed1-e30b-4faa-871a-3654eca36080 NV,BS 020017007935 = 
3a 50 01 00  10 00 af af  04 00 00 00  01 00 00 00  00 00 00 00  

global NV,BS,RS Boot0001 = UEFI Misc Device
        VenHw(93E34C7E-B50E-11DF-9223-2443DFD72085,00)
global NV,BS,RS Boot0002 = UEFI ORACLE BlockVolume 
        PciRoot(0x0)/Pci(0x12,0x7)/Pci(0x0,0x0)/Scsi(0x0,0x1)
global NV,BS,RS Boot0004 = EFI Internal Shell
        Fv(64074AFE-340A-4BE6-94BA-91B5B4D0F71E)/FvFile(7C04A583-9E3E-4F1C-AD65-E05268D0B4D1)
MemoryTypeInformation NV,BS MemoryTypeInformation = 
              ACPIReclaimMemory pages: 20
                  ACPIMemoryNVS pages: 40
global NV,BS,RS BootOrder = 2 0 1 4
MTC NV,BS,RS MTC = 4

Terraform Stacks

When you do the manual A1.Flex launch, you’ll notice an option to save the stack. If you do this, you can download a zip file, which contains a terraform provisioning script.

This encapsulates the usual network and image settings, so yours won’t be the same as mine, here’s what I have:

provider "oci" {}

resource "oci_core_instance" "generated_oci_core_instance" {
	agent_config {
		is_management_disabled = "true"
		is_monitoring_disabled = "true"
		plugins_config {
			desired_state = "DISABLED"
			name = "Compute Instance Monitoring"
		}
	}
	availability_config {
		recovery_action = "RESTORE_INSTANCE"
	}
	availability_domain = "Ynwz:eu-amsterdam-1-AD-1"
	compartment_id = "ocid1.tenancy.oc1..blah"
	create_vnic_details {
		assign_private_dns_record = "true"
		assign_public_ip = "true"
		subnet_id = "ocid1.subnet.oc1.eu-amsterdam-1.blah"
	}
	display_name = "beastie"
	instance_options {
		are_legacy_imds_endpoints_disabled = "false"
	}
	metadata = {
		"ssh_authorized_keys" = "ssh-ed25519 AAAAC3Nzblah"
	}
	shape = "VM.Standard.A1.Flex"
	shape_config {
		memory_in_gbs = "24"
		ocpus = "4"
	}
	source_details {
		source_id = "ocid1.image.oc1.eu-amsterdam-blah"
		source_type = "image"
	}
}

Setting up terraform on FreeBSD is extraordinarily easy compared to ports. Run this in the same folder as your unzipped main.tf config.

I skimmed this OCI Terraform tutorial and managed to get far enough along to automate spinning up systems. This is an excellent approach.

$ sudo pkg install -yr FreeBSD \
    sysutils/tflint \
    sysutils/terragrunt \
    sysutils/terraform \
    sysutils/terraform-docs
...

$ terraform init

Initializing the backend...

Initializing provider plugins...
- Finding latest version of hashicorp/oci...
- Installing hashicorp/oci v4.33.0...
- Installed hashicorp/oci v4.33.0 (signed by HashiCorp)

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

$ terraform apply

Terraform used the selected providers to generate the following
execution plan. Resource actions are indicated with the following
symbols:
  + create

Terraform will perform the following actions:

  # oci_core_instance.generated_oci_core_instance will be created
  + resource "oci_core_instance" "generated_oci_core_instance" {
      + availability_domain                 = "Ynwz:eu-amsterdam-1-AD-1"
      + boot_volume_id                      = (known after apply)
      + capacity_reservation_id             = (known after apply)
      + compartment_id                      = "ocid1.tenancy.oc1..aaaaaaaaoblah"
      + dedicated_vm_host_id                = (known after apply)
      + defined_tags                        = (known after apply)
      + display_name                        = "beastie"
      + fault_domain                        = (known after apply)
      + freeform_tags                       = (known after apply)
      + hostname_label                      = (known after apply)
      + id                                  = (known after apply)
      + image                               = (known after apply)
      + ipxe_script                         = (known after apply)
      + is_pv_encryption_in_transit_enabled = (known after apply)
      + launch_mode                         = (known after apply)
      + metadata                            = {
          + "ssh_authorized_keys" = "ssh-ed25519 AAAAC3Nblah"
        }
      + private_ip                          = (known after apply)
      + public_ip                           = (known after apply)
      + region                              = (known after apply)
      + shape                               = "VM.Standard.A1.Flex"
      + state                               = (known after apply)
      + subnet_id                           = (known after apply)
      + system_tags                         = (known after apply)
      + time_created                        = (known after apply)
      + time_maintenance_reboot_due         = (known after apply)

      + agent_config {
          + are_all_plugins_disabled = (known after apply)
          + is_management_disabled   = true
          + is_monitoring_disabled   = true

          + plugins_config {
              + desired_state = "DISABLED"
              + name          = "Compute Instance Monitoring"
            }
        }

      + availability_config {
          + is_live_migration_preferred = (known after apply)
          + recovery_action             = "RESTORE_INSTANCE"
        }

      + create_vnic_details {
          + assign_private_dns_record = true
          + assign_public_ip          = "true"
          + defined_tags              = (known after apply)
          + display_name              = (known after apply)
          + freeform_tags             = (known after apply)
          + hostname_label            = (known after apply)
          + private_ip                = (known after apply)
          + skip_source_dest_check    = (known after apply)
          + subnet_id                 = "ocid1.subnet.oc1.eu-amsterdam-1.blah"
          + vlan_id                   = (known after apply)
        }

      + instance_options {
          + are_legacy_imds_endpoints_disabled = false
        }

      + launch_options {
          + boot_volume_type                    = (known after apply)
          + firmware                            = (known after apply)
          + is_consistent_volume_naming_enabled = (known after apply)
          + is_pv_encryption_in_transit_enabled = (known after apply)
          + network_type                        = (known after apply)
          + remote_data_volume_type             = (known after apply)
        }

      + platform_config {
          + numa_nodes_per_socket = (known after apply)
          + type                  = (known after apply)
        }

      + preemptible_instance_config {
          + preemption_action {
              + preserve_boot_volume = (known after apply)
              + type                 = (known after apply)
            }
        }

      + shape_config {
          + baseline_ocpu_utilization     = (known after apply)
          + gpu_description               = (known after apply)
          + gpus                          = (known after apply)
          + local_disk_description        = (known after apply)
          + local_disks                   = (known after apply)
          + local_disks_total_size_in_gbs = (known after apply)
          + max_vnic_attachments          = (known after apply)
          + memory_in_gbs                 = 4
          + networking_bandwidth_in_gbps  = (known after apply)
          + ocpus                         = 1
          + processor_description         = (known after apply)
        }

      + source_details {
          + boot_volume_size_in_gbs = (known after apply)
          + kms_key_id              = (known after apply)
          + source_id               =
          "ocid1.image.oc1.eu-amsterdam-1.blah"
          + source_type             = "image"
        }
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

oci_core_instance.generated_oci_core_instance: Creating...
oci_core_instance.generated_oci_core_instance: Still creating... [10s elapsed]
oci_core_instance.generated_oci_core_instance: Still creating... [20s elapsed]
oci_core_instance.generated_oci_core_instance: Still creating... [30s elapsed]
oci_core_instance.generated_oci_core_instance: Still creating... [40s elapsed]
oci_core_instance.generated_oci_core_instance: Still creating... [50s elapsed]
oci_core_instance.generated_oci_core_instance: Still creating... [1m0s elapsed]
oci_core_instance.generated_oci_core_instance: Still creating... [1m10s elapsed]
oci_core_instance.generated_oci_core_instance: Still creating... [1m20s elapsed]
oci_core_instance.generated_oci_core_instance: Still creating... [1m30s elapsed]
oci_core_instance.generated_oci_core_instance: Still creating... [1m40s elapsed]
oci_core_instance.generated_oci_core_instance: Still creating... [1m50s elapsed]
oci_core_instance.generated_oci_core_instance: Still creating... [2m0s elapsed]
oci_core_instance.generated_oci_core_instance: Creation complete after 2m8s [id=ocid1.instance.oc1.eu-amsterdam-1.blah

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Once the terraforming is complete, go to your OCI console, find the image, and create a Console Connection again as before.

Things I Want To Try

I’d like to get ipxe booting directly. This is mainly because uploading an image each to OCI is slow and painful.

Also, understanding why the loader bails with error 19 and addressing that would be even better.