Booting FreeBSD on Ampere's Awesome Altra CPUs
Friday, 2 Jul 2021
This is a series of posts exploring how we can get FreeBSD to boot up in Oracle Cloud Infrastructure OCI, which is the commercially available source of the 160-core dual socket Altra CPUs from Ampere.
Geek Envy
Yes, 160 cores on a single system. These are not your scabby x86
Symmetric Hyperthreading cores, each one is a fully capable 3.3GHz
behemoth. Here are the specs from the initial Altra announcement:
Altra® Multi Core Server Processors
Ampere server processors are a complete system on chip (SOC) solution built for Cloud Native server applications. The device supports up to 128 Arm 64 bit cores, 8 Channels of DDR4 and 128 Channels of PCIe Gen 4 interfaces.
- Up to 128 cloud optimized 64 bit Armv8.2 cores
- Up to 3.3GHz frequency of operation
- 8×72 bit DDR4 3200, ECC Support, 4TB Capacity
- 128 PCIe Gen4 lanes
- Coherent multi socket support
- I/O and Interrupt Virtualization
- Enterprise server class RAS
There is some performance propaganda available too.
Support
The arm64 platform has had tier 1 support since FreeBSD 13.0-RELEASE,
and I have been running it very happily on the “small” eMAG systems
from Ampere for many months. When I say “small” it only has 32 cores,
runs at a leisurely 3.0GHz and my install only has a single PCIe4
NVMe drive, with just 256GiB RAM… so only small in comparison to its
newer, bigger twin brothers!
We’ve had access to the newer Altra system hardware already, and the functionality is identical.
The performance is excellent, all the nice FreeBSD bits like ZFS, jails, and firewalls just work, as you’d expect. Some functionality (full IPMI, various sensors, including cpu frequency & temperature sysctls), are not yet exposed but all the important stuff - networking, storage, remote console, is fantastic. Of particular note is that the NICs support HTTP boot, as well as full remote console (mounting images etc), so once you’ve physically plugged the box in, you’ll rarely need to go to it.
ipxe also works, so a simple recovery solution is to configure the UEFI firmware to load ipxe over the network, and from there to pull in the FreeBSD loader via HTTP boot.
OCI
Oracle has production Altra systems available, both as para-virtualised cloud VMs, and also as bare metal behemoths. There’s a bunch of propaganda docs with some useful general info:
- migration
- performance propaganda
- Arm Accelerator Program sign up for this
We are interested in 2 things:
- boot FreeBSD on PV mode
- boot FreeBSD in BM mode
Hijacking Linux, aka de-penguinification
As a starting point, I tried “de-penguinating” an existing Linux
instance. This was easy: drop loader.efi into the EFI partition,
dd(1) the FreeBSD installer on top of the Linux swap partition, and
reboot. But, on reboot, the EFI partition mysteriously returned to
Oracle defaults. I am not sure if this is a loader issue, or an OCI
feature.
# cd /root
# curl -#LO https://download.freebsd.org/ftp/releases/arm64/aarch64/13.0-RELEASE/kernel.txz
# curl -#LO https://download.freebsd.org/ftp/releases/arm64/aarch64/13.0-RELEASE/base.txz
# tar xvJf base.txz ./boot/loader_lua.efi
# mkdir -p /boot/efi/EFI/FreeBSD
# mv ./boot/loader_lua.efi /boot/efi/EFI/FreeBSD/13.efi
# efibootmgr -c -L FreeBSD -l /boot/efi/EFI/FreeBSD/13.efi
BootCurrent: 0001
Timeout: 0 seconds
BootOrder: 0005,0004,0000,0001,0002,0003
Boot0000* UiApp
Boot0001* UEFI ORACLE BlockVolume
Boot0002* UEFI PXEv4 (MAC:020017009D5B)
Boot0003* EFI Internal Shell
Boot0004* Oracle Linux
Boot0005* FreeBSD
# efibootmgr -n 5
BootNext: 0005
BootCurrent: 0001
Timeout: 0 seconds
BootOrder: 0005,0004,0000,0001,0002,0003
Boot0000* UiApp
Boot0001* UEFI ORACLE BlockVolume
Boot0002* UEFI PXEv4 (MAC:020017009D5B)
Boot0003* EFI Internal Shell
Boot0004* Oracle Linux
Boot0005* FreeBSDI re-tried with refind, also no luck.
Bring Your Own Images
My second attempt was to upload various FreeBSD release installers into OCI Cloud Storage, then create new Bring-Your-Own bootable images from those uploaded files, and try to boot off those.
$ qemu-img convert -f raw -O qcow2 \
FreeBSD-13.0-RELEASE-arm64-aarch64-memstick.img \
FreeBSD-13.0-RELEASE-arm64-aarch64-memstick.qcow2
I also realised in hindsight that FreeBSD provides both vmdk and raw
ready-to-roll images, but I did not get as far with those.
This process is a little muddly, I will update this post in future on
exactly how to go about that. In a nutshell, qemu-img convert your
desired setup, upload it into blob storage, create a new “cloud image”,
tweak the supported shapes to only A1.Flex and A1.BM types
(Para-virtualised and Bare Metal respectively), and guess randomly the
bewildering array of options you’re presented with. I would hope that
Native eventually works, but it’s not entirely clear what underlying
hardware this actually is. I expect it’s some sort of KVM / QEMU
combined with iSCSI for storage, and VFIO for networking.
When you create the image, you need to define its capabilities, and compatible shapes:
BM.Standard.A1.160VM.Standard.A1.Flex- firmware: uefi_64
- launch mode: emulated
- nic attachment type: paravirtualised
- boot: paravirtualised
- boot volume type: paravirtualised
- local data volume: paravirtualised
- remote data volume: paravirtualised
- paravirtualisation version: 2 (using v1 also fails)
I tried creating a native image type, which got quite a bit further.
Unfortunately it’s not clear what the image source was, but I think
it’s my qemu-img converted raw FreeBSD installer.
Image: freebsd-13.0-arm64-native
Launch Mode: EMULATED
Shape: VM.Standard.A1.Flex
OCPU Count: 1
Network Bandwidth (Gbps): 1
Memory (GB): 6
Local Disk: Block Storage Only
NIC Attachment Type: PARAVIRTUALIZED
Remote Data Volume: PARAVIRTUALIZED
Firmware: UEFI_64
Boot Volume Type: PARAVIRTUALIZED
After the instance has been created, add a console connection which
is just a bunch of ssh wrapping whatever KVM provides us over serial
port, and then force restart the instance, (its dead at this point
anyway) and see the console garbage fly past.
$ ssh -o ProxyCommand='ssh -W %h:%p -p 443 ocid1.@oci.oraclecloud.com' \
ocid1.instance.oc1.eu-amsterdam-1....
The authenticity of host
'[instance-console.eu-amsterdam-1.oci.oraclecloud.com]:443
([140.204.36.211]:443)' can't be established. RSA key fingerprint is
SHA256:R4nCgf3DqENCAEJtA2e+UfkzG3dTUhgUqLvLak/Y8bk.
No matching host key fingerprint found in DNS.
This host key is known by the following other names/addresses:
~/.ssh/known_hosts:74: [instance-console.eu-amsterdam-1.oraclecloud.com]:443
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added
'[instance-console.eu-amsterdam-1.oci.oraclecloud.com]:443' (RSA) to the
list of known hosts.
=================================================
IMPORTANT: Use a console connection to troubleshoot a malfunctioning
instance. For normal operations, you should connect to the instance
using a Secure Shell (SSH) or Remote Desktop connection. For steps, see
https://docs.cloud.oracle.com/iaas/Content/Compute/Tasks/accessinginstance.htm
For more information about troubleshooting your instance using a console
connection, see the documentation:
https://docs.cloud.oracle.com/en-us/iaas/Content/Compute/References/serialconsole.htm#four
=================================================
The authenticity of host 'ocid1.instance.oc1.eu-amsterdam-1...
(<no hostip for proxy command>)' can't be established. RSA key
fingerprint is SHA256:cVF7Hw5kRv4gyaxZwj0yqAWUlvU2LwW5VXJ124LgxhI.
No matching host key fingerprint found in DNS.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added
'ocid1.instance.oc1.eu-amsterdam-1....'
(RSA) to the list of known hosts.
I could get to the loader prompt, as far as the infamous error 19:
mountroot: waiting for device /dev/ufs/FreeBSD_Install...
Mounting from ufs:/dev/ufs/FreeBSD_Install failed with error 19.This presumably fail without appropriate iSCSI or similar drivers already available to the loader itself. This may require building a custom FreeBSD kernel with the drivers already embedded into loader.efi, if that’s even possible.
The full dmesg, as far as we get, follows:
______ ____ _____ _____
| ____| | _ \ / ____| __ \
| |___ _ __ ___ ___ | |_) | (___ | | | |
| ___| '__/ _ \/ _ \| _ < \___ \| | | |
| | | | | __/ __/| |_) |____) | |__| |
| | | | | | || | | |
|_| |_| \___|\___||____/|_____/|_____/ ``` `
s` `.....---.......--.``` -/
/-----------Welcome to FreeBSD------------\ +o .--` /y:` +.
| | yo`:. :o `+-
| 1. Boot Multi user [Enter] | y/ -/` -o/
| 2. Boot Single user | .- ::/sy+:.
| 3. Escape to loader prompt | / `-- /
| 4. Reboot | `: :`
| 5. Cons: Dual (Video primary) | `: :`
| | / /
| Options: | .- -.
| 6. Kernel: default/kernel (1 of 1) | -- -.
| 7. Boot Options | `:` `:`
| | .-- `--.
| | .---.....----.
\-----------------------------------------/
Autoboot in 0 seconds, hit [Enter] to boot or any other key to stop
Loading kernel...
/boot/kernel/kernel text=0x2a8 text=0x8a7b2c text=0x1e9934 data=0x1a31c8 data=0x0+0x37f000 syms=[0x8+0x117420+0x8+0x13cea2]
Loading configured modules...
can't find '/boot/entropy'
can't find '/etc/hostid'
No valid device tree blob found!
WARNING! Trying to fire up the kernel, but no device tree blob found!
EFI framebuffer information:
addr, size 0x0, 0x0
dimensions 800 x 600
stride 800
masks 0x000000ff, 0x0000ff00, 0x00ff0000, 0xff000000
---<<BOOT>>---
Copyright (c) 1992-2021 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 13.0-RELEASE #0 releng/13.0-n244733-ea31abc261f: Fri Apr 9 03:54:53 UTC 2021
root@releng1.nyi.freebsd.org:/usr/obj/usr/src/arm64.aarch64/sys/GENERIC arm64
FreeBSD clang version 11.0.1 (git@github.com:llvm/llvm-project.git llvmorg-11.0.1-0-g43ff75f2c3fe)
VT: init without driver.
module firmware already present!
real memory = 6305218560 (6013 MB)
avail memory = 6123745280 (5840 MB)
FreeBSD/SMP: Multiprocessor System Detected: 1 CPUs
arc4random: WARNING: initial seeding bypassed the cryptographic random device because it was not yet seeded and the knob 'bypass_before_seeding' was enabled.
random: entropy device external interface
MAP 1b06b0000 mode 2 pages 80
MAP 1b0700000 mode 2 pages 80
MAP 1b0750000 mode 2 pages 80
MAP 1b07a0000 mode 2 pages 80
MAP 1b07f0000 mode 2 pages 192
MAP 1b08b0000 mode 2 pages 96
MAP 1b0910000 mode 2 pages 160
MAP 1b09b0000 mode 2 pages 80
MAP 1b0a00000 mode 2 pages 80
MAP 1b0a50000 mode 2 pages 240
MAP 1b3e20000 mode 2 pages 144
MAP 1b3ec0000 mode 2 pages 288
MAP 4000000 mode 0 pages 16384
MAP 9010000 mode 0 pages 1
WARNING: Device "openfirm" is Giant locked and may be deleted before FreeBSD 14.0.
WARNING: Device "kbd" is Giant locked and may be deleted before FreeBSD 14.0.
kbd0 at kbdmux0
acpi0: <BOCHS BXPCFACP>
acpi0: Power Button (fixed)
acpi0: Sleep Button (fixed)
acpi0: Could not update all GPEs: AE_NOT_CONFIGURED
psci0: <ARM Power State Co-ordination Interface Driver> on acpi0
gic0: <ARM Generic Interrupt Controller v3.0> iomem 0x8000000-0x801ffff,0x80a0000-0x8ffffff on acpi0
its0: <ARM GIC Interrupt Translation Service> on gic0
generic_timer0: <ARM Generic Timer> irq 34,35,36 on acpi0
Timecounter "ARM MPCore Timecounter" frequency 25000000 Hz quality 1000
Event timer "ARM MPCore Eventtimer" frequency 25000000 Hz quality 1000
efirtc0: <EFI Realtime Clock>
efirtc0: registered as a time-of-day clock, resolution 1.000000s
cpu0: <ACPI CPU> on acpi0
uart0: <PrimeCell UART (PL011)> iomem 0x9000000-0x9000fff irq 0 on acpi0
uart0: console (9600,n,8,1)
pcib0: <Generic PCI host controller> on acpi0
pci0: <PCI bus> on pcib0
virtio_pci0: <VirtIO PCI (modern) GPU adapter> mem 0x10119000-0x10119fff,0x8000100000-0x8000103fff at device 1.0 on pci0
virtio_pci0: cannot alloc resource for COMMON_CFG
virtio_pci0: cannot map configs
device_attach: virtio_pci0 attach returned 22
xhci0: <XHCI (generic) USB 3.0 controller> mem 0x8000108000-0x800010bfff at device 2.0 on pci0
xhci0: 32 bytes context size, 64-bit DMA
usbus0 on xhci0
virtio_pci0: <VirtIO PCI (legacy) Network adapter> mem 0x10118000-0x10118fff,0x8000104000-0x8000107fff at device 3.0 on pci0
vtnet0: <VirtIO Networking Adapter> on virtio_pci0
vtnet0: Ethernet address: 02:00:17:00:32:6a
pcib1: <PCI-PCI bridge> mem 0x10117000-0x10117fff at device 16.0 on pci0
pci1: <PCI bus> on pcib1
pcib2: <PCI-PCI bridge> mem 0x10116000-0x10116fff at device 16.1 on pci0
pci2: <PCI bus> on pcib2
pcib3: <PCI-PCI bridge> mem 0x10115000-0x10115fff at device 16.2 on pci0
pci3: <PCI bus> on pcib3
pcib4: <PCI-PCI bridge> mem 0x10114000-0x10114fff at device 16.3 on pci0
pci4: <PCI bus> on pcib4
pcib5: <PCI-PCI bridge> mem 0x10113000-0x10113fff at device 16.4 on pci0
pci5: <PCI bus> on pcib5
pcib6: <PCI-PCI bridge> mem 0x10112000-0x10112fff at device 16.5 on pci0
pci6: <PCI bus> on pcib6
pcib7: <PCI-PCI bridge> mem 0x10111000-0x10111fff at device 16.6 on pci0
pci7: <PCI bus> on pcib7
pcib8: <PCI-PCI bridge> mem 0x10110000-0x10110fff at device 16.7 on pci0
pci8: <PCI bus> on pcib8
pcib9: <PCI-PCI bridge> mem 0x1010f000-0x1010ffff at device 17.0 on pci0
pci9: <PCI bus> on pcib9
pcib10: <PCI-PCI bridge> mem 0x1010e000-0x1010efff at device 17.1 on pci0
pci10: <PCI bus> on pcib10
pcib11: <PCI-PCI bridge> mem 0x1010d000-0x1010dfff at device 17.2 on pci0
pci11: <PCI bus> on pcib11
pcib12: <PCI-PCI bridge> mem 0x1010c000-0x1010cfff at device 17.3 on pci0
pci12: <PCI bus> on pcib12
pcib13: <PCI-PCI bridge> mem 0x1010b000-0x1010bfff at device 17.4 on pci0
pci13: <PCI bus> on pcib13
pcib14: <PCI-PCI bridge> mem 0x1010a000-0x1010afff at device 17.5 on pci0
pci14: <PCI bus> on pcib14
pcib15: <PCI-PCI bridge> mem 0x10109000-0x10109fff at device 17.6 on pci0
pci15: <PCI bus> on pcib15
pcib16: <PCI-PCI bridge> mem 0x10108000-0x10108fff at device 17.7 on pci0
pci16: <PCI bus> on pcib16
pcib17: <PCI-PCI bridge> mem 0x10107000-0x10107fff at device 18.0 on pci0
pci17: <PCI bus> on pcib17
pcib18: <PCI-PCI bridge> mem 0x10106000-0x10106fff at device 18.1 on pci0
pci18: <PCI bus> on pcib18
pcib19: <PCI-PCI bridge> mem 0x10105000-0x10105fff at device 18.2 on pci0
pci19: <PCI bus> on pcib19
pcib20: <PCI-PCI bridge> mem 0x10104000-0x10104fff at device 18.3 on pci0
pci20: <PCI bus> on pcib20
pcib21: <PCI-PCI bridge> mem 0x10103000-0x10103fff at device 18.4 on pci0
pci21: <PCI bus> on pcib21
pcib22: <PCI-PCI bridge> mem 0x10102000-0x10102fff at device 18.5 on pci0
pci22: <PCI bus> on pcib22
pcib23: <PCI-PCI bridge> mem 0x10101000-0x10101fff at device 18.6 on pci0
pci23: <PCI bus> on pcib23
pcib24: <PCI-PCI bridge> mem 0x10100000-0x10100fff at device 18.7 on pci0
pci24: <PCI bus> on pcib24
virtio_pci1: <VirtIO PCI (modern) SCSI adapter> mem 0x10000000-0x10000fff,0x8000000000-0x8000003fff at device 0.0 on pci24
virtio_pci1: cannot alloc resource for COMMON_CFG
virtio_pci1: cannot map configs
device_attach: virtio_pci1 attach returned 22
acpi_button0: <Power Button> on acpi0
cryptosoft0: <software crypto>
armv8crypto0: <AES-CBC,AES-XTS,AES-GCM>
Timecounters tick every 1.000 msec
CPU 0: ARM Neoverse-N1 r3p1 affinity: 0
Cache Type = <64 byte D-cacheline,64 byte I-cacheline,PIPT ICache,64 byte ERG,64 byte CWG,IDC>
Instruction Set Attributes 0 = <DP,RDM,Atomic,CRC32,SHA2,SHA1,AES+PMULL>
Instruction Set Attributes 1 = <RCPC-8.3,DCPoP>
Processor Features 0 = <CSV3,CSV2,RASv1,GIC,AdvSIMD+HP,FP+HP,EL3,EL2,EL1,EL0 32>
Processor Features 1 = <PSTATE.SSBS MSR>
Memory Model Features 0 = <TGran4,TGran64,TGran16,SNSMem,BigEnd,16bit ASID,256TB PA>
Memory Model Features 1 = <PAN+ATS1E1,LO,HPD+TTPBHA,VH,16bit VMID,HAF+DS>
Memory Model Features 2 = <32bit CCIDX,48bit VA,UAO,CnP>
Debug Features 0 = <SPE,2 CTX BKPTs,4 Watchpoints,6 Breakpoints,PMUv3+16 bit evtCount,Debugv8.2>
Debug Features 1 = <>
Auxiliary Features 0 = <>
Auxiliary Features 1 = <>
usbus0: 5.0Gbps Super Speed USB v3.0
Trying to mount root from ufs:/dev/ufs/FreeBSD_Install [ro,noatime]...
Root mount waiting for:ugen0.1: <0x1b36 XHCI root HUB> at usbus0
uhub0 on usbus0
uhub0: <0x1b36 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
usbus0 CAM
uhub0: 8 ports with 8 removable, self powered
Root mount waiting for: usbus0
ugen0.2: <QEMU QEMU USB Tablet> at usbus0
mountroot: waiting for device /dev/ufs/FreeBSD_Install...
Mounting from ufs:/dev/ufs/FreeBSD_Install failed with error 19.
Loader variables:
vfs.root.mountfrom=ufs:/dev/ufs/FreeBSD_Install
vfs.root.mountfrom.options=ro,noatime
Manual root filesystem specification:
<fstype>:<device> [options]
Mount <device> using filesystem <fstype>
and with the specified (optional) option list.
eg. ufs:/dev/da0s1a
zfs:zroot/ROOT/default
cd9660:/dev/cd0 ro
(which is equivalent to: mount -t cd9660 -o ro /dev/cd0 /)
? List valid disk boot devices
. Yield 1 second (for background tasks)
<empty line> Abort manual input
mountroot>At this point, I fiddled with the loader, but made no progress. Whatever image we boot from, I can’t get past this point. Maybe the info below helps somebody else.
Type '?' for a list of commands, 'help' for more detailed help.
OK configuration
NumberOfTableEntries=9
fc1bcdb0-7d31-49aa-936a-a4600d9dd083 at 0x633a37998
DXE Table at 0x637528180
HOB List Table at 0x633a35018
MemoryTypeInformation at 0x637528c38
Debug Image Info Table at 0x637529520
a4ee0728-e5d7-4ac5-b21e-658ed857e834 at 0x633fdf698
SMBIOS3 Table at 0x633ed0000
dcfa911d-26eb-469f-a220-38b7dc461220 at 0x632ae7318
ACPI 2.0 Table at 0x6305f0018
OK
OK lsdev
disk devices:
disk0: 97677312 X 512 blocks
disk0p1: EFI
disk0p2: FreeBSD UFS
http: (unknown)
net devices:
OK ls /
/
d lib
d boot
d root
d proc
d etc
d net
.profile
d libexec
d media
d usr
d bin
d rescue
d var
d dev
d sbin
d mnt
.cshrc
COPYRIGHT
d tmp
OK lsefi
Handle 0x633a37f98
loaded image
Handle 0x633a37a18
decompress
Handle 0x633a2db18
firmware volume2
device path
firmware volume block
Handle 0x633a2dd98
firmware volume2
device path
firmware volume block
Handle 0x633a2a218
fc1bcdb0-7d31-49aa-936a-a4600d9dd083
Handle 0x633017498
loaded image device path
loaded image
Handle 0x632fc6c18
05c99a21-c70f-4ad2-8a5f-35df3343f51e
device path to text
0379be4e-d706-437d-b037-edb82fb772a4
Handle 0x632fc6b18
loaded image device path
loaded image
Handle 0x632fdeb18
fd0f4478-0efd-461d-ba2d-e58c45fd5f5e
5be40f57-fa68-4610-bbbf-e9c5fcdad365
EFI PCD
PCD
Handle 0x632fde818
e11faca0-4710-4c8e-a7a2-01baa2591b4c
loaded image device path
loaded image
Handle 0x632fd0a98
loaded image device path
loaded image
Handle 0x632fd0398
Runtime arch
Handle 0x632fd0518
loaded image device path
loaded image
Handle 0x632fcfe98
Security arch
Security2 arch
Handle 0x632fcfb18
15853d7c-3ddf-43e0-a1cb-ebf85b8f872c
Handle 0x632fcf798
loaded image device path
loaded image
Handle 0x632fd3c18
metronome arch
Handle 0x632fd3998
loaded image device path
loaded image
Handle 0x632fd3418
1a1241e6-8f19-41a9-bc0e-e8ef39e06546
HII image
0a8badd5-03b8-4d19-b128-7b8f0edaa596
HII config routing
HII database
HII string
HII font
Handle 0x632fdd098
loaded image device path
loaded image
Handle 0x632fdd718
device path
serial io
Handle 0x632fdd418
loaded image device path
loaded image
Handle 0x632fdca18
... boring
OK memmap
Type Physical Virtual #Pages Attr
ConventionalMemory 000040000000 000000000000 005e83dd WB
LoaderData 0006283dd000 000000000000 00008000 WB
LoaderCode 0006303dd000 000000000000 00000134 WB
BootServicesCode 000630511000 000000000000 000000df WB
ACPIReclaimMemory 0006305f0000 000000000000 00000010 WB
ConventionalMemory 000630600000 000000000000 00000006 WB
BootServicesCode 000630606000 000000000000 000000aa WB
RuntimeServicesData 0006306b0000 000000000000 00000050 WB
RuntimeServicesCode 000630700000 000000000000 00000050 WB
RuntimeServicesData 000630750000 000000000000 00000050 WB
RuntimeServicesCode 0006307a0000 000000000000 00000050 WB
RuntimeServicesData 0006307f0000 000000000000 000000c0 WB
RuntimeServicesCode 0006308b0000 000000000000 00000060 WB
RuntimeServicesData 000630910000 000000000000 000000a0 WB
RuntimeServicesCode 0006309b0000 000000000000 00000050 WB
RuntimeServicesData 000630a00000 000000000000 00000050 WB
RuntimeServicesCode 000630a50000 000000000000 000000f0 WB
ConventionalMemory 000630b40000 000000000000 00000fe1 WB
BootServicesData 000631b21000 000000000000 00001f17 WB
ConventionalMemory 000633a38000 000000000000 0000000a WB
BootServicesCode 000633a42000 000000000000 000003de WB
RuntimeServicesCode 000633e20000 000000000000 00000090 WB
ConventionalMemory 000633eb0000 000000000000 00000010 WB
RuntimeServicesData 000633ec0000 000000000000 00000120 WB
ConventionalMemory 000633fe0000 000000000000 0000001f WB
BootServicesData 000633fff000 000000000000 00000021 WB
ConventionalMemory 000634020000 000000000000 0000349d WB
BootServicesData 0006374bd000 000000000000 00000021 WB
BootServicesCode 0006374de000 000000000000 00000069 WB
BootServicesData 000637547000 000000000000 0000096f WB
BootServicesCode 000637eb6000 000000000000 00000012 WB
BootServicesData 000637ec8000 000000000000 00000101 WB
BootServicesCode 000637fc9000 000000000000 00000032 WB
BootServicesData 000637ffb000 000000000000 00000005 WB
MemoryMappedIO 000004000000 000000000000 00004000 UC
MemoryMappedIO 000009010000 000000000000 00000001 UC
OK efi-show
global BS,RS OsIndicationsSupported = 0x41
global BS,RS BootOptionSupport = 0x313
global BS,RS LangCodes = engfraengfra
global BS,RS PlatformLangCodes = en;fr;en-US;fr-FR
global BS,RS PlatformRecovery0000 =
01 00 00 00 36 00 44 00 65 00 66 00 61 00 75 00 6c 00 74 00
20 00 50 00 6c 00 61 00 74 00 66 00 6f 00 72 00 6d 00 52 00
65 00 63 00 6f 00 76 00 65 00 72 00 79 00 00 00 04 04 32 00
5c 00 45 00 46 00 49 00 5c 00 42 00 4f 00 4f 00 54 00 5c 00
42 00 4f 00 4f 00 54 00 41 00 41 00 36 00 34 00 2e 00 45 00
46 00 49 00 00 00 7f ff 04 00
global BS,RS ConInDev = VenHw(D3987D4B-971A-435F-8CAF-4967EB627241)/Uart(38400,8,N,1)/VenPcAnsi(),/VenHw(D3987D4B-971A-435F-8CAF-4967EB627241)/Uart(38400,8,N,1)/VenVt100(),/VenHw(D3987D4B-971A-435F-8CAF-4967EB627241)/Uart(38400,8,N,1)/VenVt100Plus(),/VenHw(D3987D4B-971A-435F-8CAF-4967EB627241)/Uart(38400,8,N,1)/VenUtf8(),/VenHw(D3987D4B-971A-435F-8CAF-4967EB627241)/Uart(38400,8,N,1)/VenMsg(7D916D80-5BB1-458C-A48F-E25FDD51EF94),/VenHw(D3987D4B-971A-435F-8CAF-4967EB627241)/Uart(38400,8,N,1)/VenMsg(E4364A7F-F825-430E-9D3A-9C9BE6817CA5),/VenHw(D3987D4B-971A-435F-8CAF-4967EB627241)/Uart(38400,8,N,1)/VenMsg(FBFCA56B-BB36-4B78-AAAB-BE1B97EC7CCB),/VenHw(D3987D4B-971A-435F-8CAF-4967EB627241)/Uart(38400,8,N,1)/VenMsg(8E46DDDD-3D49-4A9D-B875-3C086F6AA2BD),/VenHw(D3987D4B-971A-435F-8CAF-4967EB627241)/Uart(38400,8,N,1)/VenMsg(FC7DD6E0-813C-434D-B4DA-3BD649E9E15A)
global BS,RS ConOutDev = PciRoot(0x0)/Pci(0x1,0x0)/AcpiAdr(0x80010300),/VenHw(D3987D4B-971A-435F-8CAF-4967EB627241)/Uart(38400,8,N,1)/VenPcAnsi(),/VenHw(D3987D4B-971A-435F-8CAF-4967EB627241)/Uart(38400,8,N,1)/VenVt100(),/VenHw(D3987D4B-971A-435F-8CAF-4967EB627241)/Uart(38400,8,N,1)/VenVt100Plus(),/VenHw(D3987D4B-971A-435F-8CAF-4967EB627241)/Uart(38400,8,N,1)/VenUtf8(),/VenHw(D3987D4B-971A-435F-8CAF-4967EB627241)/Uart(38400,8,N,1)/VenMsg(7D916D80-5BB1-458C-A48F-E25FDD51EF94),/VenHw(D3987D4B-971A-435F-8CAF-4967EB627241)/Uart(38400,8,N,1)/VenMsg(E4364A7F-F825-430E-9D3A-9C9BE6817CA5),/VenHw(D3987D4B-971A-435F-8CAF-4967EB627241)/Uart(38400,8,N,1)/VenMsg(FBFCA56B-BB36-4B78-AAAB-BE1B97EC7CCB),/VenHw(D3987D4B-971A-435F-8CAF-4967EB627241)/Uart(38400,8,N,1)/VenMsg(8E46DDDD-3D49-4A9D-B875-3C086F6AA2BD),/VenHw(D3987D4B-971A-435F-8CAF-4967EB627241)/Uart(38400,8,N,1)/VenMsg(FC7DD6E0-813C-434D-B4DA-3BD649E9E15A)
global BS,RS ErrOutDev = PciRoot(0x0)/Pci(0x1,0x0)/AcpiAdr(0x80010300),/VenHw(D3987D4B-971A-435F-8CAF-4967EB627241)/Uart(38400,8,N,1)/VenPcAnsi(),/VenHw(D3987D4B-971A-435F-8CAF-4967EB627241)/Uart(38400,8,N,1)/VenVt100(),/VenHw(D3987D4B-971A-435F-8CAF-4967EB627241)/Uart(38400,8,N,1)/VenVt100Plus(),/VenHw(D3987D4B-971A-435F-8CAF-4967EB627241)/Uart(38400,8,N,1)/VenUtf8(),/VenHw(D3987D4B-971A-435F-8CAF-4967EB627241)/Uart(38400,8,N,1)/VenMsg(7D916D80-5BB1-458C-A48F-E25FDD51EF94),/VenHw(D3987D4B-971A-435F-8CAF-4967EB627241)/Uart(38400,8,N,1)/VenMsg(E4364A7F-F825-430E-9D3A-9C9BE6817CA5),/VenHw(D3987D4B-971A-435F-8CAF-4967EB627241)/Uart(38400,8,N,1)/VenMsg(FBFCA56B-BB36-4B78-AAAB-BE1B97EC7CCB),/VenHw(D3987D4B-971A-435F-8CAF-4967EB627241)/Uart(38400,8,N,1)/VenMsg(8E46DDDD-3D49-4A9D-B875-3C086F6AA2BD),/VenHw(D3987D4B-971A-435F-8CAF-4967EB627241)/Uart(38400,8,N,1)/VenMsg(FC7DD6E0-813C-434D-B4DA-3BD649E9E15A)
global BS,RS BootCurrent = 2
freebsd BS,RS LoaderPath = \EFI\BOOT\BOOTAA64.EFI
freebsd BS,RS LoaderDev = PciRoot(0x0)/Pci(0x12,0x7)/Pci(0x0,0x0)/Scsi(0x0,0x1)/HD(1,GPT,FCE27ED8-9904-11EB-BF4C-002590EC5BF2,0x3,0x10418)
59324945-ec44-4c0d-b1cd-9db139df070c NV,BS Attempt 1 =
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 01 00 00
00 00 00 00 00 00 41 74 74 65 6d 70 74 20 31 00 00 00 00 00
... boring
global NV,BS,RS Boot0000 = UiApp
Fv(64074AFE-340A-4BE6-94BA-91B5B4D0F71E)/FvFile(462CAA21-7614-4503-836E-8AB6F4662331)
global NV,BS,RS Timeout = 3
global NV,BS,RS PlatformLang = en
global NV,BS,RS Lang = eng
04b37fe8-f6ae-480b-bdd5-37d98c5e89aa NV,BS,RS VarErrorFlag =
ff
global NV,BS,RS ConIn = UsbHID(0xFFFF,0xFFFF,0x1,0x1),/VenHw(D3987D4B-971A-435F-8CAF-4967EB627241)/Uart(38400,8,N,1)/VenVt100()
global NV,BS,RS ConOut = PciRoot(0x0)/Pci(0x1,0x0)/AcpiAdr(0x80010300),/VenHw(D3987D4B-971A-435F-8CAF-4967EB627241)/Uart(38400,8,N,1)/VenVt100()
global NV,BS,RS ErrOut = PciRoot(0x0)/Pci(0x1,0x0)/AcpiAdr(0x80010300),/VenHw(D3987D4B-971A-435F-8CAF-4967EB627241)/Uart(38400,8,N,1)/VenVt100()
global NV,BS,RS Key0000 =
00 00 00 40 5d 4d 80 dd 00 00 0c 00 00 00
global NV,BS,RS Key0001 =
00 00 00 40 5d 4d 80 dd 00 00 17 00 00 00
5b446ed1-e30b-4faa-871a-3654eca36080 NV,BS 020017007935 =
3a 50 01 00 10 00 af af 04 00 00 00 01 00 00 00 00 00 00 00
global NV,BS,RS Boot0001 = UEFI Misc Device
VenHw(93E34C7E-B50E-11DF-9223-2443DFD72085,00)
global NV,BS,RS Boot0002 = UEFI ORACLE BlockVolume
PciRoot(0x0)/Pci(0x12,0x7)/Pci(0x0,0x0)/Scsi(0x0,0x1)
global NV,BS,RS Boot0004 = EFI Internal Shell
Fv(64074AFE-340A-4BE6-94BA-91B5B4D0F71E)/FvFile(7C04A583-9E3E-4F1C-AD65-E05268D0B4D1)
MemoryTypeInformation NV,BS MemoryTypeInformation =
ACPIReclaimMemory pages: 20
ACPIMemoryNVS pages: 40
global NV,BS,RS BootOrder = 2 0 1 4
MTC NV,BS,RS MTC = 4Terraform Stacks
When you do the manual A1.Flex launch, you’ll notice an option to save
the stack. If you do this, you can download a zip file, which contains a
terraform provisioning script.
This encapsulates the usual network and image settings, so yours won’t be the same as mine, here’s what I have:
provider "oci" {}
resource "oci_core_instance" "generated_oci_core_instance" {
agent_config {
is_management_disabled = "true"
is_monitoring_disabled = "true"
plugins_config {
desired_state = "DISABLED"
name = "Compute Instance Monitoring"
}
}
availability_config {
recovery_action = "RESTORE_INSTANCE"
}
availability_domain = "Ynwz:eu-amsterdam-1-AD-1"
compartment_id = "ocid1.tenancy.oc1..blah"
create_vnic_details {
assign_private_dns_record = "true"
assign_public_ip = "true"
subnet_id = "ocid1.subnet.oc1.eu-amsterdam-1.blah"
}
display_name = "beastie"
instance_options {
are_legacy_imds_endpoints_disabled = "false"
}
metadata = {
"ssh_authorized_keys" = "ssh-ed25519 AAAAC3Nzblah"
}
shape = "VM.Standard.A1.Flex"
shape_config {
memory_in_gbs = "24"
ocpus = "4"
}
source_details {
source_id = "ocid1.image.oc1.eu-amsterdam-blah"
source_type = "image"
}
}
Setting up terraform on FreeBSD is extraordinarily easy compared to
ports. Run this in the same folder as your unzipped main.tf config.
I skimmed this OCI Terraform tutorial and managed to get far enough along to automate spinning up systems. This is an excellent approach.
$ sudo pkg install -yr FreeBSD \
sysutils/tflint \
sysutils/terragrunt \
sysutils/terraform \
sysutils/terraform-docs
...
$ terraform init
Initializing the backend...
Initializing provider plugins...
- Finding latest version of hashicorp/oci...
- Installing hashicorp/oci v4.33.0...
- Installed hashicorp/oci v4.33.0 (signed by HashiCorp)
Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.
Terraform has been successfully initialized!
You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.
If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
$ terraform apply
Terraform used the selected providers to generate the following
execution plan. Resource actions are indicated with the following
symbols:
+ create
Terraform will perform the following actions:
# oci_core_instance.generated_oci_core_instance will be created
+ resource "oci_core_instance" "generated_oci_core_instance" {
+ availability_domain = "Ynwz:eu-amsterdam-1-AD-1"
+ boot_volume_id = (known after apply)
+ capacity_reservation_id = (known after apply)
+ compartment_id = "ocid1.tenancy.oc1..aaaaaaaaoblah"
+ dedicated_vm_host_id = (known after apply)
+ defined_tags = (known after apply)
+ display_name = "beastie"
+ fault_domain = (known after apply)
+ freeform_tags = (known after apply)
+ hostname_label = (known after apply)
+ id = (known after apply)
+ image = (known after apply)
+ ipxe_script = (known after apply)
+ is_pv_encryption_in_transit_enabled = (known after apply)
+ launch_mode = (known after apply)
+ metadata = {
+ "ssh_authorized_keys" = "ssh-ed25519 AAAAC3Nblah"
}
+ private_ip = (known after apply)
+ public_ip = (known after apply)
+ region = (known after apply)
+ shape = "VM.Standard.A1.Flex"
+ state = (known after apply)
+ subnet_id = (known after apply)
+ system_tags = (known after apply)
+ time_created = (known after apply)
+ time_maintenance_reboot_due = (known after apply)
+ agent_config {
+ are_all_plugins_disabled = (known after apply)
+ is_management_disabled = true
+ is_monitoring_disabled = true
+ plugins_config {
+ desired_state = "DISABLED"
+ name = "Compute Instance Monitoring"
}
}
+ availability_config {
+ is_live_migration_preferred = (known after apply)
+ recovery_action = "RESTORE_INSTANCE"
}
+ create_vnic_details {
+ assign_private_dns_record = true
+ assign_public_ip = "true"
+ defined_tags = (known after apply)
+ display_name = (known after apply)
+ freeform_tags = (known after apply)
+ hostname_label = (known after apply)
+ private_ip = (known after apply)
+ skip_source_dest_check = (known after apply)
+ subnet_id = "ocid1.subnet.oc1.eu-amsterdam-1.blah"
+ vlan_id = (known after apply)
}
+ instance_options {
+ are_legacy_imds_endpoints_disabled = false
}
+ launch_options {
+ boot_volume_type = (known after apply)
+ firmware = (known after apply)
+ is_consistent_volume_naming_enabled = (known after apply)
+ is_pv_encryption_in_transit_enabled = (known after apply)
+ network_type = (known after apply)
+ remote_data_volume_type = (known after apply)
}
+ platform_config {
+ numa_nodes_per_socket = (known after apply)
+ type = (known after apply)
}
+ preemptible_instance_config {
+ preemption_action {
+ preserve_boot_volume = (known after apply)
+ type = (known after apply)
}
}
+ shape_config {
+ baseline_ocpu_utilization = (known after apply)
+ gpu_description = (known after apply)
+ gpus = (known after apply)
+ local_disk_description = (known after apply)
+ local_disks = (known after apply)
+ local_disks_total_size_in_gbs = (known after apply)
+ max_vnic_attachments = (known after apply)
+ memory_in_gbs = 4
+ networking_bandwidth_in_gbps = (known after apply)
+ ocpus = 1
+ processor_description = (known after apply)
}
+ source_details {
+ boot_volume_size_in_gbs = (known after apply)
+ kms_key_id = (known after apply)
+ source_id =
"ocid1.image.oc1.eu-amsterdam-1.blah"
+ source_type = "image"
}
}
Plan: 1 to add, 0 to change, 0 to destroy.
Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.
Enter a value: yes
oci_core_instance.generated_oci_core_instance: Creating...
oci_core_instance.generated_oci_core_instance: Still creating... [10s elapsed]
oci_core_instance.generated_oci_core_instance: Still creating... [20s elapsed]
oci_core_instance.generated_oci_core_instance: Still creating... [30s elapsed]
oci_core_instance.generated_oci_core_instance: Still creating... [40s elapsed]
oci_core_instance.generated_oci_core_instance: Still creating... [50s elapsed]
oci_core_instance.generated_oci_core_instance: Still creating... [1m0s elapsed]
oci_core_instance.generated_oci_core_instance: Still creating... [1m10s elapsed]
oci_core_instance.generated_oci_core_instance: Still creating... [1m20s elapsed]
oci_core_instance.generated_oci_core_instance: Still creating... [1m30s elapsed]
oci_core_instance.generated_oci_core_instance: Still creating... [1m40s elapsed]
oci_core_instance.generated_oci_core_instance: Still creating... [1m50s elapsed]
oci_core_instance.generated_oci_core_instance: Still creating... [2m0s elapsed]
oci_core_instance.generated_oci_core_instance: Creation complete after 2m8s [id=ocid1.instance.oc1.eu-amsterdam-1.blah
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
Once the terraforming is complete, go to your OCI console, find the
image, and create a Console Connection again as before.
Things I Want To Try
I’d like to get ipxe booting directly. This is mainly because uploading an image each to OCI is slow and painful.
Also, understanding why the loader bails with error 19 and addressing
that would be even better.