Random Musings

O for a muse of fire, that would ascend the brightest heaven of invention!

Using OCI command-line tools to Upload FreeBSD images

Monday, 4 Apr 2022 Tags: freebsdoci

This post is focused on uploading FreeBSD qcow2 images into Oracle’s OCI infrastructure, and massaging them to be available for general usage.

It will be updated periodically, as we move from custom personal images, to proper “Marketplace” images, directly from the FreeBSD Release Engineering team.

Special thanks is due to Alessandro Sagratini, who ported the oci command-line tool to FreeBSD. This will be available in binary packages before the Easter Bunny comes!

This enabled me to whizz through most of the deployment work, along with the comprehensive OCI docs.

Setup

$ sudo pkg install -r FreeBSD devel/oci-cli textproc/jq

Pre-requisites

The OCI signup process needs a completely separate post on it’s own, but basically:

  • OCI signup complete
  • you’ll need to supply a valid credit card to avoid having your account shut down after ~ 30 days
  • create an API key
  • download both API private and public keys
  • grab the sample config that the same page provides to ~/.oci/
  • ~/.oci/config downloaded & amended with private key path

Assuming you got all of that correct, let’s get started.

As you go along, we’ll keep stashing various OCIDs (Oracle Cloud IDs, sort of a GUID for each object in OCI) in OCI environment variables.

Namespace Check

The namespace is a general container for all resources. You can create your own, but for the moment, we’ll keep this simple, and use the default one.

$ oci os ns get
{
    "data": "fribble"
}
$ export OCI_NS=$(oci os ns get | jq -r .data)

Get User ID

This isn’t needed until much later, but it’s easy enough to acquire now.

$ export OCI_USER=$(oci iam user list | jq -r '.data[0].id')

Fetch Compartment ID

Our initial compartment-id will be the tenancy value from your OCI config file. All our data lives in this compartment. Again, you can create your own.

$ oci iam user list
{
  "data": [
    {
      "capabilities": {
        "can-use-api-keys": true,
        "can-use-auth-tokens": true,
        "can-use-console-password": true,
        "can-use-customer-secret-keys": true,
        "can-use-db-credentials": true,
        "can-use-o-auth2-client-credentials": true,
        "can-use-smtp-credentials": true
      },
      "compartment-id": "ocid1.tenancy.oc1..aaaabbccddee123
      "db-user-name": null,
      "defined-tags": {},
      "description": null,
      "email": "dch@freebsd.org",
      "email-verified": true,
      "external-identifier": "5ccf44319ef0f7753dfbb05ae8be6382",
      "freeform-tags": {},
      "id": "ocid1.user.oc1..aaaabbccddee123
      "identity-provider-id": null,
      "inactive-status": null,
      "is-mfa-activated": false,
      "last-successful-login-time": "2022-04-04T14:05:18.440000+00:00",
      "lifecycle-state": "ACTIVE",
      "name": "dch@freebsd.org",
      "previous-successful-login-time": null,
      "time-created": "2022-03-27T14:24:30.794000+00:00"
    }
  ]
}
$ export OCI_CID=$(oci iam user list | jq -r '.data[0]."compartment-id"')

NB if you have multiple user accounts (which is not the initial / default configuration), we have simply selected the first created one, from the array. Adapt the jq(1) invocation accordingly.

Create Bucket

The bucket for uploaded images should be:

  • standard (best performance)
  • auto-tiering (move to archival after a period of non use)
  • public & listable (so people can discover image versions)

According to the OCI bucket docs, the incantation is:

$ oci os bucket create \
    --namespace $OCI_NS \
    --compartment-id $OCI_CID \
    --public-access-type ObjectRead \
    --auto-tiering InfrequentAccess \
    --name freebsd-images
{
  "data": {
    "approximate-count": null,
    "approximate-size": null,
    "auto-tiering": "InfrequentAccess",
    "compartment-id": "ocid1.tenancy.oc1..aaaabbccddee123",
    "created-by": "ocid1.user.oc1..aaaabbccddee123",
    "defined-tags": {
      "Oracle-Tags": {
        "CreatedBy": "default/dch@freebsd.org",
        "CreatedOn": "2022-04-04T22:43:56.206Z"
      }
    },
    "etag": "93a99c99-d207-9f61-896a-e9e6ac3793f1",
    "freeform-tags": {},
    "id": "ocid1.bucket.oc1.eu-frankfurt-1.aaaabbccddee123",
    "is-read-only": false,
    "kms-key-id": null,
    "metadata": {},
    "name": "freebsd-images",
    "namespace": "fribble",
    "object-events-enabled": false,
    "object-lifecycle-policy-etag": null,
    "public-access-type": "ObjectRead",
    "replication-enabled": false,
    "storage-tier": "Standard",
    "time-created": "2022-04-04T22:43:56.220000+00:00",
    "versioning": "Disabled"
  },
  "etag": "93a99c99-d207-9f61-896a-e9e6ac3793f1"
}

This only needs doing once, and we can in future refer to the bucket by name.

Upload local file into Blob Storage

This is straightforwards enough, note you may need to tweak parallelism, and note the use of (MD5 sigh) checksums to ensure uploaded content matches local version. This step, and the subsequent conversion, are very time consuming.

$ time oci os object bulk-upload \
    --namespace $OCI_NS \
    --bucket-name freebsd-images \
    --parallel-upload-count 20 \
    --verify-checksum \
    --no-overwrite \
    --src-dir . \
    --include *.qcow2
Uploaded item  [########################.....]  100%
________________________________________________________
Executed in  328.93 secs    fish           external
   usr time    4.09 secs  894.00 micros    4.09 secs
   sys time    1.90 secs   25.00 micros    1.90 secs
{
  "skipped-objects": [],
  "upload-failures": {},
  "uploaded-objects": {
    "FreeBSD-13.1-RC1-arm64-aarch64-20220404-b23fad18b209.qcow2": {
      "etag": "e2e57085-3aed-47f9-a2c0-245c690e26ee",
      "last-modified": "Mon, 04 Apr 2022 14:59:36 GMT",
      "opc-multipart-md5": "jco3dcKKLL8wF8FdjdCoew==-6",
      "verify-md5-checksum": "md5 checksum matches [Local: jco3dcKKLL8wF8FdjdCoew==-6]"
    }
  }
}

Convert Object to Image

While we have uploaded our data into the blob store, it still needs to be imported as a custom image. This next set of steps takes quite a while for no particularly good reason.

The choice of EMULATED launch mode seems rather arbitrary, but the VM will not boot without it. Ideally we could set further properties here, but this is not exposed in the command line tool yet.

The operating system name and version numbers are useful as we will be able to filter and search on them later.

The display name should match the original image name, to simplify cleaning up old images in the future.

$ oci compute image import from-object \
    --bucket-name freebsd-images \
    --compartment-id $OCI_CID \
    --namespace $OCI_NS \
    --operating-system FreeBSD \
    --operating-system-version 13.1-RC1 \
    --source-image-type QCOW2 \
    --launch-mode emulated \
    --name FreeBSD-13.1-RC1-arm64-aarch64-20220404-b23fad18b209.qcow2 \
    --display-name FreeBSD-13.1-RC1-arm64-aarch64-20220404-b23fad18b209.qcow2
{
  "data": {
    "agent-features": null,
    "base-image-id": null,
    "billable-size-in-gbs": null,
    "compartment-id": "ocid1.tenancy.oc1..aaaabbcc123",
    "create-image-allowed": true,
    "defined-tags": {
      "Oracle-Tags": {
        "CreatedBy": "default/dch@freebsd.org",
        "CreatedOn": "2022-04-04T15:01:52.615Z"
      }
    },
    "display-name": "FreeBSD-13.1-RC1-arm64-aarch64-20220404-c78d300e1b639ed1a2dc7035a8acecf10ceb0a9b.qcow2",
    "freeform-tags": {},
    "id": "ocid1.image.oc1.eu-frankfurt-1.aaaabbcc123",
    "launch-mode": "NATIVE",
    "launch-options": {
      "boot-volume-type": "ISCSI",
      "firmware": "UEFI_64",
      "is-consistent-volume-naming-enabled": false,
      "is-pv-encryption-in-transit-enabled": false,
      "network-type": "VFIO",
      "remote-data-volume-type": "PARAVIRTUALIZED"
    },
    "lifecycle-state": "IMPORTING",
    "listing-type": null,
    "operating-system": "FreeBSD",
    "operating-system-version": "13.1-RC1",
    "size-in-mbs": null,
    "time-created": "2022-04-04T15:01:53.204000+00:00"
  },
  "etag": "6fc8fd47711de318f30bd96daaf38320d720cec934f69dd119be8b4cd78ca2e6",
  "opc-work-request-id": "ocid1.coreservicesworkrequest.oc1.eu-frankfurt-1.abtheljsu45yuefsju3eda7odgm4hdx2hdfjyoqxelj3kgtjig7abnvfpwua"
}

List Images

We can use the lifecycle state to poll for completion of image import:

$ oci compute image list \
    --compartment-id $OCI_CID \
    --operating-system FreeBSD \
    --lifecycle-state IMPORTING
{
  "data": [
    {
      "agent-features": null,
      "base-image-id": null,
      "billable-size-in-gbs": 0,
      "compartment-id": "ocid1.tenancy.oc1..aaaabbccddee123",
      "create-image-allowed": true,
      "defined-tags": {
        "Oracle-Tags": {
          "CreatedBy": "default/dch@freebsd.org",
          "CreatedOn": "2022-04-04T23:20:19.929Z"
        }
      },
      "display-name": "FreeBSD-13.1-RC1-arm64-aarch64-20220404-b23fad18b209.qcow2",
      "freeform-tags": {},
      "id": "ocid1.image.oc1.eu-frankfurt-1.aaaabbccddee123",
      "launch-mode": "NATIVE",
      "launch-options": {
        "boot-volume-type": "ISCSI",
        "firmware": "UEFI_64",
        "is-consistent-volume-naming-enabled": false,
        "is-pv-encryption-in-transit-enabled": false,
        "network-type": "VFIO",
        "remote-data-volume-type": "PARAVIRTUALIZED"
      },
      "lifecycle-state": "IMPORTING",
      "listing-type": null,
      "operating-system": "FreeBSD",
      "operating-system-version": "13.1-RC1",
      "size-in-mbs": null,
      "time-created": "2022-04-04T23:20:20.228000+00:00"
    }
  ]
}

Wait for output to be non-empty

You can use the same pattern to poll on IMPORTING, to the same effect.

$ oci compute image list \
    --compartment-id $OCI_CID \
    --lifecycle-state AVAILABLE \
    --display-name FreeBSD-13.1-RC1-arm64-aarch64-20220404-b23fad18b209.qcow2 \
    | jq .
...

$ oci compute image list \
    --compartment-id $OCI_CID \
    --lifecycle-state AVAILABLE \
    --display-name FreeBSD-13.1-RC1-arm64-aarch64-20220404-b23fad18b209.qcow2 | jq .
{
  "data": [
    {
      "agent-features": null,
      "base-image-id": null,
      "billable-size-in-gbs": 3,
      "compartment-id": "ocid1.tenancy.oc1..aaaabbcc123",
      "create-image-allowed": true,
      "defined-tags": {
        "Oracle-Tags": {
          "CreatedBy": "default/dch@freebsd.org",
          "CreatedOn": "2022-04-04T15:01:52.615Z"
        }
      },
      "display-name": "FreeBSD-13.1-RC1-arm64-aarch64-20220404-c78d300e1b639ed1a2dc7035a8acecf10ceb0a9b.qcow2",
      "freeform-tags": {},
      "id": "ocid1.image.oc1.eu-frankfurt-1.aaaabbcc123",
      "launch-mode": "NATIVE",
      "launch-options": {
        "boot-volume-type": "ISCSI",
        "firmware": "UEFI_64",
        "is-consistent-volume-naming-enabled": false,
        "is-pv-encryption-in-transit-enabled": false,
        "network-type": "VFIO",
        "remote-data-volume-type": "PARAVIRTUALIZED"
      },
      "lifecycle-state": "AVAILABLE",
      "listing-type": null,
      "operating-system": "FreeBSD",
      "operating-system-version": "13.1-RC1",
      "size-in-mbs": 47694,
      "time-created": "2022-04-04T15:01:53.204000+00:00"
    },
    ... more images
  ]
}

Update Compatible Image Types

  • list all possible shapes for our tenancy 
$ oci compute shape list \
    --compartment-id $OCI_CID \
    | jq -r '.data[].shape'
BM.Standard.A1.160
...
VM.Standard3.Flex
  • remove them all from the uploaded image, shell loop is left as exercise to the reader 
$ oci compute image-shape-compatibility-entry remove \
    --image-id $OCI_IMAGE \
    --force \
    --shape-name  ...
  • add the ones we need in, they are missing by default 
$ oci compute image-shape-compatibility-entry add \
    --image-id $OCI_IMAGE \
    --shape-name VM.Standard.A1.Flex
{
  "data": {
    "image-id": "ocid1.image.oc1.eu-frankfurt-1.aaaabbccddee123",
    "memory-constraints": null,
    "ocpu-constraints": null,
    "shape": "VM.Standard.A1.Flex"
  }
}
$ oci compute image-shape-compatibility-entry add \
    --image-id $OCI_IMAGE \
    --shape-name BM.Standard.A1.160
{
  "data": {
    "image-id": "ocid1.image.oc1.eu-frankfurt-1.aaaabbccddee123",
    "memory-constraints": null,
    "ocpu-constraints": null,
    "shape": "BM.Standard.A1.160"
  }
}

The shape properties should resemble this now, for arm64:

custom shape properties

Remove VFIO and BIOS options from Image

Booting an instance requires matching the Schema Capabilities of both the source image, and the target shape, the actual hardware. Testing shows that if VFIO networking is still enabled in the VM image, it is not possible to deploy arm64 instances, so we need to remove that first.

It’s possible that this can be set globally, for the entire compartment, but this may break other hardware configurations, so lets address this per-image at this point.

SR-IOV networking is supported on all platform images, with the following exceptions: Images for Arm-based shapes do not support SR-IOV networking. – https://docs.oracle.com/en-us/iaas/Content/Compute/Tasks/launchinginstance.htm#networking

First, list the specific schema applied to this image, then extract its ID for later use:

$ oci compute image-capability-schema list --image-id $OCI_IMAGE | jq -S .
{
  "data": [
    {
      "compartment-id": "ocid1.tenancy.oc1..aaaabbcc123",
      "compute-global-image-capability-schema-version-name": "31bbd754-18ef-48ec-8fd4-4e5b2e5c1903",
      "defined-tags": {
        "Oracle-Tags": {
          "CreatedBy": "default/dch@freebsd.org",
          "CreatedOn": "2022-04-04T20:28:21.856Z"
        }
      },
      "display-name": "computeimgcapschema20220404202821",
      "freeform-tags": {},
      "id": "ocid1.computeimgcapschema.oc1.eu-frankfurt-1.aaaabbcc123",
      "image-id": "ocid1.image.oc1.eu-frankfurt-1.aaaabbcc123",
      "schema-data": {
        "Compute.Firmware": {
          "default-value": "UEFI_64",
          "descriptor-type": "enumstring",
          "source": "IMAGE",
          "values": [
            "UEFI_64",
            "BIOS"
          ]
        },
        "Compute.LaunchMode": {
          "default-value": "PARAVIRTUALIZED",
          "descriptor-type": "enumstring",
          "source": "IMAGE",
          "values": [
            "NATIVE",
            "EMULATED",
            "PARAVIRTUALIZED",
            "CUSTOM"
          ]
        },
        "Compute.SecureBoot": {
          "default-value": false,
          "descriptor-type": "boolean",
          "source": "IMAGE"
        },
        "Network.AttachmentType": {
          "default-value": "PARAVIRTUALIZED",
          "descriptor-type": "enumstring",
          "source": "IMAGE",
          "values": [
            "E1000",
            "VFIO",
            "PARAVIRTUALIZED"
          ]
        },
        "Storage.BootVolumeType": {
          "default-value": "PARAVIRTUALIZED",
          "descriptor-type": "enumstring",
          "source": "IMAGE",
          "values": [
            "ISCSI",
            "SCSI",
            "IDE",
            "PARAVIRTUALIZED"
          ]
        },
        "Storage.ConsistentVolumeNaming": {
          "default-value": true,
          "descriptor-type": "boolean",
          "source": "IMAGE"
        },
        "Storage.Iscsi.MultipathDeviceSupported": {
          "default-value": false,
          "descriptor-type": "boolean",
          "source": "IMAGE"
        },
        "Storage.LocalDataVolumeType": {
          "default-value": "PARAVIRTUALIZED",
          "descriptor-type": "enumstring",
          "source": "IMAGE",
          "values": [
            "ISCSI",
            "SCSI",
            "IDE",
            "PARAVIRTUALIZED"
          ]
        },
        "Storage.ParaVirtualization.AttachmentVersion": {
          "default-value": 2,
          "descriptor-type": "enuminteger",
          "source": "IMAGE",
          "values": [
            1,
            2
          ]
        },
        "Storage.ParaVirtualization.EncryptionInTransit": {
          "default-value": true,
          "descriptor-type": "boolean",
          "source": "IMAGE"
        },
        "Storage.RemoteDataVolumeType": {
          "default-value": "PARAVIRTUALIZED",
          "descriptor-type": "enumstring",
          "source": "IMAGE",
          "values": [
            "ISCSI",
            "SCSI",
            "IDE",
            "PARAVIRTUALIZED"
          ]
        }
      },
      "time-created": "2022-04-04T20:28:21.876000+00:00"
    }
  ]
}

$ export OCI_SCHEMA=$(oci compute image-capability-schema list \
  --image-id $OCI_IMAGE \
  | jq -rS '.data[0].id')

Read Configuring Image Capabilities for further info.

We can now retrieve the schema and filter out the unwanted VFIO capability from the array, and then write it back again. Note that the SchemaData JSON key is different, so we need to fix that too.

$ oci compute image-capability-schema list \
    --image-id $OCI_IMAGE \
    | jq -c '{ "SchemaData": (.data[0]."schema-data")}' \
    | sed -E -e 's/,?"VFIO",?//' \
    | jq -S .
    > restricted-schema.json

$ oci compute image-capability-schema update \
    --force \
    --image-capability-schema-id $OCI_SCHEMA \
    --from-json file://./restricted-schema.json
{
  "data": {
    "compartment-id": "ocid1.tenancy.oc1..aaaabbcc123",
    "compute-global-image-capability-schema-id": "ocid1.computeglobalimgcapschema.oc1.eu-frankfurt-1.aaaabbcc123",
    "compute-global-image-capability-schema-version-name": "31bbd754-18ef-48ec-8fd4-4e5b2e5c1903",
    "defined-tags": {
      "Oracle-Tags": {
        "CreatedBy": "default/dch@freebsd.org",
        "CreatedOn": "2022-04-04T20:28:21.856Z"
      }
    },
    "display-name": "computeimgcapschema20220404202821",
    "freeform-tags": {},
    "id": "ocid1.computeimgcapschema.oc1.eu-frankfurt-1.aaaabbcc123",
    "image-id": "ocid1.image.oc1.eu-frankfurt-1.aaaabbcc123",
    "schema-data": {
      "Compute.Firmware": {
        "default-value": "UEFI_64",
        "descriptor-type": "enumstring",
        "source": "IMAGE",
        "values": [
          "UEFI_64",
          "BIOS"
        ]
      },
      "Compute.LaunchMode": {
        "default-value": "PARAVIRTUALIZED",
        "descriptor-type": "enumstring",
        "source": "IMAGE",
        "values": [
          "NATIVE",
          "EMULATED",
          "PARAVIRTUALIZED",
          "CUSTOM"
        ]
      },
      "Compute.SecureBoot": {
        "default-value": false,
        "descriptor-type": "boolean",
        "source": "IMAGE"
      },
      "Network.AttachmentType": {
        "default-value": "PARAVIRTUALIZED",
        "descriptor-type": "enumstring",
        "source": "IMAGE",
        "values": [
          "E1000",
          "PARAVIRTUALIZED"
        ]
      },
      "Storage.BootVolumeType": {
        "default-value": "PARAVIRTUALIZED",
        "descriptor-type": "enumstring",
        "source": "IMAGE",
        "values": [
          "ISCSI",
          "SCSI",
          "IDE",
          "PARAVIRTUALIZED"
        ]
      },
      "Storage.ConsistentVolumeNaming": {
        "default-value": true,
        "descriptor-type": "boolean",
        "source": "IMAGE"
      },
      "Storage.Iscsi.MultipathDeviceSupported": {
        "default-value": false,
        "descriptor-type": "boolean",
        "source": "IMAGE"
      },
      "Storage.LocalDataVolumeType": {
        "default-value": "PARAVIRTUALIZED",
        "descriptor-type": "enumstring",
        "source": "IMAGE",
        "values": [
          "ISCSI",
          "SCSI",
          "IDE",
          "PARAVIRTUALIZED"
        ]
      },
      "Storage.ParaVirtualization.AttachmentVersion": {
        "default-value": 2,
        "descriptor-type": "enuminteger",
        "source": "IMAGE",
        "values": [
          1,
          2
        ]
      },
      "Storage.ParaVirtualization.EncryptionInTransit": {
        "default-value": true,
        "descriptor-type": "boolean",
        "source": "IMAGE"
      },
      "Storage.RemoteDataVolumeType": {
        "default-value": "PARAVIRTUALIZED",
        "descriptor-type": "enumstring",
        "source": "IMAGE",
        "values": [
          "ISCSI",
          "SCSI",
          "IDE",
          "PARAVIRTUALIZED"
        ]
      }
    },
    "time-created": "2022-04-04T20:28:21.876000+00:00"
  },
  "etag": "1486d30da15fb34fe82a1084ee28f2edad59f2f4a8e1134f3888e401383528dd"
}

Create an Instance from our image

Let’s do this via terraform. It seems to be less painful, as we are missing all the network, availability domains and stuff.

Once your network config is set up, then you can summon machines like so:

# NB still to be reviewed
# https://docs.cloud.oracle.com/en-us/iaas/tools/oci-cli/latest/oci_cli_docs/cmdref/network/vcn/create.html#cmdoption-cidr-block
$ export cidr_block=<substitute-value-of-cidr_block>

# https://docs.cloud.oracle.com/en-us/iaas/tools/oci-cli/latest/oci_cli_docs/cmdref/network/vcn/create.html#cmdoption-compartment-id
$ export compartment_id=<substitute-value-of-compartment_id>

# https://docs.cloud.oracle.com/en-us/iaas/tools/oci-cli/latest/oci_cli_docs/cmdref/compute/instance/launch.html#cmdoption-availability-domain
$ export availability_domain=<substitute-value-of-availability_domain>

# https://docs.cloud.oracle.com/en-us/iaas/tools/oci-cli/latest/oci_cli_docs/cmdref/compute/instance/launch.html#cmdoption-image-id
$ export image_id=<substitute-value-of-image_id>

# https://docs.cloud.oracle.com/en-us/iaas/tools/oci-cli/latest/oci_cli_docs/cmdref/compute/instance/launch.html#cmdoption-shape
$ export shape=<substitute-value-of-shape>

$ vcn_id=$(oci network vcn create --cidr-block $cidr_block \
  --compartment-id $compartment_id --query data.id --raw-output)

$ subnet_id=$(oci network subnet create --cidr-block $cidr_block \
  --compartment-id $compartment_id --vcn-id $vcn_id --query data.id --raw-output)

$ oci compute instance launch \
    --compartment-id ... \
    --availability-domain ... \
    --subnet-id ... \
    --shape VM.Standard.A1.Flex \
    --display-name icarus \
    --hostname-label icarus \
    --image-id ... \
    --ipxe-script-file ... \