FreeBSD’s Release Engineering team has a strong preference for building all of FreeBSD, and creating the release artifacts, using only the base system. This is done to minimise the risk of untrusted code, possibly even malicious code, being smuggled into the system. Unfortunately, this means that we don’t have access to the full range of tools that are available in ports or packages.

One of the tools that we don’t have in base is jq, a command-line JSON processor. This is a very useful tool for manipulating JSON data, in a semantic way.

As part of automating builds for OCI containers, and for Oracle Cloud, we need to extract various bits of build-time results, and spit out a suitable blob of JSON.

My first variation used jq(1) via ports, and it was straightforwards to produce the expected output. However, I wanted to see if I could do this with base FreeBSD tools.

After a fair bit of code golf, I came up with a sed(1) example, which definitely worked, but was neither readable nor maintainable. If there were errors in future, it would be very frustrating to debug.

FreeBSD has a built-in Lua interpreter, along with a few FreeBSD-native extensions, intended to be used within the base system, instead of shell scripts, and as a more tractable alternative to C.

Without further ado, I present flua, worked on by Kyle Evans, Warner Losh, and many others. One of the first C modules that was added to flua is the UCL module, which includes amongst other things, a JSON parser, and can emit Lua, JSON, and YAML.

$ /usr/libexec/flua -v
Lua 5.4.6  Copyright (C) 1994-2023 Lua.org, PUC-Rio

#!/usr/libexec/flua
local ucl = require("ucl")
local parser = ucl.parser()

-- use ucl to parse an example JSON string into a lua table
local json = '{"key": "value", "array": [1, 2, 3], "yes": true, "no": false, "empty": null}'
parser:parse_string(json)
local lua = parser:get_object()
-- pretty print the lua table
print(ucl.to_format(lua, "json"))

This script uses the ucl module to parse a JSON string into a Lua table, and then re-uses the ucl module to pretty-print the Lua table as JSON again, round-tripping the data.

{
    "no": false,
    "empty": null,
    "key": "value",
    "yes": true,
    "array": [
        1,
        2,
        3
    ]
}

flua supports many other functions, including manipulating jails, and additional POSIX functions. It’s a very powerful tool, and I’m very keen to see if be extended in the future, possibly even writing entire daemons in Lua, or more complex services.

For the build system, we can now use readable Lua to generate the JSON blobs, and to be certain that the resulting JSON is valid. There are always some unavoidable exceptions, such as requiring qemu-tools to manipulate disk images, or curl to upload files, but in general these happen after the build artefact is created, and can easily be verified if needed by the user. The build itself is done with base tools only, to provide as robust a trusted chain of provenance as possible.