Random Musings

O for a muse of fire, that would ascend the brightest heaven of invention!


Adding per-jail metadata

Sunday, 12 Jan 2025 Tags: freebsdjails

Igor & I have been working on adding arbitrary metadata for and within jails D47668 along with the weekly jails working group.

The intent is to commit this reasonably soon to 15.0-CURRENT, and allow a few months for practical feedback before MFC to 14-STABLE for polishing, after real-world usage. Your feedback and testing is welcomed!

Overview

D47668 stores, reads and writes, per-jail metadata in the kernel as a blob. Userland tools have been extended to support accessing this blob both directly, and also to provide minimal key-value usage of that data. There are 2 separate stores, one called “meta” which is visible only to the jail host, and another called “env” which is visible inside the jail as well.

This is not a replacement for a full key/value store, nor a config management system, but may be enough to avoid them in many cases.

man page

A brief summary from the manual page:

meta, env

An arbitrary string associated with the jail. Its maximum buffer size is controlled by the global security.jail.meta_maxbufsize sysctl, which can only be adjusted by the non-jailed root user. While the meta is hidden from the jail, the env is readable through the security.jail.env sysctl. The set of allowed single-byte characters for both buffers is limited by the global security.jail.meta_allowedchars sysctl, which is also tunable by the non-jailed root user. All characters are allowed if it is set to an empty string.

Each buffer can be treated as a set of key=value\n strings. In order to add or replace a specific key the meta.keyname=value or env.keyname=value parameter notations must be used. The meta.keyname or env.keyname notations are used for reading. Multiple keys can be queried or modified with a single command.

Igor has also prepared a video demonstrating the feature!

Usage

  • create a jail with external and internal data
  • note alternatives for setting data (full string, individual items)
  • observe that meta is only visible outside the jail
# jail -c name=testy path=/ persist=true \
  meta="$(printf 'k=v\nempty')" \
  env.inside=jail
# jls -j testy env
inside=jail
# jls -j testy meta.k
v
# jexec testy sysctl security.jail.meta
sysctl: unknown oid 'security.jail.meta'
# jexec testy sysctl security.jail.env
    security.jail.env: inside=jail

List data using libxo and textproc/jq

  • we list both buffers in their entirety (raw mode)
  • and also show how the tags can be fetched separately (cooked mode)
# jls -j testy --libxo=json meta env meta.k meta.empty_tag env.inside \
  |  jq -r '."jail-information".jail[]'
{
  "meta": "k=v\nempty",
  "env": "inside=jail",
  "meta.k": "v",
  "meta.empty_tag": "",
  "env.inside": "jail"
}

Modify the data

  • the other keys remain unchanged
  • the newline separator causes the entries to spill over successive lines
  • this is not ideal, but is a reflection of the existing implementation
# jail -m name=testy meta.port=5432
# jls -j testy meta env
port=5432
k=v
empty inside=jail

Setting with jail.conf(5)

Metadata can also be set via jail.conf(5) files:

# https://man.freebsd.org/jail.conf

db1 {
    meta.tags = "db couchdb";
    meta.ports = "5984:5984";
    meta.backend = "couchdb";
...

A More Complex Example

Multi-jail usage with Load Balancers

Assuming we have a dynamic set of jails, representing some fancy app, and that the output of this command doesn’t go to stdout, but is actually used to update a load balancer.

The load balancer does not care about the type of the jail, only to group all instances of a given meta.backend together.

There is no need to poll continuously here, it is sufficient to add an additional startup/shutdown hook to the jail management script to trigger these updates.

### create 3 jails each with an IP but otherwise common metadata
# jail -c name=couch1 ip4.addr=10.0.0.1 \
  path=/ persist=true meta.backend=couchdb meta.port=5984
# jail -c name=couch2 ip4.addr=10.0.0.2 \
  path=/ persist=true meta.backend=couchdb meta.port=5984
# jail -c name=couch3 ip4.addr=10.0.0.3 \
  path=/ persist=true meta.backend=couchdb meta.port=5984

### sprinkle awk judiciously

$ echo backend couchdb_be; jls name ip4.addr meta.port meta.backend \
  | grep couchdb\$ \
  | awk '{print "  server " $1 " " $2 ":" $3 " check observe"}'
backend couchdb_be
  server couch1 10.0.0.1:5984 check observe
  server couch2 10.0.0.2:5984 check observe
  server couch3 10.0.0.3:5984 check observe

A more robust version could be done with --libxo json, or flua:

#!/usr/libexec/flua

local jail = require("jail")
print("backend couchdb_be")

for j in jail.list({"name", "meta.port", "meta.backend", "ip4.addr"}) do
	if j["meta.backend"] == "couchdb" then
		print(string.format("  server %s %s:%s check observe",
		j["name"], j["ip4.addr"], j["meta.port"]))
	end
end