Adding per-jail metadata
Sunday, 12 Jan 2025
Igor & I have been working on adding arbitrary metadata for and within jails D47668 along with the weekly jails working group.
The intent is to commit this reasonably soon to 15.0-CURRENT, and allow a few months for practical feedback before MFC to 14-STABLE for polishing, after real-world usage. Your feedback and testing is welcomed!
Overview
D47668 stores, reads and writes, per-jail metadata in the kernel as a blob. Userland tools have been extended to support accessing this blob both directly, and also to provide minimal key-value usage of that data. There are 2 separate stores, one called “meta” which is visible only to the jail host, and another called “env” which is visible inside the jail as well.
This is not a replacement for a full key/value store, nor a config management system, but may be enough to avoid them in many cases.
man
page
A brief summary from the manual page:
meta
,env
An arbitrary string associated with the jail. Its maximum buffer size is controlled by the global
security.jail.meta_maxbufsize
sysctl, which can only be adjusted by the non-jailed root user. While the meta is hidden from the jail, the env is readable through the security.jail.env sysctl. The set of allowedsingle-byte characters
for both buffers is limited by the globalsecurity.jail.meta_allowedchars
sysctl, which is also tunable by the non-jailed root user. All characters are allowed if it is set to an empty string.
Each buffer can be treated as a set of key=value\n
strings. In order to
add or replace a specific key the meta.keyname=value
or
env.keyname=value
parameter notations must be used. The meta.keyname
or env.keyname
notations are used for reading. Multiple keys can be
queried or modified with a single command.
Igor has also prepared a video demonstrating the feature!
Usage
- create a jail with external and internal data
- note alternatives for setting data (full string, individual items)
- observe that meta is only visible outside the jail
# jail -c name=testy path=/ persist=true \
meta="$(printf 'k=v\nempty')" \
env.inside=jail
# jls -j testy env
inside=jail
# jls -j testy meta.k
v
# jexec testy sysctl security.jail.meta
sysctl: unknown oid 'security.jail.meta'
# jexec testy sysctl security.jail.env
security.jail.env: inside=jail
List data using libxo
and textproc/jq
- we list both buffers in their entirety (raw mode)
- and also show how the tags can be fetched separately (cooked mode)
# jls -j testy --libxo=json meta env meta.k meta.empty_tag env.inside \
| jq -r '."jail-information".jail[]'
{
"meta": "k=v\nempty",
"env": "inside=jail",
"meta.k": "v",
"meta.empty_tag": "",
"env.inside": "jail"
}
Modify the data
- the other keys remain unchanged
- the newline separator causes the entries to spill over successive lines
- this is not ideal, but is a reflection of the existing implementation
# jail -m name=testy meta.port=5432
# jls -j testy meta env
port=5432
k=v
empty inside=jail
Setting with jail.conf(5)
Metadata can also be set via jail.conf(5) files:
# https://man.freebsd.org/jail.conf
db1 {
meta.tags = "db couchdb";
meta.ports = "5984:5984";
meta.backend = "couchdb";
...
A More Complex Example
Multi-jail usage with Load Balancers
Assuming we have a dynamic set of jails, representing some fancy app,
and that the output of this command doesn’t go to stdout
, but is
actually used to update a load balancer.
The load balancer does not care about the type of the jail, only to
group all instances of a given meta.backend
together.
There is no need to poll continuously here, it is sufficient to add an additional startup/shutdown hook to the jail management script to trigger these updates.
### create 3 jails each with an IP but otherwise common metadata
# jail -c name=couch1 ip4.addr=10.0.0.1 \
path=/ persist=true meta.backend=couchdb meta.port=5984
# jail -c name=couch2 ip4.addr=10.0.0.2 \
path=/ persist=true meta.backend=couchdb meta.port=5984
# jail -c name=couch3 ip4.addr=10.0.0.3 \
path=/ persist=true meta.backend=couchdb meta.port=5984
### sprinkle awk judiciously
$ echo backend couchdb_be; jls name ip4.addr meta.port meta.backend \
| grep couchdb\$ \
| awk '{print " server " $1 " " $2 ":" $3 " check observe"}'
backend couchdb_be
server couch1 10.0.0.1:5984 check observe
server couch2 10.0.0.2:5984 check observe
server couch3 10.0.0.3:5984 check observe
A more robust version could be done with --libxo json
, or flua:
#!/usr/libexec/flua
local jail = require("jail")
print("backend couchdb_be")
for j in jail.list({"name", "meta.port", "meta.backend", "ip4.addr"}) do
if j["meta.backend"] == "couchdb" then
print(string.format(" server %s %s:%s check observe",
j["name"], j["ip4.addr"], j["meta.port"]))
end
end