Random Musings

O for a muse of fire, that would ascend the brightest heaven of invention!

Using FoundationDB on FreeBSD

Wednesday, 30 Jun 2021 Tags: foundationdb

The default databases/foundationdb port is at 6.3.13. There is also a 7.0.0-master port tracking upstream work reasonably closely.

basic config

The port installs a basic /usr/local/etc/foundationdb/foundationdb.conf but you need to set up the cluster file yourself. A starting point is:

# echo foundationdb:foundationdb@127.0.0.1:4500 \
    | tee /usr/local/etc/foundationdb/fdb.cluster

Alternatively, use a native TLS setup directly:

# echo foundationdb:foundationdb@127.0.0.1:4600:tls \
    | tee /usr/local/etc/foundationdb/fdb.cluster

Add users to the foundationdb group so that they can read the cluster file, and access the DB, via pw groupmod -m foundationdb $USER.

set permissions

# echo foundationdb_enable=YES > /etc/rc.conf.d/foundationdb
# zfs create zroot/var/db/foundationdb
# chown -R foundationdb:foundationdb \
    /var/db/foundationdb
# chmod -R u=rw,g=r,o-rwx,ug+X \
    /var/db/foundationdb

amend the config for TLS

If you didn’t use TLS above, the default port config is fine.

# /usr/local/etc/foundationdb/foundationdb.conf
## foundationdb.conf
##
## Configuration file for FoundationDB server processes
## Full documentation is available at
## https://apple.github.io/foundationdb/configuration.html#the-configuration-file

[fdbmonitor]
user = foundationdb
group = foundationdb

[general]
trace_format = json
restart_delay = 60
## by default, restart_backoff = restart_delay_reset_interval = restart_delay
# initial_restart_delay = 0
# restart_backoff = 60
# restart_delay_reset_interval = 60
cluster_file = /usr/local/etc/foundationdb/fdb.cluster
# delete_envvars =
# kill_on_configuration_change = true

## Default parameters for individual fdbserver processes
[fdbserver]
command = /usr/local/bin/fdbserver
public_address = auto:$ID:tls
listen_address = public
datadir = /var/db/foundationdb/data/$ID
logdir = /var/log/foundationdb
# logsize = 10MiB
# maxlogssize = 100MiB
# machine_id =
# datacenter_id =
# class =
# memory = 8GiB
# storage_memory = 1GiB
# cache_memory = 2GiB
# metrics_cluster =
# metrics_prefix =

# TLS settings
tls_ca_file=/usr/local/etc/foundationdb/cert.crt
tls_key_file=/usr/local/etc/foundationdb/fdb.pem
tls_certificate_file=/usr/local/etc/foundationdb/fdb.pem
# required if we are using self-signed certificates
tls_verify_peers=Check.Valid=0

# NB parameters set here override defaults from the [fdbserver] section
# each [fdbserver.XYZ] entry here will create an additional fdbserver
# process listening on that port, and inheriting existing TLS options.
[fdbserver.4600]

[backup_agent]
command = /usr/local/bin/backup_agent
logdir = /var/log/foundationdb

[backup_agent.1]

create TLS certificates is required

$ cd /usr/local/etc/foundationdb
$ openssl req -x509 -nodes -days 3650 -newkey rsa:2048 \
  -keyout private.key -out cert.crt
$ cat cert.crt private.key > fdb.pem

connect to the DB

As root, service foundationdb start and then run the rest as a normal user. Again, note that the user requires read access to all of /usr/local/etc/foundationdb which is unusual on FreeBSD, especially for TLS certificates, and also write access to the fdb.cluster file!

If you are not running TLS, then skip the additional parameters below.

$ fdbcli --tls_verify_peers Check.Valid=0 \
    --tls_key_file /usr/local/etc/foundationdb/private.key

Using cluster file `fdb.cluster'.

The database is unavailable; type `status' for more information.

initialise the DB

Now we have an fdb shell, it’s time to initialise an in-memory DB for testing, and write some data. If you run status a few times quickly after the configure new single memory command, you’ll also see FDB initialising and starting up - it’s not instant.

Welcome to the fdbcli. For help, type `help'.

fdb> configure new single memory
Database created

fdb> status

Using cluster file `fdb.cluster'.

Configuration:
  Redundancy mode        - single
  Storage engine         - memory-2
  Coordinators           - 1
  Usable Regions         - 1

Cluster:
  FoundationDB processes - 1
  Zones                  - 1
  Machines               - 1
  Memory availability    - 8.0 GB per process on machine with least available
  Retransmissions rate   - 0 Hz
  Fault Tolerance        - 0 machines
  Server time            - 06/30/21 10:30:57

Data:
  Replication health     - Healthy
  Moving data            - 0.000 GB
  Sum of key-value sizes - 0 MB
  Disk space used        - 105 MB

Operating space:
  Storage server         - 1.0 GB free on most full server
  Log server             - 4157.8 GB free on most full server

Workload:
  Read rate              - 6 Hz
  Write rate             - 0 Hz
  Transactions started   - 2 Hz
  Transactions committed - 0 Hz
  Conflict rate          - 0 Hz

Backup and DR:
  Running backups        - 0
  Running DRs            - 0

Client time: 06/30/21 10:30:57

fdb>

use the DB

fdb> writemode on
fdb> set poop emoji
Committed (219266389)
fdb> get poop
`poop' is `emoji'
fdb> set poop 💩
Committed (252691029)
fdb> get poop
`poop' is `\xf0\x9f\x92\xa9'
fdb>

processes when using TLS

Note that in the fdbserver entry, parameters seem to be missing from the supplied options. This is … unexpected.

# pgrep -ilU foundationdb
77073 backup_agent
94492 fdbmonitor
72585 fdbserver

# pgrep -ilf fdb
6981 /usr/local/bin/fdbmonitor \
    --conffile /usr/local/etc/foundationdb/foundationdb.conf \
    --lockfile /var/run/fdbmonitor.pid \
    --daemonize
44182 /usr/local/bin/fdbserver \
    --cluster_file \
    --datadir \
    --listen_address \
    --logdir \
    --public_address \
    --tls_ca_file \
    --tls_certificate_file \
    --tls_key_file \
    --tls_verify_peers \
    --trace_format
18843 /usr/local/bin/backup_agent \
    --cluster_file=/usr/local/etc/foundationdb/fdb.cluster \
    --logdir=/var/log/foundationdb \
    --trace_format=json

processes without TLS

# pgrep -ilfU foundationdb
77073 /usr/local/bin/backup_agent \
    --cluster_file=/usr/local/etc/foundationdb/fdb.cluster \
    --logdir=/var/log/foundationdb
94492 fdbmonitor \
    --conffile /usr/local/etc/foundationdb/foundationdb.conf \
    --lockfile /var/run/fdbmonitor.pid
72585 /usr/local/bin/fdbserver \
    --cluster_file=/usr/local/etc/foundationdb/fdb.cluster \
    --datadir=/var/db/foundationdb/data/4500 \
    --listen_address=public \
    --logdir=/var/log/foundationdb \
    --public_address=auto:4500

files

#  tree -AF /usr/local/etc/foundationdb/ /var/log/foundationdb/ /var/run/fdbmonitor.pid /var/db/foundationdb/
/usr/local/etc/foundationdb/
├── cert.crt
├── fdb.cluster
├── fdb.pem
├── foundationdb.conf
├── foundationdb.conf.sample
└── private.key
/var/log/foundationdb/
├── trace.127.0.0.1.18843.1625048558.1QF9z6.0.1.json
└── trace.127.0.0.1.4600.1625048558.jSIe2n.0.1.json
/var/run/fdbmonitor.pid [error opening dir]
/var/db/foundationdb/
└── data/
    └── 4600/
        ├── coordination-0.fdq
        ├── coordination-1.fdq
        ├── log2-V_4_LS_2-2ab07e66d565347b440298a035eaa3fa.sqlite
        ├── log2-V_4_LS_2-2ab07e66d565347b440298a035eaa3fa.sqlite-wal
        ├── logqueue-V_4_LS_2-2ab07e66d565347b440298a035eaa3fa-0.fdq
        ├── logqueue-V_4_LS_2-2ab07e66d565347b440298a035eaa3fa-1.fdq
        ├── processId
        ├── storage-50936d3cd197ddd6b529413c8420d20c-0.fdq
        └── storage-50936d3cd197ddd6b529413c8420d20c-1.fdq

erlang

I was unable to get access to FDB from the erlfdb module when TLS was enabled, so this info is from the DB in plain text mode.

$ erl -pz /usr/local/lib/erlang/lib/erlfdb-*/ebin
Erlang/OTP 21 [erts-10.3.5.7] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:1] [hipe] [dtrace]

Eshell V10.3.5.7  (abort with ^G)
1> l(erlfdb).
{module,erlfdb}
2> Db = erlfdb:open(<<"/usr/local/etc/foundationdb/fdb.cluster">>).
{erlfdb_database,#Ref<0.3052190807.3020816385.61233>}
4> erlfdb:get(Db, <<"poop">>).
<<"emojis">>
5> ok = erlfdb:set(Db, <<"foo">>, <<"bar">>).
ok
6> erlfdb:get(Db, <<"foo">>).
<<"bar">>

elixir

$ iex -pz /usr/local/lib/erlang/lib/erlfdb-*/ebin
Erlang/OTP 22 [erts-10.6.1] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [dtrace]

Interactive Elixir (1.9.4) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> l :erlfdb
{:module, :erlfdb}
iex(2)> db = :erlfdb.open "/usr/local/etc/foundationdb/fdb.cluster"
{:erlfdb_database, #Reference<0.2873199434.75890693.158035>}
iex(3)> :erlfdb.get(db, "poop")
"emoji"