Random Musings

O for a muse of fire, that would ascend the brightest heaven of invention!

An introduction to FreeBSD packaging

Thursday, 3 May 2018 Tags: freebsdpkgpoudriere

FreeBSD 10+ provides a new pkg(7) tool and a new efficient binary format for storing package info, with similar functionality to apt or yum in Linux distros.

Despite this, building custom packages on FreeBSD is a very simple task and we should use it as much as possible, both to set build -specific options, and to make our own private ports, instead of just relying on upstream packages and changes.

poudriere is the tool of choice to do this, synth is an alternative but not as flexible. Its job is to build each package and all its dependencies within a FreeBSD jail, and to provide annotated build logs and final packages that are directly usable as an external package repository.

setting up poudriere

basic configuration

poudriere(8) has a very complicated set of options, but in practice, the file below is all you need:

# the master config file /usr/local/etc/poudriere.conf
ZPOOL=zroot
FREEBSD_HOST=http://ftp.freebsd.org
RESOLV_CONF=/etc/resolv.conf
BASEFS=/usr/local/poudriere
USE_PORTLINT=yes
USE_TMPFS=yes
DISTFILES_CACHE=/var/cache/distfiles
CHECK_CHANGED_OPTIONS=verbose
PKG_REPO_SIGNING_KEY=/usr/local/etc/ssl/keys/pkg.key
WRKDIR_ARCHIVE_FORMAT=txz
URL_BASE=http://pkg.example.net/poudriere
KEEP_OLD_PACKAGES=yes
BUILDER_HOSTNAME=pkg.example.net

set make.conf

Tweaking /usr/local/etc/poudriere.d/make.conf ensures we can build dtrace-enabled ports and avoid openssl as a dependency:

# don't prompt me just do the work
BATCH=yes
# emit some dev friendly warnings when building ports
DEVELOPER=yes
PROXYDEPS_FATAL=yes
KEEP_OLD_PACKAGES=yes

# keep as much as possible in ram
# it saves your disks and is faster
USE_TMPFS=all
TMPFS_LIMIT=2
WRKDIRPREFIX=/tmp

# use ccache it makes things faster
CCACHE_DIR=/var/cache/ccache
# the default is /usr/ports/distfiles but if you use
# an alternate dir, then you can `git clean -fdx`
# your ports tree. Also, when you're low on space,
# everybody remembers to clean that cache dir first.
DISTFILES_CACHE=/var/cache/distfiles

# we can inject settings *only* for specific packages
# this is very handy to bundle a single port with a 
# static LibreSSL or OpenSSL and not clutter up the
# rest of your ports tree.
# Erlang/OTP with DTrace
.if ${.CURDIR:M*/lang/erlang*}
STRIP=
WITH_CTF=1
CFLAGS+=-fno-omit-frame-pointer
CFLAGS+=-g
.endif

# some ports require acceptance of licenses such as security/vault
LICENSES_ACCEPTED+=APACHE20 BSD2CLAUSE BSD3CLAUSE CDDL EPL GPLv2+ LGPL2+ LGPLV3 MIT MPL MPL20 NONE

# you can set default options for ports (perl, ruby, postgres, mysql
# versions), exclude or enable docs, and force all ports to use
# libressl if that's your thing.
CFLAGS+=-fno-omit-frame-pointer
# OpenSSL avoidance
DEFAULT_VERSIONS+=ssl=libressl
OPTIONS_UNSET= GSSAPI_BASE
OPTIONS_SET=   GSSAPI_MIT

# you can also have a custom ports category all of your own
VALID_CATEGORIES+=example

Note a few tricks above for handling port options and variants - for more details, read make.conf(5).

I also ln -s /usr/local/etc/poudriere.d/make.conf /etc/make.conf so that by default, my ports tree uses the same options, if I do manual testing.

set up signing keys for our packages

All package repos are signed which is great as it means we can bootstrap packages over HTTP without worrying about strong encryption before we have a suitable chain of trust available.

mkdir -p /usr/local/etc/ssl/keys /usr/local/etc/ssl/certs
chmod 600 /usr/local/etc/ssl/keys
openssl genrsa -out /usr/local/etc/ssl/keys/pkg.key 4096
chmod 0400 /usr/local/etc/ssl/keys/pkg.key
openssl rsa -in /usr/local/etc/ssl/keys/pkg.key \
  -pubout -out /usr/local/etc/ssl/certs/pkg.cert

set up ccache for faster builds

pkg install -yr FreeBSD devel/ccache
zfs create -o canmount=off zroot/var/cache
zfs create zroot/var/cache/ccache
zfs create zroot/var/cache/distfiles

set up our ports tree

We can actually have 2 ports trees - a public one, in /usr/ports, and a private one, in /usr/ports/example, if you like. I prefer to keep them all in a single repo, and just rebase occasionally with the main tree. I find this makes keeping my patches in sync between FreeBSD and downstream easier.

poudriere ports -c -F -f none -M /usr/ports -m git -n default
mkdir -p /usr/local/etc/poudriere.d/ports/default/
echo git> /usr/local/etc/poudriere.d/ports/default/method
echo /usr/ports> /usr/local/etc/poudriere.d/ports/default/mnt
git clone https://github.com/example/ports /usr/ports
cd /usr/ports
git remote add upstream https://github.com/freebsd/freebsd-ports
git fetch --all
git clone git@github.com:example/wharf /usr/ports/example

install poudriere

pkg install -y dialog4ports poudriere ccache
poudriere jail -c -j 11_amd64  -v 11.0-RELEASE -a amd64

customise and build a port

# make a list of builds we are interested in having a custom port for
echo lang/erlang-runtime18 > /usr/local/etc/poudriere.d/packages.lst

We have some port, say, lang/erlang where we wish to enable dtrace, hipe, and dirty schedulers. For dtrace support, make sure kernel support is started via kldload dtraceall or the build will fail mysteriously. This is typically enabled by adding dtraceall_load="YES" to /boot/loader.conf.

# we could simply customise all packages listed in packages.lst
poudriere options -f /usr/local/etc/poudriere.d/packages.lst
# or just customise a single package and assume defaults for its descendants
poudriere options -cn lang/erlang-runtime18
# look in here for the specific set options
/usr/local/etc/poudriere.d/options/lang_erlang-runtime18/options
# clear and re-do options for a specific set of ports
poudriere options -cn editors/vim lang/erlang net-mgmt/collectd5 lang/erlang-runtime18 devel/protobuf-c

pulling in a specific port from upstream

Sometimes, instead of forking and merging all the updates from the main FreeBSD ports tree, we simply want to cherry-pick a particularly juicy and flavoursome port. The catch is that, there may have been several intermediate updates, and also dependencies that we want, so a simply git cherry-pick is not going to cut the mustard.

Instead, we remove the entire port, then check it out from the upstream git master branch, and commit that whole lot. Remember to check the dependencies, and if necessary update them too.

Once you’ve done that you can simply ask poudriere to rebuild the port(s) for you: sudo poudriere bulk -j 11_amd64 category/port, but as your normal user, not as the build user which has no sudo privileges.

Note the -A to ensure you have ssh-forwarded private keys available for git to use on builder.

Remember not to use -c or -C here as it will wipe all our ports. If you do this, recover them from zfs snapshots rather than rebuild all the ports, as the checksums are different in the rebuilt ports, and therefore reinstalls of all packages may be needed. Also, building all the ports takes hours.

 ~ ssh -A build@builder                                                                                                                                                    (34s 913ms)
Last login: Mon Mar 13 21:57:39 2017 from 2001:569:77f0:3200:1579:a70e:3f88:af26
FreeBSD 11.0-RELEASE-p8 (GENERIC) #0: Wed Feb 22 06:12:04 UTC 2017

$ cd /usr/ports
$ git fetch --all
Fetching example
Fetching upstream
remote: Counting objects: 1057, done.
remote: Compressing objects: 100% (315/315), done.
remote: Total 1057 (delta 478), reused 387 (delta 387), pack-reused 352
Receiving objects: 100% (1057/1057), 1.87 MiB | 397.00 KiB/s, done.
Resolving deltas: 100% (519/519), completed with 177 local objects.
From git://github.com/freebsd/freebsd-ports
   a29c5c848d50..9789ea3b3b86  master     -> upstream/master
   d73362eecec2..23e2fff4f6ff  svn_head   -> upstream/svn_head

$ git status
On branch 2017Q1
Your branch is up-to-date with 'example/2017Q1'.
nothing to commit, working tree clean

$ more sysutils/graylog/distinfo
TIMESTAMP = 1473800220
SHA256 (graylog-2.1.0.tgz) = 76bf9bb4da57bfbfb146f0eaf22458328f5cb76c8e39bf735265886dbb7c2c27
SIZE (graylog-2.1.0.tgz) = 95402379

$ rm -rf sysutils/graylog/

$ git checkout upstream/master -- sysutils/graylog

$ tree sysutils/graylog/
sysutils/graylog/
├── Makefile
├── distinfo
├── files
│   ├── graylog.in
│   ├── graylog_logging.xml
│   ├── pkg-message.in
│   └── server.conf.sample.in
├── pkg-descr
└── pkg-plist

1 directory, 8 files
$ git status
On branch 2017Q1
Your branch is up-to-date with 'example/2017Q1'.
Changes to be committed:
  (use "git reset HEAD <file>..." to unstage)

    modified:   sysutils/graylog/Makefile
    modified:   sysutils/graylog/distinfo
    modified:   sysutils/graylog/files/graylog.in
    new file:   sysutils/graylog/files/pkg-message.in
    new file:   sysutils/graylog/files/server.conf.sample.in
    modified:   sysutils/graylog/pkg-plist

Changes not staged for commit:
  (use "git add/rm <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

    deleted:    sysutils/graylog/files/graylog.conf.example

$ git commit -am "sysutils/graylog: import 2.2.2 from upstream"
[2017Q1 69a8d5522b1e] sysutils/graylog: import 2.2.2 from upstream
 6 files changed, 111 insertions(+), 96 deletions(-)
 rewrite sysutils/graylog/Makefile (61%)
 create mode 100644 sysutils/graylog/files/pkg-message.in
 rename sysutils/graylog/files/{graylog.conf.example => server.conf.sample.in} (99%)
 rewrite sysutils/graylog/pkg-plist (100%)

 $ git push example master
Counting objects: 11, done.
Delta compression using up to 12 threads.
Compressing objects: 100% (10/10), done.
Writing objects: 100% (11/11), 8.66 KiB | 0 bytes/s, done.
Total 11 (delta 4), reused 6 (delta 1)
remote: Resolving deltas: 100% (4/4), completed with 4 local objects.
To github.com:example/ports
   7dfcf176cfd4..69a8d5522b1e  2017Q1 -> 2017Q1

keeping things updated

  • the port tree itself needs to be merged/rebased off an appropriate upstream ports branch periodically
  • the jail needs to be updated for each major release, it’s simpler just to build a fresh jail for this
  • the host system also needs to be updated 
# ports tree
cd /usr/ports
git fetch --all
git rebase upstream/master
# .... hackety hackety
git push --force origin/master
# update jail
poudriere jail -u -j 11_amd64
# rebuild its packages nicely
nice -n 18 idprio 29 poudriere bulk -j 11_amd64 -f /usr/local/etc/poudriere.d/server.pkg
# once the core server packages are done do your server upgrade and then rebuild all the packages
nice -n 18 idprio 29 \
    poudriere bulk -j 11_amd64 \
    -f /usr/local/etc/poudriere.d/server.pkg

I have a short helper script to do this which runs as part of cron, for my server that runs CURRENT:

/root/update.sh
#!/bin/sh -e
freebsd-update fetch install || /usr/bin/true
# poudriere ports -u
cd /usr/ports
git fetch upstream
git rebase upstream/master
chown -R root:wheel /usr/ports
chmod -R g+rw /usr/ports
# update the ports
poudriere jail -u -j 11_amd64
poudriere bulk -f /usr/local/etc/poudriere.d/core.pkg -j 11_amd64
pkg backup -d /var/db/pkg/backup
pkg update
pkg upgrade -y
echo OK done

Just keep an eye out for any daemons that may now have updated binaries and need restarting.

exporting our packages

  • rsync and file:/// urls are perfectly fine, as is zfs send etc
  • www/h2o has a configuration for exporting custom packages over HTTP

Advanced pkg repo security

Some package repositories need security. While a certificate-based approach may be possible, I’ve successfully used ssh://, http basic authentication, and even a humble rsync, local file:// url or nfs.

using file URL and rsync

The details of rsync or zfs send|recv are left to the user. Ensure softlinks are preserved appropriately. Within your /usr/local/etc/pkg/repos/repo.conf file, simply use an appropriate URI-style path:

example: {
        url: file:///usr/local/poudriere/data/packages/11_amd64-default
        mirror_type: http
        signature_type: pubkey
        pubkey: /usr/local/etc/ssl/certs/pkg.cert
        enabled: yes
}

ssh repo security

Use ssh:// urls, including username, url, and path, to retrieve packages securely, in your /usr/local/etc/pkg/repos/repo.conf file. Specific ssh keys can be configured in /root/.ssh/config as required. Ensure you restrict access on the server side too, via sshd_config.

url: ssh://pkg.example.net:/usr/local/poudriere/data/packages/11_amd64-default

http basic auth

The underlying libfetch library is used by pkg to do the dirty work. See https://www.freebsd.org/cgi/man.cgi?query=fetch&sektion=3 for details, but if fetch supports it, so does package.

Add this to your h2o.conf file for serving packages, making use the jail name you are using in poudriere matches the file.dir location — in this case, it’s 11_amd64-default:

    pkg.example.net:
        paths:
            "/poudriere":
                file.dir: "/usr/local/share/poudriere/html/"
            "/poudriere/data":
                file.dir: "/usr/local/poudriere/data/logs/bulk/"
            "/FreeBSD:11:amd64":
                mruby.handler: |
                    require "htpasswd.rb"
                    Htpasswd.new("/usr/local/etc/h2o/private/htpasswd", "example")
                file.dir: "/usr/local/poudriere/data/packages/11_amd64-default/"

When running pkg update and similar, you can export HTTP_AUTH into the environment to be consumed by pkg:

env HTTP_AUTH="basic:example:user:passwd" pkg update

integrating custom private ports into the ports tree via overlay

  • use git to edit and maintain any public-facing port changes in /usr/ports
  • git clone git://github.com/example/wharf /usr/ports/example
  • add /usr/ports/Makefile.local to add our example category into the ports tree
  • there is a VALID_CATEGORIES+=example in /etc/make.conf and /usr/local/etc/poudriere.d/make.conf for poudriere automated package builds and normal by-hand builds as well

tips

  • there is a single Makefile for each port to describe the build process
  • see simple for a minimal port that uses a git tag to build from
  • wharf is the equivalent of the https://github.com/freebsd/freebsd-ports repo to provide Makefiles for build and packaging scripts for our private ports

pkg(8) hacks

FreeBSD’s pkg(8) tool supports a few handy package management commands:

pkg lock -y <thing>
pkg unlock -y <thing>

# get a list of all manually installed packages (i.e. "really useful" vs
# automatic dependencies)
pkg leaf

# what package installed this file?
pkg which /usr/local/bin/erl

# read any major ports-related notes for upgrading packages since a
# given date
pkg updating -d 20160101

You can also retrieve an older version of a package from the local cache in /var/cache/pkg/<name-version>.txz and install it directly via pkg add <path>, although generally it’s better to keep packages consistent within the set built at the same time on the package server pkg.example.net and use them from there.

resources

packaging

canonical

useful but not up to date:

main ports tree diff

If you want to include your own category in ports, include the following snippet in your main ports tree:

diff --git a/.gitignore b/.gitignore
index 13ad354..51ffbb7 100644
--- a/.gitignore
+++ b/.gitignore
@@ -6,3 +6,4 @@
 *.rej
 *.orig
 *.sw[p-z]
+/example
diff --git a/Makefile.local b/Makefile.local
new file mode 100644
index 0000000..790f37f
--- /dev/null
+++ b/Makefile.local
@@ -0,0 +1,3 @@
+# $FreeBSD$
+#
+SUBDIR                  += example