pkg(8) is the new package manager for FreeBSD, it is the only package manager support for FreeBSD 10 and newer, while it is available along with the old package management tools on FreeBSD 8 and 9.
In September 2014 it will be the only package management tool supported on all version of FreeBSD. Unlike the old package management tools, pkg(8) is not integrated directly into the base system, instead of that a bootstrap tool pkg(7).
This article will cover pkg(8) as of version 1.2.6.
The old package management tools has been developed in 1993, have been maintained since but received very little improvements. While in 1993 it did fits the requirements (few packages, simple semantics involved in packaging) in the Y2K, they are becoming less and less accurate. The design of those tool was not accurate anymore for modern packages, and the package format itself was weak:
Beside this, it was really hard to improve, the code base was showing its age.
After spending time on trying to improve it, we had to face the fact that writting a new tool from scratch will be easier and would allow to start directly on modern package management concept.
In 2010 Julien Laffaye and I decided to have a look at how complicated it would be to create a package management tool that would be able to properly handle binary upgrades, be fast with lots of packages installed, be safe, and allows to have control over the whole processes of package manipulations. We were also willing to provide a high level library so anyone willing to write tools that deals with packages would be able to just plug on the library and do safe operations.
A few years later, pkg(8) has grown, and regular contributors has join the development including (but not only): Bryan Drewery, Matthew Seaman, Vsevolod Stakhov.
pkg(8) is built around some basic principles:
pkg(8) is not available in base and we made the decision to never push it into base for the following reasons:
As pkg(8) is not available directly from the base system, a bootstrap mechanism (pkg(7)) as been introduced in the base system, it has been designed to be as seamless.
FreeBSD 8.4 is the oldest version including the bootstrap tool, FreeBSD 10 is the only one to have in its final version. This is focuses on what pkg(7) does on FreeBSD 10.
pkg(7) is in fact /usr/sbin/pkg, when run it will look up for the pkg(8) binary (by default in ${LOCALBASE}/sbin), if the binary has been found then it will directly execute it, if not found it will propose to the user to bootstrap it.
Bootstrapping consist in fetching the latest pkg(8) packages from the official FreeBSD repositories for a given version on a given architecture.
To determine where to fetch the package from and how to validate it pkg(7) shares the exact same mechanism of configuration files for repositories which will be describe later.
In pkg(8) almost all options can be overwritten by an environment variable. the command pkg -vv will show all the support options and the current setup:
Version : 1.2.6
PACKAGESITE :
PKG_DBDIR : /var/db/pkg
PKG_CACHEDIR : /var/cache/pkg
PORTSDIR : /usr/ports
PUBKEY :
HANDLE_RC_SCRIPTS : no
ASSUME_ALWAYS_YES : no
REPOS_DIR : [
/etc/pkg/,
/usr/local/etc/pkg/repos/,
]
PLIST_KEYWORDS_DIR :
SYSLOG : yes
AUTODEPS : yes
ABI : freebsd:11:x86:64
DEVELOPER_MODE : no
PORTAUDIT_SITE : http://portaudit.FreeBSD.org/auditfile.tbz
VULNXML_SITE : http://www.vuxml.org/freebsd/vuln.xml.bz2
MIRROR_TYPE : SRV
FETCH_RETRY : 3
PKG_PLUGINS_DIR : /usr/local/lib/pkg/
PKG_ENABLE_PLUGINS : yes
PLUGINS : [
]
DEBUG_SCRIPTS : no
PLUGINS_CONF_DIR : /usr/local/etc/pkg/
PERMISSIVE : no
REPO_AUTOUPDATE : yes
NAMESERVER :
EVENT_PIPE :
FETCH_TIMEOUT : 30
UNSET_TIMESTAMP : no
SSH_RESTRICT_DIR :
PKG_SSH_ARGS :
PKG_ENV : {
}
DISABLE_MTREE : no
DEBUG_LEVEL : 0
ALIAS : {
}
Repositories:
FreeBSD: {
url : "pkg+http://pkg.FreeBSD.org/freebsd:11:x86:64/latest",
enabled : yes,
mirror_type : "SRV",
signature_type : "FINGERPRINTS",
fingerprints : "/usr/share/keys/pkg"
}
For the rest of this article we will consider a setup where LOCALBASE is defined as /usr/local.
Important files or directory around pkg(8):
/etc/pkg/*.conf
/usr/local/etc/pkg.conf
/usr/local/etc/pkg/repos/*.conf
/usr/local/lib/libpkg.so.1
/usr/local/sbin/pkg
/usr/local/sbin/pkg-static
pkg-static is a statically linked version of pkg(8) it will allow users to do binary package manipulation even when upgrade from a major version to another major version of FreeBSD.
When the pkg(8) binary is invoked, it will first load /usr/local/etc/pkg.conf except if specified another location via the -C option. For all configuration entries if there is already an environment variable defining it, it will have the priority over the configuration file.
Now the configuration file has been read, pkg(8) can lookup for for repository definitions in the directory defined via the REPOS_DIR. It will read all the .conf files in those directories and will look for entries 1 or N entries like this:
<NAME> : {
url : "<url>",
enabled: true,
mirror_type: "SRV",
signature_type: "FINGERPRINTS",
fingerprints: "/usr/share/keys/pkg",
pubkey: "/etc/pkg/pub.key"
}
First time <NAME> is found then the "url" element is mandatory. Each time <NAME> the options it defines will overwrite the previously defined one.
Once all the above is done pkg(8) is ready to operate. The bootstrap follow the exact same sequence.
Unlike the old package management tool, pkg is a single binary with lots of small commands
$ pkg help
Usage: pkg [-v] [-d] [-l] [-N] [-j <jail name or id>|-c <chroot path>] [-C <configuration file>] [-R <repo config dir>] <command> [<args>]
Global options supported:
-d Increment debug level
-j Execute pkg(8) inside a jail(8)
-c Execute pkg(8) inside a chroot(8)
-C Use the specified configuration file
-R Directory to search for individual repository configurations
-l List available commands and exit
-v Display pkg(8) version
-N Test if pkg(8) is activated and avoid auto-activation
Commands supported:
add Registers a package and installs it on the system
annotate Add, modify or delete tag-value style annotations on packages
audit Reports vulnerable packages
autoremove Removes orphan packages
backup Backs-up and restores the local package database
check Checks for missing dependencies and database consistency
clean Cleans old packages from the cache
config Display the value of the configuration options
convert Convert database from/to pkgng
create Creates software package distributions
delete Deletes packages from the database and the system
fetch Fetches packages from a remote repository
help Displays help information
info Displays information about installed packages
install Installs packages from remote package repositories
lock Locks package against modifications or deletion
plugins Manages plugins and displays information about plugins
query Queries information about installed packages
register Registers a package into the local database
remove Deletes packages from the database and the system
repo Creates a package repository catalogue
rquery Queries information in repository catalogues
search Performs a search of package repository catalogues
set Modifies information about packages in the local database
ssh ssh packages to be used via ssh
shell Opens a debug shell
shlib Displays which packages link against a specific shared library
stats Displays package database statistics
unlock Unlocks a package, allowing modification or deletion
update Updates package repository catalogues
updating Displays UPDATING information for a package
upgrade Performs upgrades of packaged software distributions
version Displays the versions of installed packages
which Displays which package installed a specific file
A normal user maintain his system up to date will just have to run a couple of command:
$ pkg upgrade
$ pkg autoremove
The first will upgrade all the installed package to their latest version, the second will remove the now orphans (not depend on anymore, and not installed on purpose by the user).
While the ports tree is the natural way of creating packages for FreeBSD, it is rather easy to create your packages outside of the ports tree.
First let's have a look at what pkg create expects:
Usage: pkg create [-On] [-f format] [-o outdir] [-p plist] [-r rootdir] -m manifestdir
pkg create [-Ognx] [-f format] [-o outdir] [-r rootdir] pkg-name ...
pkg create [-On] [-f format] [-o outdir] [-r rootdir] -a
The first line it what we are looking for. We are considering here that the sources has been built and installed in a given DESTDIR and we want to package /usr/local/bin/foo and /usr/local/lib/libbar.so.1
$ find ${DESTDIR}
${DESTDIR}/usr/local/bin/foo
${DESTDIR}/usr/local/lib/libbar.so
${DESTDIR}/usr/local/lib/libbar.so.1
First we need to create a directory to push the manifest in there
$ mkdir ${MANIFESTDIR}
now create a ${MANIFESTDIR}/+MANIFEST
name: foo
version: 1.0
origin: mycompany/foo
categories: [ mycompany, devel ]
comment: foo is a nice tool
www: http://mycompany/foo
maintainer: me@mycompany
prefix: /usr/local
The above is the minimum necesarry it is written in libucl format which is very flexible and allow differents syntaxes The description for the packages can be added via a file ${MANIFESTDIR}/+DESC or by adding the following lines to the manifest
desc: <<EOD
foo is a fantastic tool developed by my company
on top of libbar.
EOD
A message can be added to the package by either writing it in a ${MANIFESTDIR}/+DISPLAY or by adding the following line to the manifest message: <<EOD This is a message for our user I hope you do like foo EOD
Create a plist (like the ports do):
bin/foo
lib/libbar.so.1
The package can now be created:
$ pkg create -m ${MANIFESTDIR} -r ${DESTDIR} -o ${OUTDIR}
This will create a ${OUTDIR}/foo-1.0.txz ready to be distributed.
The best way to distribute it is to create your own repository:
$ pkg repo ${OUTDIR}
pkg(8) does support a couple of different transport mechanism: - http - ftp - file - ssh (this requires the remove server to also have pkg(8) installed)
In the design of pkg, it has been decided that the list of mirrors should not be on the end user side, but rather dynamically discovered, therefor 2 differents mechanism are supported to allow retreiving the list of mirrors:
HTTP: The repository URL should download a text document containing a sequence of lines beginning with `URL:' followed by any amount of while space and one URL for a repository mirror. Any lines not matching this pattern are ignored. Mirrors are tried in the order listed until a download succeeds.
SRV: For an SRV mirrored repository where the URL is specified as http://pkgrepo.example.org/ SRV records should be set up in the DNS:
$ORIGIN example.com
_http._tcp.pkgrepo IN SRV 10 1 80 mirror0
IN SRV 20 1 80 mirror1
where the SRV priority and weight parameters are used to control
search order and traffic weighting between sites, and the port num-
ber and hostname are used to construct the individual mirror URLs.
Given we fetches packages from the Internet it is imporant we can make sure the packages are the one you really expect to receive. Unlike most of the packages system pkg(8) does not sign individual packages but rather the "catalogue" which contains checksums for every single packages available. Two mechanism are provided:
While a system is living, lots of packages get installed and uninstalled along with their dependencies. pkg(8) do track what has been explicitly installed by a user from what as been installed as automatically as a depencency. Being able to track gives pkg(8) the ability to provide this "autoremove" sub command
$ pkg autoremove
It will propose to remove all the packages that has been installed as a dependency but are not needed anymore. The ports tree has been modified to be able to provide this information when installing a package meaning that after installing a package with lots of build dependencies, "pkg autoremove" will propose to remove all the build only dependencies.
It is possible to mark a package as automaically installed with the following command
$ pkg set -A 1 mypkg
Or to unmark it:
$ pkg set -A 0 mypkg
pkg(8) is focused on binary packages as a primary target, therefor it supports the repository concept which consist in a bunch of packages and a catalogue containing all the metadata about those packages.
It supports having multiple repositories, for example having the official FreeBSD repository along with a custom repository. A simple use case is if a user needs the apache module for php, he cannot use the official repositories because the default php package does not provide such module. It is possible in that case to simply build the php packages using a building tool like poudriere and create a repository with the custom php packages in it, and use the FreeBSD repository for all other packages.
$ pkg install -r myrepo php
The above command will install the php package enforcing the installation from the "myrepo" repository
$ pkg annotate -A php repository myrepo
Will make sure that during the regular "pkg upgrade" in the futur, pkg(8) will only check "myrepo" for updates concerning php
Sometime keeping a given version of a package can be critical for a system, but upgrading all other packages is required (security fixes for example) pkg(8) provides a subcommand to allow upgrading the system without upgrading given packages:
$ pkg lock mysql-server
pkg unlock will allow to remove that lock.
FreeBSD used to provide a third party tool (portaudit) to audit the installed package database against the list of known vulnerabilities provided via vuxml. In pkg(8) it has been decided that this is an important feature that should be part of the package management tool.
$ pkg audit
will do such reports.
pkg(8) provides 2 interfaces (query and rquery) to help gathering information about locally installed packages and remote available packages Both interface allows to create powerful custom queries on the database but also to control its output.
This allows to simplify integration with scripts:
#!/bin/sh
eval `pkg query "WWW=%w ; NAME="%n" ; VERSION="%v" fossil `
echo "${NAME} is installed at version ${VERSION} and its website is: ${WWW}"
# can be also done that way
pkg query "%n is install at version %v and its website is: %w"
This can be combined with the pkg(8) alias mechanism to provide new subcommand easily to create a command that will list all package explicitly installed by the user, Add the following to pkg.conf:
alias: {
explicit: query -e "%a == 0" "%n-%v"
}
now the command pkg explicit exists and is presenting all the packages explicitly installed in the form of one line per package with <name>-<version>
For next version we aim at bringing lots of new features and improve the user experience. pkg(8) has recently gain a real sat which is the basement to improve the flexibility of the package format and package building tools (the ports tree).
Allowing to bring smart dependencies: instead of depending on the specific version of a package we can depend on a valid range of versions
Allowing to bring provides/requires: do not depend on a package but rather on a feature (Requires: perl, http). Provides/Requires can bevery useful in multiple case for example:
Work as also begun on having plugable backend for pkg(8) with the current one (the binary repositories) being just one of them, but one could imagine a "ports tree" backend, a CPAN,pear,pypy,gem backend.
For now in the background, pkg(8) uses origin (aka category/portname) to identify a package (determining what is an upgrade of what and so one) this is due to the fact the the same port could have multiple different name in the ports tree depending on options or multiple port could have the same name but different version (and still being different ports, not 2 version of the same port). In the meantime we have been able to fix the ports so that package names are consistent again, meaning we can switch back pkgname as the identifier for packages.
This change while it sounds not very signicant opens a lot of new possibilities for the ports tree: