Random Musings

O for a muse of fire, that would ascend the brightest heaven of invention!

IOCaging up the RabbitMQ

Saturday, 22 Aug 2015 Tags: epmdfreebsdiocagerabbitmqspiped

Here, I’ll show you how to set up rabbitmq, a well-known open source message queue system, based on Erlang/OTP, in a container, using a tool called iocage that runs on FreeBSD, using zfs and jails that have been in base FreeBSD for a decade or longer, connected via tunnels between endpoints using spiped, a robust and reliable tool designed specifically for reliably & securely tunnelling network services across unsecured networks.

I’ve updated this for FreeBSD 10.3-RELEASE and iocage 1.7.4.

Background

I host my servers on real hardware, mainly because the price and performance is just so much better than virtualised ephemeral cloud services — I’m guaranteed my ECC RAM, and have no concerns about competing with some other transient cloud-induced heisenbug not of my own making. As I’m stingy, I refuse to pay for an additional IPv4 address space, and I want to re-use the IPv6 allocation that comes with my physical server.

In particular, staying secure, and using only IPv6, introduces some twists and complications which make this post worth writing.

Creating the iocage

iocage is simply a set of wrapper scripts around FreeBSD jails, a secure and trusted way of creating lightweight virtual machines that share the same kernel as the host operating system. This makes them very fast, and we rely on the tried and tested jail functionality to keep things secure. Jails were first introduced at the SANE conference in 2000. Note the date — this is 15 years old tech by now.

The jailed filesystem is actually stored as a ZFS dataset, which provides high-speed, compressed block IO, and can be snapshotted, or transferred between servers as backup, or to bootstrap remote nodes. ZFS is a stable and mature filesystem that was ported from Illumos (ex OpenSolaris) into FreeBSD, starting in 2007. Note the date again. This is solid boring mainstream technology.

As a bonus, iocage stores all attributes related to the jail as properties on the ZFS dataset itself, so what we end up with is a fully writable, easily cloned and transferred, bootable virtual machine. Very much like docker images, except with less excitement, with significantly better performance, including direct access to FreeBSD’s network stack, using a battle-hardened filesystem, and security that has seen over a decade of solid production use.

There’s no real reason why you have to use the latest FreeBSD release, as iocage will work on 9.x without issues, but if you are interested later on in trying out the sysutils/docker-freebsd port, then 10.2 includes the required latest 64-bit linux emulation, which allows you to run docker images directly on FreeBSD, using the clever freebsd-docker project. But that story is for another day.

Bring out the Cage

Before we can get into iocage, we should have a few things set up. I’m running:

  • FreeBSD 10.3 amd64 with dual IPv4/IPv6 stack installed to ZFS zpool
  • unbound caching DNS server accessible on ::1/8 and 127.0.0.0/8
  • sudo access as there is much root usage going on here

Let’s install iocage, using the development package, start the service, which automatically creates the zfs dataset (to contain relevant FreeBSD release images, iocage templates and clones, as well as your jails), and grab the latest release from a fast mirror near me for future jail creation.

pkg install -y iocage-devel sysrc
sysrc iocage_enable=YES
service iocage start
export RELEASE=10.3-RELEASE
iocage fetch release=$RELEASE \
    ftphost=ftp.de.freebsd.org \
    ftpfiles="base.txz doc.txz src.txz"

Feel free to look under /iocage/ now, or zroot/iocage it should be quite straightforwards to understand what’s going on.

Next up, we will create a new iocage-based jail, for our rabbitmq service. The networking setup here is significant:

  • IPv4 is loopback only, no external network access
  • IPv6 is both loopback, and externally routable 
iocage create -b \
    tag=rabbit \
    hostname=rabbit.skunkwerks.at \
    priority=10 \
    boot=on \
    defaultrouter6='fe80::1%em0' \
    ip4_addr='lo0|127.0.0.7/8' \
    ip4=enable \
    vnet=off \
    ip6_addr='em0|2a01:4f8:200:12cf:0:0:0:7/64,lo0|::7/8'

Compare iocage list and what you can see under /iocage/jails/. Again these are just standard ZFS datasets, you can alter, tweak, and edit just as usual.

You can also take a look around from inside the as-yet not running jail via iocage chroot rabbit /bin/sh it’s just like a standard FreeBSD system.

Gild the iocage

No cage would be complete without some fancy trimmings to make our processes enjoy their confinement. Packages, config files, and a few users are all that’s required.

Add the packages

The packages are pretty standard, except I use custom builds using an amazing tool called poudriere which knocks the socks off debian and rpm based packages. Another story for another, another, day.

FreeBSD 10’s new pkg tool allows me to install from the host system directly into the jailed filesystem. You can look around from the host using iocage to look up the uuid that the filesystem’s name is based upon:

RABBIT="/iocage/jails/`iocage get host_hostuuid rabbit`/root"
cd $RABBIT

pkg 1.7.2 correctly creates the users inside the jail, which didn’t work correctly in the previous version of this post, where I needed to create these by hand.

Again note the base utilities pkg and pw all support chroot work out of the box. This is one of the secret sauces of the BSD derived operating systems - userland and kernel are developed and shipped together.

Add RabbitMQ config files

Mostly these are nothing special, however there are a few tricks here to confine Erlang/OTP’s distributed name service daemon epmd to only loopback addresses within the jail, and to use the IPv6 address for rabbitmq’s user-facing functionality

cat <<EOENV > $VOL/usr/local/etc/rabbitmq/rabbitmq-env.conf
# tame epmd port usage
ERL_EPMD_ADDRESS="127.0.0.7"
ERL_EPMD_DIST_BIND="127.0.0.7"
# cage rabbitmq to localhost IPv6
HOSTNAME=localhost
RABBITMQ_NODE_IP_ADDRESS="::7"
EOENV

cat <<EOCONF > $VOL/usr/local/etc/rabbitmq/rabbitmq.config
%% -*- tab-width: 4;erlang-indent-level: 4;indent-tabs-mode: nil -*-
%% ex: ft=erlang ts=4 sw=4 et
[
 %% restrict management port to IPv6 only
 {rabbitmq_management, [{listener, [{port, 15672},
                                    {ip, "::7"} ]}]},
 {rabbit,
  [
   %% replace default account
   {default_vhost,       <<"/">>},
   {default_user,        <<"elmer">>},
   {default_pass,        <<"fudd">>},
   {default_permissions, [<<".*">>, <<".*">>, <<".*">>]},
   {default_user_tags, [administrator]}]}].
EOCONF

echo '[rabbitmq_management,rabbitmq_management_visualiser,rabbitmq_stomp,rabbitmq_amqp1_0,rabbitmq_mqtt].' \
    > $VOL/usr/local/etc/rabbitmq/enabled_plugins

chmod 0440 $VOL/usr/local/etc/rabbitmq/*
chown root:rabbitmq $VOL/usr/local/etc/rabbitmq/*

Feel free to use a smaller set of plugins, or a less restrictive set of permissions. The intent here is that the configuration of the iocage, and the decreased permissions of the jail user, prevents an attacker from changing the rabbitmq configuration. An even stronger configuration would be to put the config files on a read-only ZFS partition. This would require an attacker to break out of the rabbitmq user to the jail root, & out of the jail into the host OS. Let’s hope my family photos are just not that interesting.

Secure remote connectivity with spiped

What we require are 3 things:

  • resilient & reliable inter-node transport
  • straightforwards security using private keys
  • independent of Erlang/OTP

While it’s possible to set up Erlang and RabbitMQ for SSL, my experience with SSL support in OTP has been unreliable and variable between releases. Also, keeping a CA setup is problematic in itself, because x509 certificates are frankly a confusing pile of steaming crap. A simpler alternative is presented, using a single symmetrics key providing end-to-end encrypted tunnels.

You can read more about spiped elsewhere, but its provenance is from a highly respected cryptographer with significant practical experience, and its small codebase significantly reduces the chance of both bugs, and of exploitable design and implementation errors.

In comparison to autossh or similar tunnel tools, spiped has only 1 function: securing connections. It supports a star model (like a webserver), decrypting individual connections from different remote peers, handles TCP heartbeats and timeouts per peer, and maps inbound or outbound ports to local ones.

Installation is embarassingly simple, and there is just 1 file to configure, along with generation of the symmetric key and transferring that to each end.

# embarassingly easy install follows
pkg install -y sysutils/spiped
# create the symmetric key
dd if=/dev/urandom of=/usr/local/etc/rabbitmq/spiped.key bs=32 count=1
chown root:wheel /usr/local/etc/rabbitmq/spiped.key
chmod 0400 /usr/local/etc/rabbitmq/spiped.key
# transfer this file to other systems securely

# /etc/rc.conf.d/spiped
spiped_enable="YES"
spiped_pipes="RMQ RMQADMIN"

spiped_pipe_RMQ_mode="server"
spiped_pipe_RMQ_source="[::0]:5672"
spiped_pipe_RMQ_target="[::7]:5672"
spiped_pipe_RMQ_key="/usr/local/etc/rabbitmq/spiped.key"

spiped_pipe_RMQADMIN_mode="server"
spiped_pipe_RMQADMIN_source="[::0]:15672"
spiped_pipe_RMQADMIN_target="[::7]:15672"
spiped_pipe_RMQADMIN_key="/usr/local/etc/rabbitmq/spiped.key"

It is possible to use different keys for the rabbitmq admin interface on port 15672, instead of the normal user interface on port 5672, or to create multiple pipes for different servers to connect with. Exercise to reader and so forth.

Open the Warrens

Before we do that, let’s recap:

  • rabbitmq is running as a normal unprivileged user inside a FreeBSD jail
  • rabbitmq is available on IPv6 loopback only for users & management
  • external access to rabbitmq is on the usual ports, but only via spiped
  • we have restricted erlang’s epmd to IPv4 loopback only for management

Seems ok, let’s go.

Unleash the Bunnies

service spiped start
iocage start rabbit
# if you are curious check the logs
tail -F $VOL/var/log/rabbitmq/rabbit*

Check Ports

Assuming we’ve done our job, the ports should be nice and clean:

sockstat -4 -6 | egrep '4369|567|ADDRESS'

USER     COMMAND    PID   FD PROTO  LOCAL ADDRESS         FOREIGN ADDRESS
rabbitmq epmd       51173 3  tcp4   127.0.0.7:4369        *:*
rabbitmq epmd       51173 5  tcp4   127.0.0.7:4369        127.0.0.7:50944
rabbitmq beam.smp   51103 25 tcp4   127.0.0.7:25672       *:*
rabbitmq beam.smp   51103 28 tcp4   127.0.0.7:50944       127.0.0.7:4369
rabbitmq beam.smp   51103 31 tcp6   ::7:15672             ::7:21511
rabbitmq beam.smp   51103 39 tcp6   ::7:5672              *:*
rabbitmq beam.smp   51103 40 tcp6   ::7:15672             *:*
rabbitmq beam.smp   51103 46 tcp6   ::7:15672             ::7:20431
root     spiped     42796 3  tcp6   *:15672               *:*
root     spiped     42796 6  tcp6   2a01:4f8:200:12cf::7:15672 2001:1620:f00:8287:384a:b197:f909:4f83:54837
root     spiped     42796 7  tcp6   ::7:21511             ::7:15672
root     spiped     42796 10 tcp6   2a01:4f8:200:12cf::7:15672 2001:1620:f00:8287:384a:b197:f909:4f83:54816
root     spiped     42796 11 tcp6   ::7:20431             ::7:15672
root     spiped     42793 3  tcp6   *:5672                *:*

Connect Remotely

Assuming you’ve copied the symmetric key securely to a local workstation, run spiped again, to provide the other end of the encryption tunnel:

spiped -k /usr/local/etc/skunkwerks/spiped.key -e -s ::0:15672 -t rabbit.skunkwerks.at:15672
spiped -k /usr/local/etc/skunkwerks/spiped.key -e -s ::0:5672  -t rabbit.skunkwerks.at:5672

These processes will just disappear into the background, but could be run by some master daemon process if preferred.

Show me the GUI

In your browser, visit http://localhost:15672/#/nodes/rabbit%40localhost to see your rabbitmq admin remote console from inside the jail delivered over a secured piping hot connection!

Thoughts

Clearly this isn’t an apples-to-apples comparison with docker, but iocage and the FreeBSD systems we touched are well-documented, very stable, and high performance. While I’ve used the latest release of FreeBSD all of the above works just fine on FreeBSD 9.x, and probably even 8 if you have to. We are not pushed into running a custom kernel to gain performance nor features.

Finally, iocage itself is straightforward shell scripts, so modifications or understanding is easily gained, and there are no 3rd party registries or services that you have to rely on. Feel free to use an HTTP server of your choice, or host your images on some fancy S3-like cloud service.

Docker has the mindshare but FreeBSD has the Power To Serve.