HAProxy has had a built-in ACME client since 3.2, but until recently the DNS-01 challenge required a fair bit of extraneous scripting.

HAProxy 3.4 introduced significant improvements to its ACME support, and the upcoming HAProxy 3.5 will add DNS-PERSIST-01 as well, even if none of the CA providers support it yet — it’s still in draft.

HAProxy 3.5 adds Lua events, so it’s now possible to issue and renew even wildcard certificates entirely from within HAProxy, similar to what Caddy and other HTTP servers do.

This post explains how I deployed it on FreeBSD, for my extremely popular site, along with wildcard support for a plethora of equally high-traffic subdomains.

Pre-requisites

• HAProxy 3.5-dev1 or later, built with Lua. On FreeBSD that requires net/haproxy-devel@lua and D60463 which is not yet committed. Note that 3.4 has DNS-01 but not the Lua events we rely on.
• security/ca_root_nss for outbound TLS
• A DNS zone with an API. I use a classic CNAME delegation to allow updating all DNS records with a low-risk dedicated ACME domain, instead of my primary zones, hosted at the iniquitous but free CloudFlare.

DNS

The ACME server looks for a TXT record at _acme-challenge.<domain> and will follow CNAMEs to find it. So in the primary zone, set once:

_acme-challenge.skunkwerks.at.  CNAME  deadcafe.example.org.

example.org lives on Cloudflare. Create an API token scoped to Zone:DNS:Edit on that zone only, and limited to the specific outbound IP of the HAProxy server. If the token leaks, an attacker can write TXT records in a zone nobody uses for anything else.

Overall, the Lua handler just uses 4 API calls:

GET    /client/v4/zones?name=example.org&status=active     → zone id
POST   /client/v4/zones/<zone>/dns_records                   → TXT, ttl 60
GET    /client/v4/zones/<zone>/dns_records?type=TXT&name=…   → ids to delete
DELETE /client/v4/zones/<zone>/dns_records/<id>

Error 81058, an identical record already exists, can be safely ignored. You will see it whenever HAProxy restarts mid-challenge, and retries with the same token.

haproxy.conf

The config file requires changes in three new sections.

global

• enable the experimental ACME directives
• feed the internal HTTP client a CA bundle
• load the handler

global
  expose-experimental-directives
  httpclient.resolvers.prefer ipv4
  httpclient.ssl.ca-file /usr/local/share/certs/ca-root-nss.crt
  tune.lua.bool-sample-conversion normal
  lua-load /usr/local/etc/haproxy/acme-cloudflare.lua

acme

The new ACME section describes the CA, and a crt-store to bind a given certificate file to it:

acme letsencrypt
  directory       https://acme-v02.api.letsencrypt.org/directory
  contact         spam@skunkwerks.at
  account-key     /usr/local/etc/haproxy/ssl/letsencrypt.account.key
  challenge       dns-01
  challenge-ready cli,delay
  dns-delay       15s
  keytype         ECDSA
  curves          P-384

crt-store certs
  crt-base /usr/local/etc/haproxy/ssl
  key-base /usr/local/etc/haproxy/ssl
  load crt "skunkwerks.at.pem" acme letsencrypt domains "skunkwerks.at,*.skunkwerks.at" alias "skunkwerks.at"

challenge-ready cli,delay means to wait for the Lua handler to say the record is in place, then wait 15 seconds, and then request the CA to validate. The alternative is cli,dns which makes HAProxy’s own resolver verify the TXT record first, but it returned REFUSED for my CNAME-delegated names, and Let’s Encrypt will ask the authoritative servers directly anyway.

frontend

Finally the frontend references its cert from the store, instead of a file as previously:

frontend default_fe
  bind      :443 ssl alpn h2,http/1.1
  ssl-f-use crt "@certs/skunkwerks.at"

If the skunkwerks.at certificate does not exist yet, HAProxy starts up and generates a self-signed placeholder, and requests an official one immediately. This way, it’s entirely possible to deploy this to a new machine, have it start up correctly, and with certificates, in a few seconds.

The Lua handler

This script is modelled off the acme-gandi-livedns.lua script distributed with HAProxy.

HAProxy generates an ACME_DEPLOY event once per authorization, with the TXT value to publish, and another, ACME_NEWCERT, when the certificate is installed. The handler maps the challenged domain to the delegated FQDN, writes the record, then calls ACME.challenge_ready(crt, domain):

-- deployed[crt][fqdn] = zone_id, so ACME_NEWCERT knows what to clean up
local deployed = {}

local function on_deploy(event, data, sub, when)
    local crt, domain, record = data.crtname, data.domain, data.dns_record
    core.register_task(function()
        local fqdn = CHALLENGE_DELEGATE[domain] or "_acme-challenge." .. domain
        local zone_id, record_id = dns_set_txt(fqdn, record)
        if not record_id then
            core.log(core.alert, string.format(
                "acme: aborting challenge for crt=%s domain=%s", crt, domain))
            return
        end
        deployed[crt] = deployed[crt] or {}
        deployed[crt][fqdn] = zone_id

        local ok, ret = pcall(ACME.challenge_ready, crt, domain)
        if not ok and tostring(ret):find("not found") then
            core.log(core.info, string.format(
                "acme: crt=%s domain=%s already marked ready", crt, domain))
        elseif not ok then
            core.log(core.alert, string.format(
                "acme: challenge_ready error for crt=%s domain=%s: %s", crt, domain, ret))
        elseif ret == 0 then
            core.log(core.notice, string.format(
                "acme: all challenges ready for crt=%s, validation starting", crt))
        else
            core.log(core.info, string.format(
                "acme: crt=%s domain=%s ready, %d challenge(s) still pending",
                crt, domain, ret))
        end
    end)
end

local function on_newcert(event, data, sub, when)
    local crt = data.crtname
    core.register_task(function()
        for fqdn, zone_id in pairs(deployed[crt] or {}) do
            dns_del_txt(zone_id, fqdn)   -- deletes every TXT at that name
        end
        deployed[crt] = nil
    end)
end

-- subscribe only when we can actually act, see below
if CLOUDFLARE_DNS_API_TOKEN and pcall(core.event_sub, {"ACME_DEPLOY"}, on_deploy) then
    core.event_sub({"ACME_NEWCERT"}, on_newcert)
else
    core.log(core.alert, "acme: dns-01 automation disabled, answer challenges "
        .. "manually with 'acme challenge_ready' on the master CLI")
end

Here is the full lua script for reference, parameterised for multiple domains with Ansible.

ACME_DEPLOY events

We ask for a single certificate covering two names, skunkwerks.at and *.skunkwerks.at. Let’s Encrypt treats those as two separate authorizations, each with its own TXT token, but both tokens have to be published at the same DNS name, _acme-challenge.skunkwerks.at, because a wildcard’s challenge record must live at the base domain name.

HAProxy’s implementation publishes a single ACME_DEPLOY event per authorisation, so the Lua handler will be called twice, which is fine, but the domain field is the bare skunkwerks.at in both cases. The ACME authorization object from the CA already names the bare domain, and signals the wildcard with a separate "wildcard": true flag. HAProxy reads that flag, but does not pass it through to the Lua event.

After this little quirk, things move much as expected, the handler writes the TXT records containing the proof, and calls ACME.challenge_ready(crt, domain). This flags both the apex and the wildcard at the same time, even though the wildcard TXT record has not yet been issued, and we have no way of disambiguating the two requests.

A few moments later, the next handler event is published, and challenge_ready() is called again. Now, nothing is left to mark, so HAProxy returns domain not found, which from our perspective isn’t really an error, so the Lua handler logs it at INFO rather than a failure.

I guess this is sort of a bug, as after the first call, the CLI condition thinks its satisfied, even though we need to wait for the second event. Either way, the cli,delay value means that HAProxy waits dns-delay 15 seconds before asking the ACME server to validate, and as the other event arrived and is delivered to CloudFlare well before that, nobody notices.

challenge_ready ought only to mark a single authorization per call, or alternatively the wildcard should be passed through somehow so the handler is aware of the difference.

The API token comes from the environment, CLOUDFLARE_DNS_API_TOKEN, never from the config file. If it is missing, the handler does not raise an error. Instead, it logs an alert & skips the subscriptions, so HAProxy still starts, and configtest also will still pass. In production, you should have monitoring to detect both HAProxy being down, and an invalid or close-to-expiration certificate being presented.

Files on disk

/usr/local/etc/haproxy/haproxy.conf                 0640 root:wheel
/usr/local/etc/haproxy/acme-cloudflare.lua          0640 root:wheel
/usr/local/etc/haproxy/ssl/                         0750 root:wheel
/usr/local/etc/haproxy/ssl/letsencrypt.account.key  0600 root:wheel
/root/bin/haproxy-dump-certs.sh                     0700 the dump script
/usr/local/etc/haproxy/ssl/skunkwerks.at.pem        0600 written by the dump script
/etc/rc.conf.d/haproxy                              0400 the daemon config

I put the FreeBSD rc settings into a separate /etc/rc.conf.d/haproxy file, so that I can set its ownership and permissions to root:wheel and 0400 respectively, as it now contains the CloudFlare DNS API token.

haproxy_enable=YES
haproxy_config=/usr/local/etc/haproxy/haproxy.conf
haproxy_flags="-W -f ${haproxy_config} -p /var/run/haproxy.pid -S /var/run/haproxy.sock,mode,660,user,root,group,operator"
haproxy_env='CLOUDFLARE_DNS_API_TOKEN=...'

-W runs master-worker mode and -S opens the master CLI socket that the ACME commands need. It is the only socket: there is no stats socket line in global any more, as the master CLI reaches the worker with an @1; prefix, which is what every command below uses.

The bind options after the comma are what stats socket used to set via the haproxy.conf file. Do not use -Ws, as that is the systemd variant and forces foreground mode, your rc script will never exit!

With haproxy_env, the FreeBSD rc script runs haproxy -c as its start_precmd, and rc.subr only puts haproxy_env into the environment of the daemon itself, not the precmd, so the config check runs without the token. If the Lua script treats a missing token as fatal, the precmd fails and the service never starts, with nothing more helpful than failed precmd routine for haproxy in the log. Hence the soft failure in the handler above.

Persisting Certificates

HAProxy keeps issued certificates in memory only, so a restart brings back the placeholder TLS cert, and triggers a fresh order, which counts against Let’s Encrypt’s five-duplicates-a-week limit. A small cron job and script dumps this to disk easily enough:

#!/bin/sh -e
set -o pipefail
umask 077
DIR=/usr/local/etc/haproxy/ssl
cli() { echo "@1; $1" | nc -NU -w 5 /var/run/haproxy.sock; }
for cert in $(cli "show ssl cert" | grep '^@certs/'); do
  info=$(cli "show ssl cert $cert")
  file=$(echo "$info" | sed -n 's/^Crt filename: //p')
  [ "$(echo "$info" | sed -n 's/^Issuer: //p')" = "/CN=expired" ] && continue
  cli "dump ssl cert $cert" > $DIR/$file.new
  if [ -s $DIR/$file.new ]; then
    mv $DIR/$file.new $DIR/$file
  else
    rm -f $DIR/$file.new
  fi
done

/CN=expired is the placeholder’s issuer, so we need to skip that self-signed cert, but store the rest. The dump will also include the private key, so the righteous umask 077 is required. The Crt filename line gives the real file name; the store entry you loop over is the alias.

Run this from cron every few hours and, importantly, from whatever reloads or restarts HAProxy, just before reloading.

The First Run

tail -f /var/log/messages | grep --line-buffered acme:
echo "@1; acme status" | nc -NU /var/run/haproxy.sock

A good run looks like:

• Starting update
• two TXT record set lines
• all challenges ready followed by waiting 15s
• Successful update of the certificate
• and the clean up 4 TXT record(s) deleted

This took around 30 seconds the first time here, which feels quite acceptable.

The final acme status then shows the renewal date, about a month before expiry, and from here on nothing else should ever need to touch it.

Oct  8 11:28:07 continuity haproxy[35642]: acme: @certs/skunkwerks.at: Starting update of the certificate.
Oct  8 11:28:08 continuity haproxy[35642]: acme: @certs/skunkwerks.at: dns-01 requires to set the "_acme-challenge.skunkwerks.at" TXT record to "Cf8SHk7ATE3TEBjoLUv7MfOJ6moEfA8RC4kWIhUOqpM" and use the "acme challenge_ready @certs/skunkwerks.at domain skunkwerks.at" command over the CLI
Oct  8 11:28:08 continuity haproxy[35642]: acme: @certs/skunkwerks.at: dns-01 requires to set the "_acme-challenge.skunkwerks.at" TXT record to "r5Jo8VxhU0fXM-Yt-jiGy1BiBdvP4dcaYR54hGmr8pY" and use the "acme challenge_ready @certs/skunkwerks.at domain skunkwerks.at" command over the CLI
Oct  8 11:28:09 continuity haproxy[35642]: acme: TXT record set: c5344ea3b24f337906fd9d0e4d74a9392bda946081e56e14b5231cc4.example.org in zone example.org (id=321ab9c018c41521e74ae8189dcb381a value=Cf8SHk7ATE3TEBjoLUv7MfOJ6moEfA8RC4kWIhUOqpM)
Oct  8 11:28:09 continuity haproxy[35642]: acme: all challenges ready for crt=@certs/skunkwerks.at, validation starting
Oct  8 11:28:09 continuity haproxy[35642]: acme: @certs/skunkwerks.at: dns-01: waiting 15s
Oct  8 11:28:09 continuity haproxy[35642]: acme: TXT record set: c5344ea3b24f337906fd9d0e4d74a9392bda946081e56e14b5231cc4.example.org in zone example.org (id=d4c0f905a2f3035d5b576bfc092d8c71 value=r5Jo8VxhU0fXM-Yt-jiGy1BiBdvP4dcaYR54hGmr8pY)
Oct  8 11:28:36 continuity haproxy[35642]: acme: @certs/skunkwerks.at: Successful update of the certificate.
Oct  8 11:28:38 continuity haproxy[35642]: acme: 4 TXT record(s) deleted: c5344ea3b24f337906fd9d0e4d74a9392bda946081e56e14b5231cc4.example.org

Most excellent!