Adventures with Elixir HTTP clients

Tuesday, 13 Apr 2021 Tags: debugging elixir http recon

Over the last few days, I have delved into what should have been a trivial task:

Retrieve a single json doc, via HTTP, from a host that might only be accessible over IPv6, without that information being known prior.

The last bit is the kicker. It turns out, that this is only possible, to my knowledge, with katipo, a NIF-based client using libcurl, or, a rather curious hack using inet64_tcp, originally from yandex, which I polished lightly to build using a recent rebar3.

The problem is, in essence, when you ask to GET http://example.net/, we cannot be certain, whether this host example.net should be:

accessible over IPv6 and IPv4
accessible over neither of them, because it’s a bad day
accessible over only one of them, but the one we checked is down
accessible over only one of them, and we’re in luck

And if we make 3 or 4 connections, including retries, to the wrong server, the user delay will start to add up. For a server style deployment, this will be unacceptable.

However, this series of posts will be more about debugging BEAM projects, and less about actually fixing HTTP clients.

Along this path, we have a chance to use Erlang project from an iex session, without having a mix project, and also to fiddle with real-time debugging of a running system. This is, without question, my favourite BEAM feature, and it’s sadly neither wildly known, nor heavily used. How tragic.

On FreeBSD, we require just a few packages to try things out:

devel/elixir-hex
devel/erlang-recon http://ferd.github.io/recon/recon.html
devel/rebar3 https://rebar3.org/
ftp/curl https://curl.se/libcurl/
lang/elixir
lang/erlang-runtime23

That recon one is of interest, as we’ll use this to debug in real time what’s happening with the various HTTP clients, and to divine their each and every perfidious inaction that consumed the first part of my week.

katipo an Erlang NIF-based libcurl HTTP client

Now, curl is definitely AWESOME and is probably the command-line tool I’ve been using the longest, since ~ 1998 or maybe even earlier, and it’s seen an incredible amount of battle-hardened testing, and plenty of vulnerability scanning, but it’s also a large library, and with such a large surface, our BEAM stability might be at risk. That said, it has extensive use, so, as we say in New Zealand, “She’ll be right, mate”.

Compilation is easy - note how we can have multiple Erlang versions pre-built from FreeBSD packages, and so easily accessed:

$ export PATH=/usr/local/lib/erlang23/bin:$PATH
$ git clone https://github.com/puzza007/katipo
$ cd katipo && rebar3 compile
===> Fetching rebar3_hex v6.10.2
...
===> Compiling metrics
===> Compiling worker_pool
gmake: Entering directory 'c_src'
cc -O3 -std=c99 -finline-functions ...
gmake: Leaving directory 'c_src'
===> Analyzing applications...
===> Compiling katipo

$ iex --name katipo -pa _build/default/lib/*/ebin \
                    -pa /usr/local/lib/erlang/lib/recon-*/ebin

And with that last trick, we launch an iex session, and allow loading both the external recon library, as well as the pre-built BEAM files from our application, even though this is not an Elixir / Mix application, but a rebar one!

Erlang/OTP 23 [erts-11.1.8] [source] [64-bit] [smp:32:32] [ds:32:32:10] [async-threads:16] [dtrace]

Interactive Elixir (1.11.4) - press Ctrl+C to exit (type h() ENTER for help)

iex1> Application.ensure_all_started :katipo

20:06:50.356 [info]  build metrics module: metrics_noop

{:ok, [:worker_pool, :metrics, :katipo]}

iex2> :katipo_pool.start(:api_server, 2 , [{:pipelining, :multiplex}])
{:ok, #PID<0.133.0>}

iex3> :katipo.req(:api_server, %{url: "http://couchdb.example.net:5984/", method: :get})
{:ok,
 %{
   body: '{"couchdb":"Welcome","version":"3.1.1"...}',
   cookiejar: [],
   headers: [
     {"Cache-Control", "must-revalidate"},
     {"Content-Length", "394"},
     {"Content-Type", "application/json"},
     {"Date", "Tue, 13 Apr 2021 20:07:12 GMT"},
     {"Server", "CouchDB/3.1.1 (Erlang OTP/22)"},
     {"X-Couch-Request-ID", "4adab452bd"},
     {"X-CouchDB-Body-Time", "0"}
   ],
   status: 200
 }}

Nothing particularly exciting here, other than it’s very nice & clean API to work with - set up a pool, make a request, get some sweet IPv6 HTTP response back. Thanks Paul!

httpc from Erlang stdlib

We’ll do all our work in an iex session again:


iex(8)> Application.ensure_all_started(:inets)
{:ok, []}

iex(9)> Application.ensure_all_started(:ssl)
{:ok, []}

iex(10)> :httpc.request('http://couchdb.example.net:5984/')
{:error,
 {:failed_connect,
  [{:to_address, {'couchdb.example.net', 5984}}, {:inet, [:inet], :nxdomain}]}}

Oh dear - we can see what’s happening - httpc is looking up only the IPv4 record - the A RR in DNS parlance, and not checking the AAAA record that has our IPv6 mapping. We get an :nxdomain error, aka “non-existent domain” which is a hard fail.

httpc requires setting an overall client option, which then forces subsequent connections over IPv6, both for DNS name resolution, and the subsequent connection. Let’s try that:

iex(18)> :httpc.info()
[
  handlers: [],
  sessions: {[], [], []},
  options: [
    proxy: {:undefined, []},
    https_proxy: {:undefined, []},
    pipeline_timeout: 0,
    max_pipeline_length: 2,
    max_keep_alive_length: 5,
    keep_alive_timeout: 120000,
    max_sessions: 2,
    cookies: :disabled,
    verbose: false,
    ipfamily: :inet,
    ip: :default,
    port: :default,
    socket_opts: [],
    unix_socket: :undefined
  ],
  cookies: [session_cookies: []]
]
iex(19)> :httpc.set_options([{:ipfamily, :inet6}])
:ok
iex(20)> :httpc.info()
[
  handlers: [],
  sessions: {[], [], []},
  options: [
    proxy: {:undefined, []},
    https_proxy: {:undefined, []},
    pipeline_timeout: 0,
    max_pipeline_length: 2,
    max_keep_alive_length: 5,
    keep_alive_timeout: 120000,
    max_sessions: 2,
    cookies: :disabled,
    verbose: false,
    ipfamily: :inet6,
    ip: :default,
    port: :default,
    socket_opts: [],
    unix_socket: :undefined
  ],
  cookies: [session_cookies: []]
]
iex(21)> :httpc.request('http://couchdb.example.net:5984/')
{:ok,
 {{'HTTP/1.1', 200, 'OK'},
  [
    {'cache-control', 'must-revalidate'},
    {'date', 'Tue, 13 Apr 2021 21:02:42 GMT'},
    {'server', 'CouchDB/3.1.1 (Erlang OTP/22)'},
    {'content-length', '394'},
    {'content-type', 'application/json'},
    {'x-couch-request-id', 'a9dc99a22f'},
    {'x-couchdb-body-time', '0'}
  ],
  '{"couchdb":"Welcome","version":"3.1.1",...}'}}

While that’s functional, we have forced :inet6 on all connections using :httpc, and we should do this only for this specific host.

This can be done using httpc profiles, which effectively starts a separate GenServer process, that we address directly via atom name as a final parameter in all our :httpc function calls. This isolates any custom settings, such as timeouts, HTTP pipelining, and of course our IPv6 settings, from other :httpc clients.

iex(1)> Application.ensure_all_started(:ssl)
{:ok, [:crypto, :asn1, :public_key, :ssl]}
iex(2)> Application.ensure_all_started(:inets)
{:ok, [:inets]}
iex(3)> url = 'http://couchdb.example.net:5984/'
'http://couchdb.example.net:5984/'
iex(4)> {:ok, profile} = :inets.start(:httpc, profile: :tcp6)
{:ok, #PID<0.150.0>}
iex(5)> :httpc.info(:tcp6)
[
  handlers: [],
  sessions: {[], [], []},
  options: [
    proxy: {:undefined, []},
    https_proxy: {:undefined, []},
    pipeline_timeout: 0,
    max_pipeline_length: 2,
    max_keep_alive_length: 5,
    keep_alive_timeout: 120000,
    max_sessions: 2,
    cookies: :disabled,
    verbose: false,
    ipfamily: :inet,
    ip: :default,
    port: :default,
    socket_opts: [],
    unix_socket: :undefined
  ],
  cookies: [session_cookies: []]
]
iex(6)> :httpc.set_options([{:ipfamily, :inet6}], :tcp6)
:ok
iex(7)> :httpc.info(:tcp6)
[
  handlers: [],
  sessions: {[], [], []},
  options: [
    proxy: {:undefined, []},
    https_proxy: {:undefined, []},
    pipeline_timeout: 0,
    max_pipeline_length: 2,
    max_keep_alive_length: 5,
    keep_alive_timeout: 120000,
    max_sessions: 2,
    cookies: :disabled,
    verbose: false,
    ipfamily: :inet6,
    ip: :default,
    port: :default,
    socket_opts: [],
    unix_socket: :undefined
  ],
  cookies: [session_cookies: []]
]
iex(8)> :httpc.request('http://couchdb.example.net:5984/', :tcp6)
{:ok,
 {{'HTTP/1.1', 200, 'OK'},
  [
    {'cache-control', 'must-revalidate'},
    {'date', 'Tue, 13 Apr 2021 21:17:05 GMT'},
    {'server', 'CouchDB/3.1.1 (Erlang OTP/22)'},
    {'content-length', '394'},
    {'content-type', 'application/json'},
    {'x-couch-request-id', '7bde722417'},
    {'x-couchdb-body-time', '0'}
  ],
  '{"couchdb":"Welcome","version":"3.1.1", ...}'}}

iex(9)> :inets.stop(:httpc, :tcp6)
:ok

Frustratingly, there is no “automatic” connection to a given host, without first looking up if it has both IPv6 and IPv4 support, and in addition, until we try to connect, we may not even know then if this should be one connection type or the other. So, at least for this use case, httpc is out of the picture, unless we make a small wrapper, that does DNS lookup first, checks and selects the appropriate connection type, and then calls httpc itself. This would be suitable for small deployments, or occasional updates within a BEAM VM, but not for any heavy usage production deployment.

More to come next time, when we delve further into using recon_trace in anger to see what’s happening on the inside.