Random Musings

O for a muse of fire, that would ascend the brightest heaven of invention!


Do I need a Load Balancer for my Phoenix Application

Wednesday, 5 Jan 2022 Tags: elixirphoenix

This post is in response to an Elixir Forum post on the same subject, albeit from a couple of years ago. I wrote this post, realised it was an old thread, and didn’t really want to lose the content. Enjoy.

It’s not necessary to run BEAM apps as root. I’ve never done it, on any OS. And you shouldn’t either.

TLDR

  • don’t run as root, use capabilities instead
  • low ports are no longer significant
  • consider injecting your TLS certs and other secrets via one-time “cubbyholes” with tools like vault
  • security is a journey, not a destination — “constant vigilance”
  • yes, you should use a load balancer

Erlang VM security

In general you should consider your attack surface, and also your threat model, to make security related decisions.

The Erlang VM is thousands of times larger in lines of code and complexity than even a high performance modern proxy, and modern proxies are usually well audited on a regular basis, if not publically, definitely privately, and many bugs are fixed.

I am not aware of such a security review done ever for the BEAM. But I’d love to read one.

Odds are, your BEAM VM has far more undiscovered security vulnerabilities, than a battle hardened internet facing load balancer.

Privileged Ports

The concept of privileged ports is obsolete, it came from the time when you had trusted hosts, who provided trusted services on trusted ports, and there was no encryption, and the sole admin was called Gale and they were down the hallway in the other department, and wasn’t around after 2pm on fridays.

Nowadays, every service you use requires encryption, at worst opportunistic, and at best with pre-shared keys or certificates.

Protecting certain ports adds no real trust, as nobody assumes anymore that your system is actually trust-worthy. Even web browsers have sandbox code, parser validation code, and much much more, all assuming that your HTTP server is malicious and serving malware.

Just Use a Load Balancer

As somebody who always uses load balancers, even for things they weren’t intended for, the main advantages are being able to block or shape malicious traffic even at high load and volume without impacting the application, add additional backends dynamically, deploy and reboot services transparently, doing weird things for debugging, encrypting and limiting outbound connections as well, and excellent metrics to boot.

Eventually somebody will figure out that they can extract cookies and so forth from your client, and send malicious traffic down your websockets.

Then you will find more and more ways to get your load balancer to assist your very frustrating and awkward situation. Good luck adding that into your project under pressure and at the last minute.

So just use a load balancer or firewall to redirect to non-privileged ports, or use capabilities to enable non-root processes to access that port.

UNIX Capabilities

  • on linux, use capabilities(7) on the beam.smp file itself, setcap 'cap_net_bind_service=+ep' /usr/local/lib/erlang/bin/beam.smp, not the shell script that runs it. IIRC that’s correct, if not somebody will try it and tell me what’s wrong.

  • with systemd, you can do similar things in your unit files with:

    CapabilityBoundingSet=CAP_NET_BIND_SERVICE
    AmbientCapabilities=CAP_NET_BIND_SERVICE
  • on BSD and probably macOS, use sysctl net.inet.ip.portrange.reservedhigh=442 or lower to allow the same thing. Or use the mac portacl framework to bind specific users even more tightly, on FreeBSD

Inject Your Certs

A very secure way to inject TLS credentials into an app is to use a Key Management Service (available in all modern Clowds, or via something like vault to roll your own), which pass in, typically via environment variable, a one-time use token that allows your app to obtain its keys.

This token only works once, and thus if it fails you know that it was already compromised and can act accordingly (panic! roll keys! execute your security plan!).

If it succeeds, your app now has keys and can do special things, knowing that the token and keys weren’t tampered with in transit, and can’t subsequently be used, as it’s now invalid.

More advanced usage of a KMS is to regenerate and rotate TLS certs on a regular basis, but this makes life a lot more complicated.