Podman containers can use Nebula mesh networking interfaces, allowing your containerised applications to participate in secure overlay networks across multiple hosts and data centers.

This builds on the ZFS hooks approach, but instead of mounting datasets, we attach Nebula network interfaces to containers. This allows containers to communicate securely across the mesh network while maintaining network isolation from the host system.

We will use Podman annotations to specify which Nebula network the container should join, and hooks to create and attach the appropriate interface.

Prerequisites

• Nebula installed and configured on the host system
• A working Nebula mesh network with certificates
• Podman with hooks support configured
• FreeBSD jail networking knowledge

Prepare the Nebula Hooks

Similar to the ZFS hooks, we’ll create hooks that run at container lifecycle events to manage Nebula interfaces.

First, add the hook metadata as /usr/local/etc/containers/hooks.d/nebula.json:

{
  "version": "1.0.0",
  "hook": {
    "path": "/usr/local/etc/containers/hooks.d/nebula.sh"
  },
  "when": {
    "annotations": {
      "^nebula.network$": ".+"
    }
  },
  "stages": ["createRuntime", "poststop"]
}

The hook will only run when the nebula.network annotation is present, allowing containers to opt-in to Nebula networking.

The Nebula Hook Script

Create the hook script as /usr/local/etc/containers/hooks.d/nebula.sh:

#!/bin/sh -e
set -o pipefail

INPUT=$(cat - | tee -a /var/log/oci/nebula.json)
ID=$(echo $INPUT | jq -r .id || exit 1)
STATUS=$(echo $INPUT | jq -r .status || exit 1)
NETWORK=$(echo $INPUT | jq -r '.annotations."nebula.network"')

# Nebula configuration directory
NEBULA_DIR="/usr/local/etc/nebula"
CONFIG_FILE="${NEBULA_DIR}/${NETWORK}.yml"

# Verify the nebula config exists
if [ ! -f "$CONFIG_FILE" ]; then
    echo "Nebula config not found: $CONFIG_FILE" >&2
    exit 1
fi

# Extract the nebula IP from the config
NEBULA_IP=$(awk '/^  cert:/ {getline; while(getline && /^    /) {if(/ip:/) {gsub(/.*ip: /, ""); gsub(/\/.*/, ""); print; exit}}}' "$CONFIG_FILE")

if [ -z "$NEBULA_IP" ]; then
    echo "Could not determine Nebula IP from $CONFIG_FILE" >&2
    exit 1
fi

# Interface names
HOST_IF="nebula_${ID:0:12}"
JAIL_IF="nebula0"

if [ "$STATUS" = "created" ]; then
    # Create epair interfaces
    EPAIR=$(ifconfig epair create)
    EPAIR_A="${EPAIR}a"
    EPAIR_B="${EPAIR}b"
    
    # Rename the host side interface
    ifconfig "$EPAIR_A" name "$HOST_IF"
    
    # Configure jail networking permissions
    jail -vm name="$ID" allow.raw_sockets=1 allow.socket_af=1
    
    # Move the jail side interface into the container
    ifconfig "$EPAIR_B" vnet "$ID"
    
    # Configure the interface inside the jail
    jexec "$ID" ifconfig "$EPAIR_B" name "$JAIL_IF"
    jexec "$ID" ifconfig "$JAIL_IF" inet "$NEBULA_IP/24" up
    
    # Start nebula inside the container
    jexec "$ID" /usr/local/bin/nebula -config "$CONFIG_FILE" &
    
    # Store the PID for cleanup
    echo $! > "/tmp/nebula_${ID}.pid"

elif [ "$STATUS" = "stopped" ]; then
    # Stop nebula process
    if [ -f "/tmp/nebula_${ID}.pid" ]; then
        kill $(cat "/tmp/nebula_${ID}.pid") 2>/dev/null || true
        rm -f "/tmp/nebula_${ID}.pid"
    fi
    
    # Clean up host interface
    ifconfig "$HOST_IF" destroy 2>/dev/null || true
fi

Make the script executable:

# chmod +x /usr/local/etc/containers/hooks.d/nebula.sh

Nebula Configuration

For each network you want containers to join, create a separate Nebula configuration file. For example, /usr/local/etc/nebula/mesh.yml:

pki:
  ca: /usr/local/etc/nebula/certs/ca.crt
  cert: /usr/local/etc/nebula/certs/container.crt
  key: /usr/local/etc/nebula/certs/container.key

static_host_map:
  "192.168.100.102": ["193.123.60.15:54321"]

lighthouse:
  am_lighthouse: false
  hosts:
    - "192.168.100.102"

listen:
  host: 0.0.0.0
  port: 54321

punchy: true

tun:
  dev: nebula0
  drop_local_broadcast: false
  drop_multicast: false
  tx_queue: 500
  mtu: 1300

firewall:
  outbound:
    - port: any
      proto: any
      host: any
  inbound:
    - port: any
      proto: any
      host: any

Generate Container Certificates

Create certificates for containers that will join the mesh:

# cd /usr/local/etc/nebula/certs
# nebula-cert sign -name container -ip 192.168.100.200/24 -groups containers

This creates container.crt and container.key files that the hook script will use.

Running Containers with Nebula

Use the nebula.network annotation to specify which network configuration the container should use:

# podman run -it --rm \
  --annotation='nebula.network=mesh' \
  freebsd:14.1-RELEASE

Inside the container, you should see the Nebula interface:

# ifconfig nebula0
nebula0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1300
        inet 192.168.100.200 netmask 0xffffff00
        groups: tun
        Opened by PID 1234

The container can now communicate with other nodes in the Nebula mesh using the overlay network addresses.

Advanced Configuration

Multiple Networks

You can create multiple Nebula configurations for different networks or security zones:

# Create configs for different environments
/usr/local/etc/nebula/production.yml
/usr/local/etc/nebula/staging.yml
/usr/local/etc/nebula/development.yml

Then specify which network when running containers:

# Production container
podman run --annotation='nebula.network=production' ...

# Development container  
podman run --annotation='nebula.network=development' ...

Dynamic IP Assignment

For more dynamic setups, you could extend the hook script to:

• Query a DHCP server or IP management system
• Use different IP ranges for different container types
• Implement IP address pooling and cleanup

Security Groups

Leverage Nebula’s group-based firewall rules to implement network segmentation between different types of containers:

firewall:
  inbound:
    - port: 80
      proto: tcp
      groups: 
        - web
    - port: 5432
      proto: tcp
      groups:
        - database

Troubleshooting

Check the hook logs:

# tail -f /var/log/oci/nebula.json

Verify Nebula connectivity from inside the container:

# jexec <container-id> nebula-cert print -path /usr/local/etc/nebula/certs/container.crt
# jexec <container-id> ping 192.168.100.102

Check interface status:

# ifconfig | grep nebula
# jexec <container-id> ifconfig nebula0

Conclusion

This approach provides secure, encrypted networking for containers across multiple hosts using Nebula’s mesh networking capabilities. Containers can communicate directly with each other and with other nodes in the mesh, regardless of their physical location or network topology.

The hook-based approach keeps the networking configuration declarative and ensures proper cleanup when containers are stopped, making it suitable for both development and production environments.

Thanks

Thanks to the Nebula team for creating such a robust mesh networking solution, and to the Podman developers for the flexible hook system that makes this integration possible.