Categories
Technology

Bootstrapping Docker Swarm Part 4: Configuring the Load Balancers

This is part of a multi-part series on getting Docker Swarm up and running. You might want to start with the original post called Bootstrapping Docker Swarm.

There are three containers involved in the load balancers: keepalived, haproxy_frontend and haproxy_backend. The first one will again be a standalone Docker container but the other two will finally be part of a Swarm stack.

Building keepalived

Keepalived is great for load balancing one or more IP addresses across two or more servers that happen to share the same layer 3 domain. The servers that share these IP addresses must be able to correctly route the IP addresses that they are sharing. If they can’t then this won’t work.

First, you’ll need to put a configuration file for keepalived on each load balancer. This file should be maintained outside of Docker. Perhaps you could maintain it with Ansible? Then when the configuration file changes you can run this command on each load balancer:

docker kill --signal HUP keepalived

But before you can do that, what does the configuration file look like? You should put the configuration file in /srv/keepalived/keepalived.conf and make it look something like this:

global_defs {
    script_user root
    enable_script_security
    router_id lb01  # TODO changeme
}

vrrp_script check_haproxy_frontend {
    script "/usr/bin/socat -u OPEN:/dev/null UNIX-CONNECT:/usr/local/etc/haproxy/frontend/haproxy.sock"
    interval 1
}

vrrp_script check_haproxy_backend {
    script "/usr/bin/socat -u OPEN:/dev/null UNIX-CONNECT:/usr/local/etc/haproxy/backend/haproxy.sock"
    interval 1
}

vrrp_instance private {
    state BACKUP
    interface eth0
    virtual_router_id 101

    advert_int 1
    authentication {
        auth_type PASS
        auth_pass password  # TODO changeme
    }

    track_script {
        check_haproxy_frontend
        check_haproxy_backend
    }

    unicast_src_ip 10.0.0.18  # TODO changeme
    unicast_peer {
        10.159.199.19  # TODO changeme
    }

    # higher numbers have higher priority
    priority 100

    virtual_ipaddress {
        10.0.0.101  # TODO changeme
        10.0.0.102  # TODO changeme
    }

    virtual_rules {
        from 10.0.0.101/32 table 1 preference 1001  # TODO changeme
        from 10.0.0.102/32 table 1 preference 1002  # TODO changeme
    }
}
vrrp_instance public {
    state BACKUP
    interface eth1
    virtual_router_id 102

    advert_int 1
    authentication {
        auth_type PASS
        auth_pass password  # TODO changeme
    }

    track_script {
        check_haproxy_frontend
        check_haproxy_backend
    }

    unicast_src_ip 10.0.0.18  # TODO changeme
    unicast_peer {
        10.0.0.19  # TODO changeme
    }

    # higher numbers have higher priority
    priority 100

    virtual_ipaddress {
        1.2.3.201  # TODO changeme
        1.2.3.202  # TODO changeme
    }

    virtual_rules {
        from 1.2.3.201/32 table 2 preference 2001  # TODO changeme
        from 1.2.3.202/32 table 2 preference 2002  # TODO changeme
    }
}

This configuration file manages two network interfaces. Let’s break down what you need to change to make it work:

  • The value for router_id should be different on each load balancer.
  • The value for auth_pass should be changed to something that you like for a password.
  • The value for unicast_src_ip should match the IP address of the load balancer where keepalived is running and the values in unicast_peer should match the IPs of the other load balancers. (You could also just set up VRRP but I’ve had trouble making that work well. This is easier.)
  • Finally, under the virtual_ipaddress and virtual_rules section you can list all of the IP addresses that you want to load balance with keepalived.

Actually deploying keepalived is easy. You only need two files:

  • Dockerfile
  • docker-compose.yml

The Dockerfile for this container looks like this:

FROM debian:buster-slim

ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update && apt-get install -y --no-install-recommends keepalived netcat socat

VOLUME ["/etc/keepalived", "/usr/local/etc/haproxy"]
ENTRYPOINT ["/usr/sbin/keepalived", "--no-syslog", "--dont-fork", "--log-console"]

The docker-compose.yml file looks like this:

version: "3.2"

services:
  keepalived:
    build: .
    restart: always
    network_mode: host
    container_name: keepalived
    image: ${IMAGE:-registry.lab.cip.uw.edu/infrastructure/keepalived}:${VERSION?undefined VERSION}
    volumes:
      - type: bind
        source: /srv/keepalived
        target: /etc/keepalived
        read_only: true
      - type: bind
        source: /srv/haproxy/frontend
        target: /usr/local/etc/haproxy/frontend
        read_only: true
      - type: bind
        source: /srv/haproxy/backend
        target: /usr/local/etc/haproxy/backend
        read_only: true
    cap_add:
      - NET_ADMIN
      - NET_BROADCAST

You will need to create these two files on both load balancers and then deploy them like this:

mkdir -p /srv/keepalived  # this is where your configuration file goes
mkdir -p /srv/haproxy/frontend
mkdir -p /srv/haproxy/backend

VERSION=latest IMAGE=keepalived docker-compose build
VERSION=latest IMAGE=keepalived docker-compose up -d

You’ll notice that both hosts immediately go into FAULT mode. You’ll also notice that our configuration file tracks HAProxy. Since we haven’t configured HAProxy on either host both hosts think that they are dead and they thus remove their IP addresses. We’ll fix this next.

An important detail to note here is that we are mounting the path /srv/keepalived to the container and not the file /srv/keepalived/keepalived.conf. The reason for this is because when a file is mounted to a container and the file changes on the host the container will not notice but when a file in a directory that is mounted to a container changes then the container notices. If we want to change the keepalived configuration without restarting the container and rather by simply sending it a HUP signal then this is incredibly important.

Building haproxy

Now we’re getting into Docker Swarm territory! There are two instances of HAProxy running: one called “frontend” and it uses host networking to bind to IP addresses and ports. It then proxies to a Unix socket where the other instance of HAproxy runs, called “backend”. That’s the one that runs in our Swarm overlay network and actually talks to containers. Thus, there are two configuration files for these.

Because these configuration files may change and we don’t want to be restarting HAProxy just to change its configuration you should keep these files on the hosts themselves and not in their container. But because the configuration files have to be maintained identically on each host you should maintain them using some other system such as maybe Ansible? Then when the configuration file changes you can run one of these commands on each load balancer:

docker kill --signal USR2 $(docker container ls --filter name=haproxy_frontend --quiet)
docker kill --signal USR2 $(docker container ls --filter name=haproxy_backend --quiet)

So we’ll get started by manually putting the two configuration files up on the hosts. The frontend configuration should be put in /srv/haproxy/frontend/haproxy.cfg and it can look similar to this:

global
    # the max level is "emerg" which sends things to the console. we don't want
    # things going to the console so set the max to "alert" which has the same
    # effect without sending messages to everyone's console.
    log 10.0.0.20 local0 info alert
    master-worker
    nbthread 4

    # ssl configuration
    ssl-default-bind-options no-sslv3
    ssl-default-bind-ciphers ECDH+AESGCM:ECDH+CHACHA20:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS
    ssl-default-server-options no-sslv3
    ssl-default-server-ciphers ECDH+AESGCM:ECDH+CHACHA20:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS
    tune.ssl.default-dh-param 2048

    # this is used by keepalived to see if haproxy is online
    stats socket /usr/local/etc/haproxy/haproxy.sock user root group root mode 600
    stats timeout 30s

defaults
    log global
    mode http
    option httplog
    option dontlognull
    option contstats
    timeout connect 5000ms
    timeout client  600000ms
    timeout server  600000ms

##### docker registry

frontend registry-http
    bind 10.0.0.101:80
    mode http
    log global
    redirect scheme https code 301 if !{ ssl_fc }

frontend registry-https
    bind 10.0.0.101:443 ssl crt /usr/local/ssl/certs/example.com.pem
    mode http
    log global

    # tell backend that we are using ssl
    http-request set-header X-Forwarded-Proto https
    http-request set-header X-Forwarded-Port 443

    use_backend registry

backend registry
    mode http
    log global
    server backend /usr/local/etc/haproxy/sockets/registry.sock send-proxy-v2-ssl-cn check

And the backend configuration should be put in /srv/haproxy/backend/haproxy.cfg and it can look similar to this:

global
    # the max level is "emerg" which sends things to the console. we don't want
    # things going to the console so set the max to "alert" which has the same
    # effect without sending messages to everyone's console.
    log 10.0.0.20 local0 info alert
    master-worker
    nbthread 4

    # this is used to collect statistics about the proxy
    stats socket /usr/local/etc/haproxy/haproxy.sock user root group root mode 600
    stats timeout 30s

defaults
    timeout connect 10s
    timeout client 30s
    timeout server 30s
    log global
    mode http
    option httplog
    option forwardfor
    option contstats
    option dontlognull
    timeout connect 5000ms
    timeout client  600000ms
    timeout server  600000ms
    default-server init-addr libc,none

resolvers docker
    nameserver ns 127.0.0.11:53
    resolve_retries 3
    timeout resolve 1s
    timeout retry   1s
    hold other      10s
    hold refused    10s
    hold nx         10s
    hold timeout    10s
    hold valid      10s
    hold obsolete   10s

#### registry application

frontend registry-frontend
    bind /usr/local/etc/haproxy/sockets/registry.sock user root group root mode 700 accept-proxy
    mode http
    use_backend registry-backend

backend registry-backend
    mode http
    balance roundrobin
    server backend registry_registry:8080 check resolvers docker resolve-prefer ipv4

The important detail here is that they communicate over Unix sockets in a shared location. You’ll notice, too, that again we want to mount a directory containing the configuration and not the configuration itself. This is for the same reason outlined above with keepalived.

Finally, we want to create a docker-stack.yml file containing the deployment information for HAProxy. It should look like this:

version: "3.6"

services:
  frontend:
    image: haproxy:2.0
    deploy:
      mode: global
      update_config:
        parallelism: 1
        delay: 10s
      restart_policy:
        condition: any
      placement:
        constraints:
          - "node.labels.lb==true"
    volumes:
      - type: bind
        source: /srv/haproxy/frontend
        target: /usr/local/etc/haproxy
      - type: bind
        source: /srv/haproxy/sockets
        target: /usr/local/etc/haproxy/sockets
      - type: bind
        source: /srv/haproxy/certs
        target: /usr/local/ssl/certs
        read_only: true
    networks:
      - host

  backend:
    image: haproxy:2.0
    deploy:
      mode: global
      update_config:
        parallelism: 1
        delay: 10s
      restart_policy:
        condition: any
      placement:
        constraints:
          - "node.labels.lb==true"
    volumes:
      - type: bind
        source: /srv/haproxy/backend
        target: /usr/local/etc/haproxy
      - type: bind
        source: /srv/haproxy/sockets
        target: /usr/local/etc/haproxy/sockets
    networks:
      - public_proxy

networks:
  host:
    name: host
    external: true
  public_proxy:
    name: public_proxy
    driver: overlay
    driver_opts:
      encrypted: "true"
    ipam:
      driver: default
      config:
      - subnet: 10.62.0.0/22

We’re bringing together a lot of things here. We are deploying the generic HAProxy 2.0 container (which is LTS as of this writing) and we’re connecting the frontend to the SSL certificates. We’re also creating an encrypted overlay network that should be big enough for HAProxy to connect to all of our containers. We’re also sharing a sockets volume that will let the backend and the frontend communicate. You can deploy this stack file from a controller node by running this command:

docker stack deploy -f docker-stack.yml haproxy

Within a short bit you will have two containers running on each of the load balancers both running HAProxy. You should also start to see IP addresses appear on one of your load balancers as they begin to recognize that HAProxy is online. If you try to connect to your application you’ll get 503 errors and that is great because things are starting to work.

Next Steps

In this part we got haproxy up and running so that you can serve up some encrypted error messages. You might even be able to use HAProxy to serve up some things that aren’t necessarily part of your Swarm. In the next part we will finally set up our registry and move all of our containers to it.

There are still a lot more steps! Follow on to read the rest of the steps.