Part of rebuilding this website was rebuilding the server and reevaluating all of the technologies that I had put together. Previously I had purchased certificates from Comodo and paid $50 for two year certificates and hoped that I guessed what names I wanted in the SAN correctly for the next two years. It was expensive and prohibitive toward innovation. So this time around I decided to use Let’s Encrypt. Since Let’s Encrypt has been around for a few years and the EFF is both a founder and a major backer I feel pretty comfortable using it.

There are a few things that are unusual about Let’s Encrypt if you’re used to using the last generation of certificate authorities. The first is that you’re required to use their API to generate certificates, though the folks behind it provide very well written and supported tools to use that API. The second is that the certificates only last ninety days, so you’re going to want to automate the process of renewing them.

The default tool provided by the EFF and the Let’s Encrypt community for generating certificates is called certbot. It automatically creates private keys and certificate requests for you and sends them to the Let’s Encrypt API to get back your certificate. It will also automatically renew your certificates when it is time to be renewed. The API validates that you have control of the domain name or names for which you are requesting the certificate and then sends back the new or updated certificate. It really could not be easier.

The most common way for certbot to do domain verification is by making an HTTP request to the domain in question and seeing if a special file exists. A lot of guides assume that you’re using Apache or Nginx and that it is serving files from a file system and that certbot can just plop some files on the file system and away you go. Another, less common way to use certbot is to let it run its own web server that serves up the files in question. That less common way is how we will use certbot with HAProxy.

Let’s look at some snippets from our HAProxy configuration file

frontend http-frontend
    bind *:80
    mode http
    log global

    # intercept requests for certbot
    acl letsencrypt-acl path_beg /.well-known/acme-challenge/

    # otherwise redirect everything to https
    redirect scheme https code 301 if !letsencrypt-acl !{ ssl_fc }

    # backend rules are always processed after redirects
    use_backend http-certbot if letsencrypt-acl

    # send all other requests to a backend that redirects to https
    default_backend http-backend

backend http-backend
    mode http
    log global
    redirect scheme https code 301

# this will take challenge requests for lets encrypt and send them
# to certbot which will answer the challenge
backend http-certbot
    mode http
    log global

    # this server only runs when we are renewing a certificate
    server localhost localhost:54321

This snippet listens on port 80, the default port for certbot, and looks for requests to the well known endpoint for Automated Certificate Management Environment challenges. If the request is not for an ACME challenge and it is not encrypted then it will be redirected to https. But if it is for an ACME challenge request then it will go to the http-certbot backend where the certbot will be waiting to serve requests on port 54321.

With this configuration all I need to do is point a host name at my server with an A or AAAA or CNAME record and I can get a certificate for it. It doesn’t matter if Apache is actually serving the domain or not. Once the host name is pointed at my server, I only need to run this command to generate a new certificate:

certbot certonly \
    --http-01-port 54321 --standalone --preferred-challenges http \
    --post-hook /usr/local/bin/letsencrypt-reload-hook \
    -d my-new-name.example.com

This will generate a private key and a certificate request and fire up a small web server that will respond to challenge requests from the API and write a new certificate to /etc/letsencrypt. Perfect. What if I want to renew the certificate? That’s easy, too:

certbot renew --post-hook /usr/local/bin/letsencrypt-reload-hook

Only certificates that are ready to be renewed will actually be renewed by this command. Rather than remember these long commands I actually put them both into shell scripts, like this:

#!/bin/sh

# this fills in the default arguments for creating a new
# certificate. all the caller needs to provide is the "-d"
# argument with a comma separated list of names to put on
# the certificate.
exec certbot certonly \
    --http-01-port 54321 --standalone --preferred-challenges http \
    --post-hook /usr/local/bin/letsencrypt-reload-hook \
    "$@"

You’re probably wondering where this letsencrypt-reload-hook is that I keep referencing. It is the secret sauce to the whole mess that configures HAProxy for us. See, HAProxy only likes it when you give it combined private key and certificate files and certbot does not create those. Additionally, HAProxy (like most servers) requires that you signal it when a certificate has been replaced. So that’s what our reload hook does:

#!/bin/sh

set -e

PATH_TO_LIVE=/etc/letsencrypt/live
PATH_TO_TARGET=/usr/local/ssl/certs
RENEWED_LINEAGE=`ls $PATH_TO_LIVE/`

# make sure we have a place for the certs
mkdir -p $PATH_TO_TARGET
mkdir -p $PATH_TO_TARGET/sites

# for each domain create a concatenated pem file
for DOMAIN in $RENEWED_LINEAGE; do
    if [ -d "$PATH_TO_LIVE/$DOMAIN" ]; then
        echo "assembling certificate $DOMAIN for sites"
        cat "$PATH_TO_LIVE/$DOMAIN/privkey.pem" \
          "$PATH_TO_LIVE/$DOMAIN/fullchain.pem" > "$PATH_TO_TARGET/sites/$DOMAIN.pem"
        chmod 400 "$PATH_TO_TARGET/sites/$DOMAIN.pem"
    fi
done

if [ -e /etc/systemd/system/multi-user.target.wants/haproxy.service ]; then
    systemctl reload haproxy
fi

Why do I write the certificates to /usr/local/ssl/certs/sites? That is the location where I have HAProxy configured to load all of its certificates. So I can drop dozens or hundreds of certificates in that directory and HAProxy will load all of them. To accomplish that I put this into my HAProxy configuration file:

frontend https-frontend
    bind *:443 ssl crt /usr/local/ssl/certs/host.pem crt /usr/local/ssl/certs/sites

With this configuration it will load the host certificate first. This is so that clients that don’t support SNI don’t get a random certificate but instead get the certificate for the host itself. If you do not have a certificate specific to the host then you can remove crt /usr/local/ssl/certs/host.pem from the bind line. After the host certificate every certificate in /usr/local/ssl/certs/sites is loaded and the correct cert will be used depending on where the client says it is looking to connect.

That is all that there is to it. One script, two commands, and a slightly modified HAProxy configuration file. I’m very happy that I’ve started using Let’s Encrypt to do all of my certificate goodness. I’m not a bank or major corporation so I don’t need advanced verification. I just need things encrypted by a valid certificate authority that will make browsers happy and I don’t want to think too hard about it. This does all of that for me.