Categories
Technology

Bootstrapping Docker Swarm Part 2: Setting Up Syslog

This is part of a multi-part series on getting Docker Swarm up and running. You might want to start with the original post called Bootstrapping Docker Swarm.

Configuring Logging

The first container that we’re going to set up will run syslog-ng. We’re going to run this on the infra host as a regular, non-Docker Swarm container. Create a directory for this container and we’re going to put three files in it:

  • Dockerfile
  • docker-compose.yml
  • syslog-ng.conf

The Dockerfile for this container looks like this:

FROM debian:buster-slim

ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update && apt-get install -y --no-install-recommends syslog-ng

# make sure we are healthy
HEALTHCHECK --interval=2m --timeout=3s --start-period=30s CMD /usr/sbin/syslog-ng-ctl stats || exit 1

# move all of the configuration files into place
COPY syslog-ng.conf /etc/syslog-ng/syslog-ng.conf

EXPOSE 514/udp
VOLUME ["/logs"]
ENTRYPOINT ["/usr/sbin/syslog-ng", "-F", "--no-caps"]

The docker-compose.yml file looks like this:

version: "3.2"

services:
  syslog:
    build: .
    restart: always
    network_mode: host
    container_name: syslog
    image: ${IMAGE:-registry.example.com/infra/syslog}:${VERSION?undefined VERSION}
    volumes:
      - type: bind
        source: /srv/syslog
        target: /logs

And the syslog-ng.conf file looks like this:

@version: 3.19

options {
    keep_hostname(yes);
    chain_hostnames(no);
    normalize_hostnames(yes);
    create_dirs(yes);
    owner("root");
    group("root");
    perm(0644);
    use_dns(yes);
    use_fqdn(yes);
    dns_cache(yes);
    stats_freq(0);
};

source s_clients {
    udp();
};

source s_internal {
    internal();
};

destination d_local0 { file("/logs/local0.log"); };
destination d_local1 { file("/logs/local1.log"); };
destination d_local2 { file("/logs/local2.log"); };
destination d_local3 { file("/logs/local3.log"); };
destination d_local4 { file("/logs/local4.log"); };
destination d_local5 { file("/logs/local5.log"); };
destination d_local6 { file("/logs/local6.log"); };
destination d_local7 { file("/logs/local7.log"); };
destination d_internal {file("/logs/syslog-ng.log"); };

filter f_local0 { facility(local0); };
filter f_local1 { facility(local1); };
filter f_local2 { facility(local2); };
filter f_local3 { facility(local3); };
filter f_local4 { facility(local4); };
filter f_local5 { facility(local5); };
filter f_local6 { facility(local6); };
filter f_local7 { facility(local7); };

log { source(s_clients); filter(f_local0); destination(d_local0); };
log { source(s_clients); filter(f_local1); destination(d_local1); };
log { source(s_clients); filter(f_local2); destination(d_local2); };
log { source(s_clients); filter(f_local3); destination(d_local3); };
log { source(s_clients); filter(f_local4); destination(d_local4); };
log { source(s_clients); filter(f_local5); destination(d_local5); };
log { source(s_clients); filter(f_local6); destination(d_local6); };
log { source(s_clients); filter(f_local7); destination(d_local7); };
log { source(s_internal); destination(d_internal); };

These files should all be created on the infra host. We’re going to build the container on that host and deploy it on that host. We can build it like this:

mkdir -p /srv/syslog
VERSION=latest IMAGE=syslog docker-compose build
VERSION=latest IMAGE=syslog docker-compose up -d

Now we have a syslog container running and it is using host networking and listening on UDP port 514 and writing its log files on the infra host to /srv/syslog.

It’s going to be helpful to rotate these logs or they will just write forever to the same location. We could put that into the container itself but that would be making one container do two things so we are not going to do that. Our host should already have logrotate installed and running so we will piggy-back on that and install a file called /etc/logrotate.d/syslog and it will contain this:

/srv/syslog/*.log {
    daily
    rotate 30
    dateext
    missingok
    compress
    delaycompress
    sharedscripts
    create
    postrotate
        docker kill --signal HUP syslog > /dev/null
    endscript
}

Now that syslog is all set up and running you can test it by running this command:

logger -d -n 10.0.0.20 -p local7.info --rfc5424=notq "test"

You should see the word “test” appear /srv/syslog/local7.log.

Now that we have syslog running let’s go back and update all of our hosts to send their Docker logs to this new syslog daemon. To do that, edit /etc/docker/daemon.json and make it now look like this:

{
    "storage-driver": "overlay2",
    "bip": "10.60.1.1/24",
    "log-driver": "syslog",
    "log-opts": {
        "syslog-facility": "local7"
        "syslog-address": "udp://10.0.0.20:514"
    }
}

You’ll need to restart Docker to make this take effect. Now all of your containers will write their logs to one location.

Next Steps

In this part we got syslog installed and configured. It’s not running Swarm mode but it is running in Docker standalone mode and it is accepting syslog messages. You might have even configured Docker to send syslog messages to it. In the next part we will configure certbot and acme-dns so that you can get SSL certificates to protect access to services within your cluster.

There are still a lot more steps! Follow on to read the rest of the steps.