Firewalls with Docker and Kubernetes

Published September 30, 2021

If you’re seasoned at configuring Linux servers then you know that you should turn on a firewall. This is true even if you’re using a cloud provider that has its own firewall service because multiple layers of protection are recommended for any security practice.

However, if you’ve used Docker on Linux then you know that it completely ignores your firewall and also makes it very hard to make firewall changes without completely breaking Docker until you restart either Docker or your server.

Additionally, Docker and Kubernetes, as of this writing, only really work with iptables and will not work with nftables. So let’s talk about how to get your server running iptables and how to use iptables to control your firewall without messing up Docker or Kubernetes (or containerd). These instructions have been tested on Debian Buster and Debian Bullseye.

First, you need to install the persistent iptables package and roll back to “legacy” iptables. This must happen for both Docker and Kubernetes.

apt-get install --no-install-recommends iptables-persistent
iptables -F
update-alternatives --set iptables /usr/sbin/iptables-legacy
update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy

Once you’ve rolled back to “legacy” iptables we can start looking at how to make our firewall work. The trick that we’re going to do for both IPv4 and IPv6 firewalls is that we’re going to create a separate iptables chain that has all of the rules in it that we want the host to abide by. We’re going to have the host set those on boot and set the default INPUT chain use the new, separate chain. This way we can clear and reload this new, separate chain without inadvertently removing the rules put in place by Docker or Kubernetes. Whenever we want to change the rules in that chain we’re going to only change the rules in that chain and not the rest of the firewall.

Our firewall files will go into /etc/iptables/rules.v4 and /etc/iptables/rules.v6. We’ll start with IPv6 because the IPv6 rules are simple since Docker and Kubernetes leave IPv6 rules alone. In this example we just allow SSH connections to our host.

*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:FILTERS - [0:0]

# allow forwarding connections
-F FORWARD
-A FORWARD -j ACCEPT

# allow all outbound connections
-F OUTPUT
-A OUTPUT -j ACCEPT

# make the INPUT chain use the FILTERS chain
-F INPUT
-A INPUT -j FILTERS

# remove any existing rules from the FILTERS chain
-F FILTERS

# accept all connections over loopback
-A FILTERS -i lo -j ACCEPT

# accept established and related connections
-A FILTERS -m state --state RELATED,ESTABLISHED -j ACCEPT

# drop invalid connections
-A FILTERS -m conntrack --ctstate INVALID -j DROP

# drop connections that come from localhost but not over the loopback
-A FILTERS -s ::1/128 ! -i lo -j DROP

# accept IPv6 ICMP packets (required for correct IPv6 functioning)
-A FILTERS -p ipv6-icmp -j ACCEPT

# allow ssh connections from anywhere
-A FILTERS -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT

# reject everything else
-A FILTERS -j REJECT

COMMIT

For IPv4 there are two tricks:

We want to tie the FILTERS chain to the INPUT chain and the DOCKER-USER chain. That way normal connections and Docker connections go through the same set of rules.
We need to specify the host’s physical interface(s) for the DOCKER-USER chain. If you have multiple interfaces you will need to repeat this rule multiple times for each interface.

The complete Docker compatible IPv4 firewall then looks like below.

*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:FILTERS - [0:0]
:DOCKER-USER - [0:0]

# allow forwarding connections
-F FORWARD
-A FORWARD -j ACCEPT

# allow all outbound connections
-F OUTPUT
-A OUTPUT -j ACCEPT

# make the INPUT chain use the FILTERS chain
-F INPUT
-A INPUT -j FILTERS

# make the DOCKER-USER chain use the FILTERS chain
# do NOT remove the interface labels -- it all breaks if those are missing
-F DOCKER-USER
-A DOCKER-USER -i eth0 -j FILTERS

# basic rules: accept related, deny invalid, deny "localhost" from off host, accept pings
-F FILTERS
-A FILTERS -i lo -j ACCEPT
-A FILTERS -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FILTERS -m conntrack --ctstate INVALID -j DROP
-A FILTERS -s 127.0.0.0/8 ! -i lo -j DROP
-A FILTERS -p icmp -j ACCEPT

# allow SSH from anywhere
-A FILTERS -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT

# reject everything else
-A FILTERS -j REJECT

COMMIT

Now, every time the server starts the rules will be installed automatically and any time you want to change your rules you can just run these commands to apply only changes to the FILTERS chain.

cat /etc/iptables/rules.v4 | egrep "^(\-[A-Z] FILTERS|COMMIT|\*)" | iptables-restore -n
cat /etc/iptables/rules.v6 | egrep "^(\-[A-Z] FILTERS|COMMIT|\*)" | ip6tables-restore -n

I promised that I’d talk about Kubernetes firewalls, too. The setup is largely the same, we just get to ignore the DOCKER-USER chain, like this:

*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:FILTERS - [0:0]

# allow forwarding connections
-F FORWARD
-A FORWARD -j ACCEPT

# allow all outbound connections
-F OUTPUT
-A OUTPUT -j ACCEPT

# make the INPUT chain use the FILTERS chain
-F INPUT
-A INPUT -j FILTERS

# basic rules: accept related, deny invalid, deny "localhost" from off host, accept pings
-F FILTERS
-A FILTERS -i lo -j ACCEPT
-A FILTERS -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FILTERS -m conntrack --ctstate INVALID -j DROP
-A FILTERS -s 127.0.0.0/8 ! -i lo -j DROP
-A FILTERS -p icmp -j ACCEPT

# allow SSH from anywhere
-A FILTERS -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT

# reject everything else
-A FILTERS -j REJECT

COMMIT

Reloading the firewall for Kubernetes is the same set of commands as for Docker. The goal here is to only populate the FILTERS chain and leave the rest of the chains alone.