There are a bunch of ways to manage secrets in Kubernetes and I surely do not have an exhaustive list of those ways. The most direct way to manage secrets is, of course, just using kubectl:

kubectl create secret generic db-user-pass \
  --from-literal=username=devuser \
  --from-literal=password='S!B\*d$zDsb='

There are obvious drawbacks to this, beyond just the fact that it puts the secret into your command line history. It also makes it really hard to track secrets that you’ve created and manage them after they’ve been created.

So you could put the secret into a YAML file:

---
apiVersion: v1
kind: Secret
metadata:
  name: db-user-pass
  creationTimestamp: null
data:
  username: ZGV2dXNlcg==
  password: UyFCXCpkJHpEc2I9

There are again some obvious drawbacks to this. You have to keep the file around and the data needs to be manually encoded with base64. This might make sense if you use some sort of GitOps but now you’re storing the password in source control and that’s unacceptable.

There exist some very clever ways to handle all of this. For example, HashiCorp Vault gets a lot of good reviews and most cloud providers have something for managing secrets. But maybe you’re just running a real small shop with Kubernetes and you don’t want to go through the hassle of setting up those clever ways. And maybe you’re already using Ansible to manage some other part of your infrastructure. So why not manage your Kubernetes secrets through Ansible using Ansible Vault? That’s what I did, and here’s how you can do it, too.

It works like this:

1. Store your Kubernetes secrets in your Ansible code repository with Ansible Vault.
2. Deploy your Kubernetes secrets somewhere that has write access to your Kubernetes cluster. For example, I deploy them to the controller nodes in my cluster.
3. Whenever the secrets file changes, apply the new copy to your cluster to change the passwords.

The only three drawbacks that I have with this setup are:

1. When I am no longer using a secret I have to manually remove secrets from the cluster.
2. Since I use Flux to do GitOps, all of my Kubernetes configurations are in one place and all of my Ansible configurations in are another place. This, unfortunately, separates my Kubernetes configuration from my Kubernetes secrets which is not ideal, so I have to keep both source repositories in mind when making changes.
3. The secrets are encrypted in such a way that it’s difficult to search or audit them. It is possible for you to change this – by not encrypting the entire file but rather only encrypt the sensitive portions – but I think that this is more effort than it is worth.

With that in mind, if you’re still on board then let’s get started.

First, you need to have Ansible Vault configured. If you do not have it configured then here is a really quick primer on getting it going.

1. Choose a vault-id. This is like a username. You might choose “admin” or something similar.
2. Choose a vault password. You can either remember what this password is and type it every time you run Ansible or you can put it into a text file and tell Ansible to use that text file, just as long as you never add that text file to your repository. For example, I keep my vault password in a file in my repository called .vault-password and I have .vault-password into my .gitignore file so it never gets committed.
3. Create the file that will contain your passwords using touch. We’ll encrypt the file in a moment. For example: touch mysecrets.yaml.
4. Encrypt your newly created file using ansible-vault. For example, if the vault-id that you chose in step 1 is “admin” then you would run ansible-vault encrypt --vault-id admin@.vault-password mysecrets.yaml.
5. You can use the view or edit options instead of encrypt to review or make modifications to your encrypted files. You can use rekey if you need to change your vault password.

Now that you’ve got the gist of Ansible Vault, we are going to create a new Ansible role to manage our Kubernetes secrets. Call this new role something like “kube-secrets” or whatever. We need three files in this role:

roles/kube-secrets/tasks/main.yaml
roles/kube-secrets/templates/secrets.yaml.tpl
roles/kube-secrets/vars/main.yaml

The tasks file is pretty trivial. It copies the secrets file over to whatever host you have set up and then it applies the secrets file to the cluster if it has changed.

- name: check for kubectl
  stat:
    path: /usr/local/bin/kubectl
  register: kubectl

- name: copy kubernetes secrets
  template:
    src: secrets.yaml.tpl
    dest: /etc/kubernetes/secrets.yaml
    owner: root
    group: root
    mode: 0600
  register: k8s_secrets
  when: kubectl.stat.exists

- name: apply secrets
  command: kubectl apply -f /etc/kubernetes/secrets.yaml
  when: k8s_secrets.changed
  ignore_errors: yes

Next is the template file. What we’re basically doing with all of this is writing YAML for Kubernetes to ingest. This template generates some YAML. It only supports “generic”/“Opaque” secrets and Docker registry secrets. It could easily be extended to support other secret types were one to have the need for those.

{% for secret in secrets %}

---

apiVersion: v1
kind: Secret
metadata:
  name: {{ secret.name }}
  namespace: {{ secret.namespace }}
  creationTimestamp: null
{% if secret.type == "generic" %}
{# nothing to do #}
{% elif secret.type == "docker-registry" %}
type: kubernetes.io/dockerconfigjson
{% endif %}
data:
  {% if secret.type == "generic" -%}
  {% for element in secret.data -%}
  {{ element.key }}: {{ element.value | b64encode }}
  {% endfor -%}
  {% elif secret.type == "docker-registry" -%}
  {% set auth = secret.data.username ~ ":" ~ secret.data.password -%}
  {% set json = '{"auths":{' ~ secret.data.hostname|tojson ~ ':{"username":' ~ secret.data.username|tojson ~ ',"password":' ~ secret.data.password|tojson ~ ',"email":' ~ secret.data.email|tojson ~ ',"auth":' ~ auth|b64encode|tojson ~ '}}}' -%}
  .dockerconfigjson: {{ json|b64encode }}
  {% endif %}

{% endfor %}

Finally, we’ll make the vars file. This is the encrypted file where your secrets live. The syntax is pretty trivial:

secrets:
  - namespace: default
    name: db-user-pass
    type: generic
    data:
      - key: username
        value: devuser
      - key: password
        value: "S!B\*d$zDsb="
  - namespace: default
    name: my-docker-creds
    type: docker-registry
    data:
      hostname: docker.io
      username: myusername
      password: s00p3rs3kr3t!
      email: me@example.com

Remember to encrypt this file with Ansible Vault before committing it to your repository!

With all of that configured, if you run this playbook it will create a secrets file on your host and attempt to apply it to your cluster whenever a password changes. And now you’re managing your Kubernetes secrets from Ansible.