Secret management with Vault and Kubernetes

Some introductions...

First a little introduction to Kubernetes, which may require a little introduction to Docker. Skip ahead if you are already familiar with these :)

 

Docker

Docker is a containerisation engine, similar in concept to a virtual machine, but more efficient as it allows all containers on a single machine to share the hosts resources, by all making use of the hosts kernel. To take advantage of this applications built with docker will typically have multiple containers running on each host.

To deploy docker applications you first package them up into a Docker image, which contains everything that is needed to run your application with docker, massively reducing configuration management headaches. I definitely recommend reading more about Docker :) Just note it needs to run on a linux machine, which if you are on Mac or Windows means you will need a linux VM first. There are lots of tools to help you get setup though.

 

Kubernetes

In order to deploy and manage large groups of docker containers Google have developed Kubernetes. The ultimate aim is that once your kubernetes cluster is setup you can ask it to deploy one of your docker images and it will choose which host(s) to put it on, it will handle failure of an instance of the application, it has tools for rolling updates, and lots more goodies, all designed to take away the hassle of deploying and managing applications. It is however still pretty new technology and has a way to go to mature.

 

Secret management

One question with Kubernetes, as with many other systems for managing applications, is how to manage the secrets that your applications need. For example certificates for terminating TLS encryption. Or credentials for accessing things from your cloud provider, such as an Amazon S3 bucket. Or database credentials. The list goes on.

The ideal is these secrets are all short-lived, so that if someone manages to compromise them they will expire in short order anyway.

 

Vault

Vault gives you a good set of tools for managing secrets. Through it's API you can configure vault users with policies to allow them access only to certain secrets. It can issue new, short lived, certificates signed by a CA which you have setup in vault. It can issue short lived credentials for AWS. It can store generic secrets of several other types. So it sounds like it solves our secret management worries but...

 

The challenge

The challenge is how to get your secrets from vault into your applications, and because they are short-lived how do you replace them at regular intervals. There are 2 possible ways we considered doing this:

• Write your applications to talk directly to the vault API to request new secrets when needed
• Have a "helper" container that manages secrets and makes them available to your application

The set of tools we're talking about are aimed at the latter option. It has some advantages - you're applications are decoupled from your secret management system, making it easy to change in the future, and you only need to write one helper container, rather than having to maintain libraries to manage vault secrets in all the different languages we use.

 

The solution

There are a few parts to the solution we're looking at:

• A vault configuration tool to setup users and policies in vault
• A "vault sidekick" container that pulls secrets from vault and makes them available to your application

Vaultconf allows you to keep configuration for vault in a version controlled repository as yaml files. It allows you to reconcile your vault server to these configuration files, making sure you know exactly what users and policies are in vault. It also generates strong passwords for each of your vault users and makes these available as Kubernetes secrets. This is important as your vault sidekick will need vault credentials to read secrets with.

Vault sidekick can read vault credentials from a kubernetes secret, and then make your chosen secrets available to your application, automatically getting new credentials before the old ones expire.

I hope to do another post at some point with a practical example in it.