Vaultconf - managing vault logins for your kubernetes applications

In my last post I gave an overview of how you can use vault and kubernetes together. In this post I want to show how to use vaultconf to configure credentials for your kubernetes applications so they can read secrets from vault.

 

Before you start you will need:

• Vault v0.3
• A kubernetes cluster, with local configuration allowing you to talk to the API
• Docker

 

Configuration documents

vaultconf is designed to allow you to reconcile users and policies between configuration files and vault. Lets look at how to set these up:

 

Policy configuration

Policies set out rules for what can be accessed in vault. Policies should all be contained in a policies folder, whose structure should look like this (replacing the names with whatever you like):

• policies
• mynamespace1
• policy1.yaml
• policy2.yaml
• mynamespace2
• policy3.yaml

Please see the vaultconf test/resources folder for examples of vault policies in yaml.

 

User configuration

Users should be contained in a users.yaml file. It defines which users you want in which namespaces, and what policies those users should have. Again you will find an example in the github repo.

 

Vault server

If you haven't got a vault server setup you can start one easily by running:

$ vault server -dev

 

Vault setup

Before you are able to use vaultconf you will first need some things setup in vault:

• Set your VAULT_ADDR and authenticate yourself. If using the dev vault server simply do:
• $ export VAULT_ADDR=http://127.0.0.1:8200
• Enable the userpass auth backend:
• $ vault auth-enable userpass
• Create a user for yourself with root access:
• $ vault write auth/userpass/users/myusername password=mypassword policies=root

 

Kubernetes setup

Before using vaultconf ensure your kubernetes context is set to use the cluster you want the username and password secrets adding to. You will also need to ensure any namespaces are already created within this cluster.

 

Creating policies

The following command will add your policies to vault. Note the --net=host is only needed if you're connecting to a vault server running on your local machine.

$ docker run --net=host -v test/resources/policies:/policies quay.io/timgent/vaultconf:v0.1 policies -c /policies -u myusername -p mypassword -a http://localhost:8200

You should now be able to see your policies in vault:

$ vault policies

dev_myproject_reader
dev_myproject_writer
uat_anotherproject_apolicy
uat_myproject_reader
uat_myproject_writer
root

 

Creating users

IMPORTANT NOTE!

vaultconf will add kubernetes secrets for the vault usernames and passwords on whichever kubernetes cluster your .kube/config file is currently set to use.

 

$ docker run --net=host -v ~/.kube:/root/.kube -v ~/WORK/vaultconf/test/resources/users:/users quay.io/timgent/vaultconf:v0.1 users -c /users/users.yaml -u myusername -p mypassword -a http://localhost:8200

You should now be able to see the secrets in kubernetes:

kubectl --namespace=dev-myproject get secrets
NAME            TYPE      DATA
mrread-vault    Opaque    1
mrwrite-vault   Opaque    1

Using the things you've created

Your applications can now mount in these secrets to gain their credentials for accessing vault. The file will look something like:

{"username":"myNamespace_testUser","password":"testPassword","method":"userpass"}

I hope to follow up with an example of how the vault-sidekick container can then use these credentials to talk to vault and make secrets available to your applications.

Secret management with Vault and Kubernetes

Some introductions...

First a little introduction to Kubernetes, which may require a little introduction to Docker. Skip ahead if you are already familiar with these :)

 

Docker

Docker is a containerisation engine, similar in concept to a virtual machine, but more efficient as it allows all containers on a single machine to share the hosts resources, by all making use of the hosts kernel. To take advantage of this applications built with docker will typically have multiple containers running on each host.

To deploy docker applications you first package them up into a Docker image, which contains everything that is needed to run your application with docker, massively reducing configuration management headaches. I definitely recommend reading more about Docker :) Just note it needs to run on a linux machine, which if you are on Mac or Windows means you will need a linux VM first. There are lots of tools to help you get setup though.

 

Kubernetes

In order to deploy and manage large groups of docker containers Google have developed Kubernetes. The ultimate aim is that once your kubernetes cluster is setup you can ask it to deploy one of your docker images and it will choose which host(s) to put it on, it will handle failure of an instance of the application, it has tools for rolling updates, and lots more goodies, all designed to take away the hassle of deploying and managing applications. It is however still pretty new technology and has a way to go to mature.

 

Secret management

One question with Kubernetes, as with many other systems for managing applications, is how to manage the secrets that your applications need. For example certificates for terminating TLS encryption. Or credentials for accessing things from your cloud provider, such as an Amazon S3 bucket. Or database credentials. The list goes on.

The ideal is these secrets are all short-lived, so that if someone manages to compromise them they will expire in short order anyway.

 

Vault

Vault gives you a good set of tools for managing secrets. Through it's API you can configure vault users with policies to allow them access only to certain secrets. It can issue new, short lived, certificates signed by a CA which you have setup in vault. It can issue short lived credentials for AWS. It can store generic secrets of several other types. So it sounds like it solves our secret management worries but...

 

The challenge

The challenge is how to get your secrets from vault into your applications, and because they are short-lived how do you replace them at regular intervals. There are 2 possible ways we considered doing this:

• Write your applications to talk directly to the vault API to request new secrets when needed
• Have a "helper" container that manages secrets and makes them available to your application

The set of tools we're talking about are aimed at the latter option. It has some advantages - you're applications are decoupled from your secret management system, making it easy to change in the future, and you only need to write one helper container, rather than having to maintain libraries to manage vault secrets in all the different languages we use.

 

The solution

There are a few parts to the solution we're looking at:

• A vault configuration tool to setup users and policies in vault
• A "vault sidekick" container that pulls secrets from vault and makes them available to your application

Vaultconf allows you to keep configuration for vault in a version controlled repository as yaml files. It allows you to reconcile your vault server to these configuration files, making sure you know exactly what users and policies are in vault. It also generates strong passwords for each of your vault users and makes these available as Kubernetes secrets. This is important as your vault sidekick will need vault credentials to read secrets with.

Vault sidekick can read vault credentials from a kubernetes secret, and then make your chosen secrets available to your application, automatically getting new credentials before the old ones expire.

I hope to do another post at some point with a practical example in it.