Vaults background
Kubernetes applications use many types of sensitive data to work correctly, including:
- Database usernames and passwords
- Middleware authentication data
- Cloud platform service accounts and tokens
- API authorization tokens and keys
- Asymmetric certificates
- Encryption keys and passphrases
Whether one is implementing a microservices or 3-tiered architecture that employs sensitive data, good design demands that we separate software code from its configuration. One of the critical elements in this approach is trusted credentials or secrets.
Simply saving credentials or secrets in a configuration file is not a reasonable solution. This approach is neither scalable nor secure. What’s needed is a solution that’s safe, robust, reliable, scalable, and accessible to programmatically store and retrieve those types of configurations. For that purpose, we utilize vaults.
What is a secrets vault?
A vault is a secrets management platform capable of handling numerous secrets for multiple applications, safely and simultaneously. Think about password managers such as 1Password, which we use on our laptops for generating, storing, and retrieving personal passwords. Vaults do the same (and much more) for applications, including Kubernetes apps.
Why are vaults vital for Kubernetes?
Although Kubernetes uses secrets to store credentials, a secret doesn’t have the same level of security as a vault. Also, if your application requires sharing a password among different Kubernetes clusters, that becomes a problem. Using a vault for Kubernetes apps is not just good practice; it’s more secure.
Another reason to use a vault inside Kubernetes is to replace the native service account token generator with one embedded in a vault. Doing so will provide higher levels of granular control for the token time-to-live and leasing, which are well-designed security measurements.
Understanding the HashiCorp Vault
HashiCorp Vault is an excellent example of a vault for managing application secrets. Besides being offered via both open source and enterprise licenses, HashiCorp Vault has additional capabilities that make the lives of software engineers and Kubernetes administrators easier. Below are a few of its core characteristics:
- Comprehensive API: Like Kubernetes, we execute all Vault operations through a call to an API endpoint. Even the CLI is simply a formatted call to the API server.
- Secure secrets storage: Vault protects all data at rest with three layers of strong encryption. It’s virtually impossible to break it through a brute force attack.
- Modular architecture: Vault has multiple components with specific functions that increase its performance and resiliency.
- Flexible administration and operation: We can work with Vault using direct calls to the API endpoint, through its CLI, through an agent, or by using any of the supported programming languages SDKs.
How does HashiCorp Vault work?
In general terms, Vault utilizes client tokens to authenticate an entity and policies based on paths (like Kubernetes Roles and ClusterRoles), and rules to authorize the requested operation. As depicted in their documentation, the core workflow is:
- Authenticate: A client entity (person or software) submits its identity using a token.
- Validate: Vault validates the client identity using a proper authentication method.
- Authorize: Vault matches the client’s identity and intended operation with the associated security policy.
- Access: Vault grants client access to secrets, keys, or passwords based on the rules.
Vault has three main components to work as described above:
- Vault API: The single interface to all operations, including the administrative ones. The CLI and all SDKs wrap commands in HTTP calls to the API.
- Secret engines: Secret engines store, generate, or encrypt data upon request. They can even manage secrets on external tools such as Kubernetes and LDAP.
- Authentication methods: These methods perform authentication and assign identity and policies to an entity (user, group, or service). They also delegate the authentication decision to configure external authenticators.
Installing Vault CLI
The Vault CLI is a unique binary file you install on your local machine to administer a Vault server. To install it, please follow the instructions from this page: https://developer.hashicorp.com/vault/downloads
If you have a MacBook, you can download Vault here: https://releases.hashicorp.com/vault/1.12.2/vault_1.12.2_darwin_amd64.zip
Then, you simply unzip the package and install it on a directory such as /usr/local/bin:
$ cd ~/Downloads
$ unzip vault_1.12.2_darwin_amd64.zip $ sudo install vault /usr/local/bin Creating and initializing your first Vault
To install the Vault server, you need a Kubernetes cluster and the kubectl set. Also, you can use a helm chart to deploy the Vault server the easiest way.
Add the charts from HashiCorp to your help repositories list:
$ helm repo add hashicorp https://helm.releases.hashicorp.com Then, update the helm repositories:
$ helm repo update Finally, install the Vault server in the Kubernetes cluster:
$ helm install vault hashicorp/vault \ --set='server.ha.enabled=true' \ --set='server.ha.raft.enabled=true' Check if the Vault pods are running:
$ kubectl get pods Sealing and unsealing your Vault
Now that Vault is running on your Kubernetes clusters, you need to initialize it. Since the vault-0 pod starts as the leader for the 3-instances high availability Vault cluster, we are going to execute a command init there:
$ kubectl exec vault-0 -- vault operator init \ -key-shares=1 \ -key-threshold=1 \ -format=json > cluster-keys.json After initializing a Vault server, it creates the initial internal data structures, the root token, the keys to unseal it, and seals itself. Those keys are irreplaceable; losing them means losing the entire Vault server. When you restart a Vault server, it returns to a sealed state, so be careful with the contents of the cluster-keys.json file.
The next step is to unseal the Vault server. First, save the unseal key to an environmental variable:
$ VAULT_UNSEAL_KEY=$(jq -r ".unseal_keys_b64[]" cluster-keys.json) Use the key to unseal the Vault server using its leader:
kubectl exec vault-0 -- vault operator unseal $VAULT_UNSEAL_KEY Now, make the other two Vault servers part of a Vault cluster:
$ kubectl exec -ti vault-1 -- vault operator raft join http://vault-0.vault-internal:8200 $ kubectl exec -ti vault-2 -- vault operator raft join http://vault-0.vault-internal:8200 Unseal them as well, using the same key:
$ kubectl exec -ti vault-1-- vault operator unseal$VAULT_UNSEAL_KEY
$ kubectl exec -ti vault-2-- vault operator unseal$VAULT_UNSEAL_KEY
Now you have a 3-members Vault cluster available!
Enabling a KV secrets engine and using it
First, connect to the Vault server from your machine through a port forwarding command:
$ kubectl port-forward svc/vault-internal 8200:8200 Export the Vault server address:
$ export VAULT_ADDR=https://localhost:8200 Check if port-forwarding is working:
$ vault status Next, log in with the root token from the initialization:
$ vault login Then, enable the KV secret engine:
$ vault secrets enable -path=secret kv-v2 Store a credential in the Vault server:
$ vault kv put secret/database/config username="rod.anami" password="s3cret!" Verify if you can retrieve it:
$ vault kv get secret/database/config Congratulations! Your Vault is ready to be used!
For more information about Vault, please visit the References section.
Next steps
If you are completely new to Kubernetes security, Introduction to Kubernetes Security is a great place to start. Kubernetes Security — Intermediate Skills Lab provides hands-on practice with secrets management, as well as other Kubernetes security topics including authentication, RBAC management, runtime security and pod-security standards.
In case you want to learn how secret management relates to storage and disaster recovery, these topics are covered in the Storage and Applications in Kubernetes lab.
References
https://developer.hashicorp.com/vault/docs/what-is-vault
https://developer.hashicorp.com/vault/docs/commands
https://developer.hashicorp.com/vault/tutorials/kubernetes/kubernetes-minikube-raft
https://www.npmjs.com/package/hashi-vault-js
