Kubernetes Secrets – Introduction

Kubernetes secrets are essential for protecting passwords, tokens, keys and other sensitive data in Kubernetes environments. But the sensitive information they contain makes them an attractive target for attackers. Once compromised, a single exploit can allow an attacker to gain control of an entire Kubernetes container and cluster infrastructure. This is because, by default, all Kubernetes-run microservices, pods and clusters are freely connected.

Very often, access to secrets is all an attacker needs to gain control of critically sensitive data that secrets are supposed to protect. Conversely, locking containers down completely — blocking all interconnections between containers and pods — would prevent applications running on the clusters from functioning.

The use of a secret is mandatory to contain sensitive data that should not be exposed in the application code. But proper management of secrets is critical to prevent them from serving as a major attack target.

Secrets management is therefore a major component of cloud native security. Proper management depends on a number of cloud security best practices. These include helping to secure the cluster components themselves, as well as the applications running in the clusters and the connections through which data is exchanged.

There are a few things to keep in mind. Whether you are new to Kubernetes security or want to check that you have no gaps in your secrets knowledge, here is a checklist of best practices for secrets management to consider:

A secure place for secrets

A major challenge for secrets management is choosing the right tool for storage and administration. For starters, sensitive information should not be hardcoded in the application or stored as plain text in running containers. Importantly, they  should not be stored in a file on a public repository such as GitHub or GitLab. Encrypted or not, such files can be obtained easily by way of phishing or similar methods, compromising the entire cluster.

Instead, within the Kubernetes ecosystem, it is highly recommended to use a secrets object. Encryption at rest is mandatory, ensuring sensitive information is encrypted before it’s stored in etcd, a distributed key-value store. Additionally, secrets can be managed through cloud provider services or external tools such as Vault. (More about these options below.)

  • Keep secret data out of containers. Secrets should never be hardcoded within an application stored on disk or accessible at the node level. They should be stored and accessible only in memory for the required containers. Once the application uses the sensitive information to complete a process and is no longer running, the secret should be removed altogether from the memory. If ephemeral volumes are used to provide secret data for the application, these should also be removed when the container stops running or is deleted.
  • Add sealed secrets. One challenge for applications deployed via pipelines in Kubernetes involves bootstrapping them securely with passwords stored in secret objects. This can be achieved with so-called SealedSecrets from Bitnami that encrypts your secret data and enables you to store it in a git repository. The data can only be decrypted by the SealedSecret controller running inside the cluster. The developer has access to SealedSecrets through RBAC control at the namespace or resource level.

Once the SealedSecret is deployed, it will appear in the Kubernetes cluster as a normal secret object that is base64 encoded. As such, a pod can access the sensitive information on an as-needed basis. This process is attractive, since encoded data can be stored in a public or private Git repository, hence lending itself well to a fully automated CI/CD process.

  • Restrict third-party access. Third-party tools, applications, platforms and other integrations will often, by default, request direct access to secrets. These requests are often unnecessary, yet add a potential access point for secrets by attackers. If certain access is required in the Kubernetes infrastructure, it is much more secure to configure access to specific namespaces or resources through RBAC. It is also important not to allow untrusted third-party integrations to access certain resources, namespaces or core controllers such as kube-apiserver, as this can lead to a full breach of the cluster.
  • Keep secrets in a single location. It is not unheard of for secrets to be accessible directly in public repositories as mentioned above, such as in YAML files on GitHub or external file storage systems. Secrets distributed and stored using multiple mechanisms can lead to an increased blast radius when data escapes the security perimeters. Proper authorization and RBAC controls, auditing, encryption and monitoring should be imposed to keep these secure.
  • Use encryption. It’s absolutely mandatory to ensure that secrets are encrypted at all times. This includes when they are being stored in a secrets object or when they’re in transit to be accessed by a pod, namespace, etc. Transport Layer Security (TLS) connections should be used for their distribution, such as when updated or accessed by the admin.
  • Detail changes in secrets logs. Any access and changes made to secrets or any sensitive data should be detailed in an audit log. If and when there is an attack, an existing audit log will offer information about when and how secrets were changed or accessed.
  • Use dynamic secrets. Dynamic secrets are continually changed after a specific time period has elapsed, also known as secret rotation. Compromised secrets thus only grant access to attackers for a specified and limited amount of time. The compromised secrets can also help with auditing to determine when a particular secret was exposed in a breach.
  • Use Custom certificate authority (CA). Kubernetes provides a certificates.k8s.io API to provision TLS certificates signed by a Certificate Authority (CA) that you control. The certificate authority itself can be stored in secrets objects to maintain the validity of end-to-end connections.
  • Use the Container storage interface (CSI) driver. Generally available since the Kubernetes v1.13 release, CSI offers an additional secure storage component not only for secrets but for other data. It was designed so that, as a plugin, a protected storage layer outside of the cluster can be added, to be used by services within the cluster.
  • Leverage etcd. etcd is a key value store that admins can access and manage directly in Kubernetes. It was designed to offer a straightforward and simple-to-use API and an automated TLC with optional client certificate identification. It’s also fast: according to the creator’s benchmarks, it can provide 100 writes per second. In addition to secrets, the etcd stores all information about the cluster and facilitates duplicating clusters in a disaster-recovery scenario.
  • Use third-party secrets storage. The major cloud providers offer secrets management services such as AWS Secrets Manager, Azure Key Vault and Google Secret Manager. A popular and widely adopted open source of secrets services is HashiCorp’s Vault. It is designed to provide lifecycle management of secrets including creation, storage, revocation, time-bound rotations and auditing with unified policy enforcement.

Vault also offers key and certificate management features to standardize lifecycle management of keys and certificates, and provides encryption as a service to enable advanced data protection. One of the more interesting features of Vault is that it acts as an identity broker by leveraging multiple trusted identities and includes many identity integrations so that no matter where your application is running, Vault can authenticate them.

When Secrets Are Lost

The unthinkable can and does occur. This is why it is critical for a proper Kubernetes backup and recovery solution to be in place, such as  Kasten K10. A secrets service like Vault can be integrated with it so that even if an attacker gains access to Vault’s secrets object, access can be overridden. Vault can also be used to protect the encryption key used by Kasten K10 to encrypt application data. The master key will still be stored in the Kasten catalog but sent to your Vault installation for encryption and decryption. Access restrictions implemented as policy can allow the organization to restore data, including secrets, that is compromised or even encrypted by a ransomware attack.

The Learning Path

Properly implementing secrets best practices requires hands-on practice. To avoid the risk of practicing how to manage and store secrets on live clusters in a real-world environment, developers can train the requisite muscle memory by working in a sandbox environment that the KubeCampus Kubernetes Learning Site provides for free.

If you are completely new to Kubernetes security, Introduction to Kubernetes Security is a great place to start. Kubernetes Security — Intermediate Skills provides hands-on practice with secrets management, as well as other Kubernetes security topics including authentication, RBAC management, runtime security and pod-security standards.

How secret management relates to storage and disaster recovery is covered in the Storage and Applications in Kubernetes lab.

KubeCampus offers a number of courses covering Kubernetes management to help you build relevant skills in a hands-on way as a developer, a backend engineer or an operations specialist. Once you have completed any of the KubeCampus courses, you can revisit them to use them as a reference to keep your Kubernetes skills sharp. Whether you are a seasoned Kubernetes professional or are a complete beginner, the courses are designed to accommodate your needs.