Handle Kubernetes Secrets the GitOps Way — Part 1

By Animesh Rastogi．Nov 13, 2022

GitOps is a way of implementing Continuous Deployment for cloud native applications. It focuses on a developer-centric experience when operating infrastructure, by using tools developers are already familiar with, including Git and Continuous Deployment tools.

The core idea of GitOps is having a Git repository that always contains declarative descriptions of the infrastructure currently desired in the production environment and an automated process to make the production environment match the described state in the repository. If you want to deploy a new application or update an existing one, you only need to update the repository — the automated process handles everything else. It’s like having cruise control for managing your applications in production.

It is important to note that GitOps is not the next thing after DevOps or even DevOps 2.0.

GitOps is a set of deployment practices while DevOps is a paradigm or better yet, a mindset. Their shared principles make it easier for teams to adopt the GitOps workflow for existing DevOps techniques.

Now that we know what GitOps is, let’s understand some of the practical challenges when implementing it in a Kubernetes Cluster.

Challenges:

Both your application and infrastructure needs access to some sensitive information like DB Credentials, API Keys, etc. In a Kubernetes environment, you typically use Kubernetes Secrets to store these values and then expose these values to your application either using environment variables or mounted volumes. However,

1. Kubernetes Secrets aren’t exactly secure since they are only base64 encoded which can easily be decoded.
2. A key requirement of GitOps is to use git as a source of truth for your application and your infrastructure. This mandates committing these insecure Kubernetes Secrets to your SCM exposing sensitive information across the entire organisation.

Approaches:

There are 2 approaches that can be used to manage secrets in GitOps:

1. Encrypted Secrets where you encrypt the Kubernetes secrets before checking them in SCM which are decrypted by a Kubernetes controller to regular Kubernetes Secrets
2. External Secrets where you store the secret in an external Secret Manager System like GCP Secret Manager, Hashicorp Vault, etc and commit a reference these external secrets which are pulled in by a Kubernetes controller and applied to the cluster as regular Kubernetes Secrets.

In this blog, we will see how to use Approach 1 using the Bitnami Sealed Secrets project.

Solution:

Sealed Secrets is composed of two parts:

1. A cluster-side controller / operator
2. 2. A client-side utility: kubeseal

Encryption is done using kubeseal which generates a SealedSecret Custom Resource that is decrypted and converted to a Kubernetes Secret object by the cluster-side controller

So, there are 3 broad things that needs to be done:

1. Installing Sealed Secrets Controller in the Target Cluster and Installing kubeseal CLI in your local client
2. Downloading Public Certificate that will be used to encrypt the secret in your client and encrypting the secret.
3. Checking SealedSecret object to your SCM repo which is the source for an ArgoCD application.

Installing Controller and kubeseal:

To install kubeseal locally, simply install it from the Github releases page like following:

wget https://github.com/bitnami-labs/sealed-secrets/releases/download/<release-tag>/kubeseal-<version>-linux-amd64.tar.gz
tar -xvzf kubeseal-<version>-linux-amd64.tar.gz kubeseal
sudo install -m 755 kubeseal /usr/local/bin/kubeseal

Installing Controller depends on your Kubernetes deployment methodology. If you already use Helm or Kustomize, then that’s your best best as described here. If you write regular Kubernetes Yaml files, then you can download the controller.yaml file from the releases page here.

Downloading Public Certificate:

In theory, you don’t need to download the public certificate since kubeseal can directly communicate with the cluster if your user has access to the cluster and the correct context is set in your kubeconfig file

However, in an enterprise setup, ideally your cluster should be private. Hence, we shall use the offline sealing approach.

Once you have installed the controller, run the following command on your bastion VM to get the public certificate

kubectl log --tail -1 -f -l name=sealed-secrets-controller -n <namespace>

Copy the certificate that is outputted and paste it locally to a pem file.

Now, let’s encrypt the secret. Run the following command to create the custom resource.

kubeseal <prod-secret.yaml --cert cert.pem -o yaml> prod-sealed-secret.yaml

This will spit out a yaml file with the SealedSecret Custom Resource consisting of the encrypted data.

Now you can safely commit this Sealed Secret resource in your SCM repo. Ensure you don’t commit the raw secret and the pem file.

Once, ArgoCD syncs the state between the cluster and the repo, you can confirm that decryption was done by running the command

kubectl get secret <secret-name> -n <namespace>

You should see a regular Kubernetes Secret object which can be used by your pods normally.

Drawbacks:

While Sealed Secrets are great, there are certainly some drawbacks to this approach. Notable drawbacks are as follows:

1. Key Rotation and then propagating it across your repositories is a big effort for your Infra teams.
2. At the end of the day, the secret object is still in your repository, encrypted or not. Malicious users can get the committer’s information and send social engineering exploits, or gather clues that can help in attacking the target infrastructure.

Conclusion:

Sealed Secrets is a good solution if you are starting your GitOps journey since it is cheap and very easy to start with and understand. However, as your Deployment processes become complex and SecOps processes evolve, you would need a proper Secret Management System.

In the next part of this blog, we’ll dive deep on Approach 2 — Using External Secret Management System where you only store a reference to the Secret in the your repository.

References:

• What is GitOps? Benefits, Principles, and How to Get Started
• GitOps
• Why You Should Avoid Sealed Secrets in Your GitOps Deployment

---

The original article published on Medium.