Cloud Run: Hot reload your Secret Manager secrets

By Guillaume Blaquiere．Dec 21, 2022

Security is paramount for any type of services and environments. Solutions exist to host, manage and serve secrets securely and at scale. On Google Cloud, Secret Manager is the native service dedicated to secret management.

To simplify accessing the secrets, Secret Manager is directly integrated in some Google Cloud services, like Cloud Build, Cloud Functions and Cloud Run.

So, usual secret management requires to change, rotate and update secrets

How to get the latest secret version in Cloud Run instances without latency and without restarting them?

The secret’s best practices

One of the top rules is: “The secrets must be kept secret”. For that, The Secret Manager service ensures the encryption at rest and in transit as well as the access permission through IAM service.

In addition, for long lived secrets (like password or API Keys), it’s recommended to rotate and renew the secrets regularly (about 90 days as mentioned in the Google Cloud documentation). And preferably without any business impact on the application.

My friend Antoine Castex documented a solution implemented at L’Oréal for key rotation on Google Cloud.

Cloud Run Secret integration

Cloud Run supports 2 modes of integration with the Secret Manager service.

• Load the secret in an environment variable
• Load the secret in a file

The limitation of environment variable solution

The most current usage of Secret Manager is to load the secret as an environment variable. It’s simple to use from code:

• Direct access to the OS environment
• No file to open, close, stream to read
• No Byte to String conversion to perform

So, the perfect solution to start!

On Cloud Run deployment, you have to bind an environment variable name with a secret reference, as in the following sample

gcloud run deploy secret-read 
  --set-secrets=mysecret=medium:latest

When you use the environment variable mode, the secret is accessed at the instance startup. At that time, the execution environment is set up with the correct standard and the custom environment variables.

Because of that, the secret is read only once and never again during all the instance lifecycle.
And so, even if the secret changes, the current running instances won’t be noticed and you can’t use the new secret. Only the new instance can.
Similarly, if you delete the secret, the current instances are not aware of that deletion.

You have 2 possible solutions to reload the latest version of the secret:

• Wait for the older instances to stop automatically.
  Cloud Run does not offer the capacity to stop the running instances. The instance stops automatically 15 minutes after the latest request processed (idle mode). If you have sustained traffic, it could take hours, or days!!
• Redeploy a new revision of your Cloud Run service.
  The solution is complex to automate on secret update, in addition to being partially ineffective.
  Of course, the new requests will be served by the newly deployed revision (with the new secret version loaded), but the existing instances continue to serve “before-deployment” traffic (and can take up to 1h (max Cloud Run timeout), especially when you are serve bi-directional streaming or websocket solution)

The file mount solution

Mounting a secret as a file is not so natural at the beginning even if Kubernetes made this way popular through the ConfigMap.

With Cloud Run, you have also this capacity, here by command line

gcloud run deploy secret-read 
  --set-secrets=/secret/mysecret=medium:latest

As you can see, the difference with environment variable mode is the prefix of the name that start with /

This time, the secret is mounted as a file but it’s not a “real file”. It’s more a proxy that catches the secret access request and performs an API call to the Secret Manager service on the fly.

Because of that, the secret is accessed and read every time you read the file in your code.
And so, the up-to-date secret version is accessed!

This solution WORKS ONLY if your Cloud Run instances mount the latest version of your secret. If you set a static version, it won’t work because the secret’s versions are immutable on Secret Manager. You can only add/delete versions and the pseudo-version latest always references the newest version.

The right feature for the right use case

Cloud Run offers different integration modes of the Secret Manager service. One is easier and more common to use, and has the benefit to never change during the instance runtime.
The other one is more versatile and adapts itself to the latest secret version.

However, keep in mind that using the latest version of a secret in a production environment must be well documented and assumed.
Using a defined version allows you to consistently deploy and rollback to a certain point of time.

No solution is perfect, but you have all the options!! Pick the right one and build awesome, and secure, things!!

Try it out yourself

If you want to have a try, there is few steps

Create a secret

echo -n "hello" | gcloud secrets create medium --data-file=-

Grant the Cloud Run default service account (if you use default service account for your deployment, even if a customer-managed service account is preferred, it’s enough for tests)

gcloud secrets add-iam-policy-binding medium   
--member=serviceAccount:<PROJECT NUMBER>-compute@developer.gserviceaccount.com   
--role=roles/secretmanager.secretAccessor

Replace the PROJECT NUMBER by your own project number

Then, you can use that piece of code in Go

package main

import (
 "fmt"
 "io/ioutil"
 "net/http"
 "os"
)

func main() {
 http.HandleFunc("/", readSecret)
 http.ListenAndServe(":8080", nil)
}

func readSecret(w http.ResponseWriter, r *http.Request) {
 secFile, _ := ioutil.ReadFile("/secret/mysecret")
 secEnvVar := os.Getenv("mysecret")
 fmt.Fprintf(w, "the file secret value is %s.nThe env var secret value is %s", string(secFile), secEnvVar)
}

And deploy it on Cloud Run

gcloud run deploy secret-read 
  --platform=managed 
  --region=us-central1 
  --allow-unauthenticated 
  --source=. 
  --set-secrets=/secret/mysecret=medium:latest,mysecret=medium:latest

That command builds the container and deploys it at the same time. Note that the service is not private (all users can access it) and you have to use the latest version of the secret to make it working.

Click on the provided link and you can see that the file and the environment variable have the same value.

Now, add a new version to your secret.

echo -n "bye" | gcloud secrets versions add medium --data-file=-

And reload the Cloud Run service page.
You have to do that in the 15 minutes following the previous page load. If you take more time, the instance is offloaded and a new one is run with the new secret load in the environment variable.

This time, you can see that only the secret read through file has the new secret value, not the environment variable one.

The original article published on Medium.