The 2 limits of Google Cloud IAM service

By guillaume blaquiere．Aug 19, 2020

The security is paramount in cloud environments and IAM (Identity and Access Management) service helps on any cloud provider (you can find IAM service on Azure, AWS and GCP for example).

On Google Cloud, this IAM service uses OAuth and OpenID protocols. It allows to authenticate and to authorize the account (user account or service account).
The authorization is performed only for Google Cloud components; you can’t add your custom authorization/permissions for your own app in IAM service.

The authentication part uses OAuth protocols to generate a credential. You can use your own credential (user account, with interactive authentication in a browser) or a technical credential (service account).

Service account credential are automatically loaded on Google Cloud environment (I explain how later). However, you can also generate a service account key file to use this credential anywhere. And it’s an ugly practice especially to keep the secret…secret!

Ugly service account key files

Service account key files are useful in some cases, but they are also ugly when bad used.
First, think about what is it: a file. A simple file.

A file is the most common object in computing. You can copy it, send it by email, commit it into a Git repository. Sometime this repository is public and you receive an email from Google Cloud that informs you about the leak of your secrets file.

In my company, we have had 2 times this kind of issue with bad actors that silently created VMs with bitcoin miner. Hopefully we fixed it quickly and we have limited the cost. Additionally, it was dev projects without more dramatic issues (confidential data exfiltration for example). I will release another article to present how we tackled the problem and increase the security.

---

Additionally, this file can be shared between the developers, more focused on the features’ development than on the keeping safe this secret file. You can also have external developers that work for a while in your company and go elsewhere with the key file still in their computer.

So, if the users aren’t aware of the confidentiality of this key file, you quickly loose the control of it. That leads to the second problem: Google recommend to regularly rotate the service account key file, at least every 90 days.

So, in summary, the service account key files are a nightmare to manage. They are useful in a specific cases, when the Application Default Credential isn’t enough to solve your authentication. In all other cases…

Secrets never stored

Based on this, IAM service allows to never have to store secrets, like service account key file, and to work seamlessly in your local environment and on Google Cloud.

Therefore, the principle is to use the environment context to be authenticated by IAM service. This strategy is called ADC (Application Default Credential) and the Google Cloud client libraries support this authentication mode in several languages. The library retrieves, according with the environment, the credential, and use it as default credential in the application.

In Python, for example, it’s the google-auth library that allows you to do this

import google.auth
credentials, project_id = google.auth.default()

You never mention your environment or an account, you let the library doing its job. It also works for component libraries (like Google Storage): default constructor() or keyword defaultCredential are used in that case.

ADC on local environment

You can use your user credentials, the same that you use in the Google Cloud console.

• To use your user account credential in the gcloud command line, get your credential like this

gcloud auth login

• To use your user credential in your code and use the ADC, configure your local environment with this command

gcloud auth application-default login

In both cases, you will have to go in you browser, to select your Google account, if many. You might have to re-authenticate yourselves, maybe with 2 factors mechanism. And finally authorize the use of your account with gcloud SDK.

Only a refresh token is stored locally, never your authentication credential login/password or secrets.

ADC on Google Cloud Environment

All Google Cloud services have access to Metadata server. This internal server provides informations on the environment, included the service account used for the service. And thus, the Google Cloud client library can detect this sever and get credential through this Metadata server.

Some products allow you to customize the service account that you want to use in the service, such as Cloud Functions and Cloud Run. Others not, it’s a default service account which is used. But, in any case, a service account is loaded and usable.

Limits of ADC

ADC works well when you use your user credential or when you run your code on Google Cloud products.

There is no magic solution, you need a credential file to be authenticated, with a secret (a private key) store in it: the service account key file. The service account key files are mainly designed and useful in this context.

Other ADC limitations

However, even with local user account credentials or service account credentials on Google Cloud environment, there is still 2 main limitations;

1. The inability to add scope to App Engine default service account on App Engine
2. The inability to generate a signed identity token to reach private Cloud Run and Cloud Functions with a local user credential.

For this 2 cases, service account impersonation is preferred (to avoid key file generation), but it’s not always the safest solution.

---

1. App Engine limitation

App Engine has been the first product of Google Cloud and have more than 12 years old! It allows you to deploy a set of (micro)services to serve a web application. However, as old and first product, there is some legacies that limits you in the future.

App Engine has 2 limitations:

1. All services on App Engine has the same default service account, and you can’t customize it (on App Engine or per service). That’s a concern because all your services on App Engine will have the same level of permission, and it breaks the least privilege principle.
2. App Engine default service account can’t be scoped. More exactly, the scope of the credential allow you to reach all Google Cloud services, not more. In my company use case, we are migrating App Maker apps (the service stops in January 2021) to App Engine. The existing App Maker apps use a lot of GSuite documents (Sheet especially). To access to Sheet API, you need to scope your credential with https://www.googleapis.com/auth/spreadsheets scope. And you can’t

The nicest way is to use service account impersonation (and thus to avoid the service account key file). In other words, you won’t use directly the ADC to access to the service, but you will use them to generate a credential on behalf of another service account. With IAM service, you can manage which service account can be impersonated or not.

---

Tradeoffs

1. You use the same ADC (the App Engine default service account) to impersonate all the service accounts that you want. That means, if “service1” in App Engine can impersonate a “service account1” thank to the App Engine default credential, “service1” can also impersonate the “service account2” (initially created for the “service2”). And at the end, it’s like if all services can access to all “impersonate-able” service accounts. And that breaks the least privilege principle.
2. The Cloud Functions default service account is also the App Engine default service account. If you don’t customize your Cloud Functions identity during the deployment, your function has automatically the same permissions as App Engine services.

Additional issue: Python and Java Google Auth library natively include impersonation methods. It’s not the case for the other languages.

Benefits

When you use impersonation on App Engine, you can also use it with your user credential in your local environment. Therefore, your code on App Engine and on your local is the same and the code that you test locally is exactly the same as this one that will be run on App Engine.

Alternative

You can also generate service account key file per service and load them with your code.
It’s useless to store key files inside Secret Manager and to retrieve them, at runtime, with the App Engine default service account credential, because we go back to the #1 tradeoffs. And in this case, impersonation is a far more better solution.

In my company, we sadly use service account key file for App Engine critical services only, not for all.

Tips

If you are deploying on another component than App Engine, like Cloud Functions and Cloud Run, you can use ADC. However, you will have scope issue with your personal user account in your local environment.

To solve this, scope your credential when you create it by defining the scope like this

gcloud auth application-default login \
  --scopes='https://www.googleapis.com/auth/spreadsheets',\
'https://www.googleapis.com/auth/cloud-platform'

2. Private Cloud Functions and Cloud Run limitation

The second issue is when your local code try to call a service on Cloud Functions or on Cloud Run deployed in private mode. This mode implies the caller to present an signed identity token.

To call the privately deployed services with a simple curl, you can do this

curl -H "Authorization: Bearer \
  $(gcloud auth print-identity-token)" https://service.url.run.app

Gcloud SDK has the capability to generate a signed identity token with your user credential. However, the Google Auth client libraries doesn’t implement this feature for user account credential to reach the deployed service directly from your app code (for example a service being developed locally that call another service deployed privately on Google Cloud).

---

Impersonation can work as previously here. But, I don’t like this solution because you have to perform a hook in your code

If "I'm using user account credential"
  then "impersonate a service account"
  else "use ADC"

And thus, your code doesn’t run exactly in the same way in local and in the Cloud. Therefore, you could have issues on Google Cloud that you havent detected locally. It’s not really safe (a bug point-of-view, not at security point-of-view)

Of course, you could impersonate service account in any case, but that increases your app complexity (and thus decreases the maintainability).

The latest option is to use service account key file, only for local development, and with only the role run.invoker to limit the impact in case of leak.

I create an open source project, and an article, on this topic to help the local test, with a service account key file.

---

What to do?

IAM service offers lot of possibility, and also lot of cases to do the wrong things with security.

One solution can work for any use cases: generate a service account key file every time. It’s also the worst and the most dangerous solution.

Security implies tradeoffs and to understand what we want to achieve. It’s not always easy and automatic. There isn’t a unique solution.

The best starting point is to think ADC for all the use cases except for 3 situations:

• Your workload/application runs outside Google Cloud (API, website backend, CI/CD,…)
• You want to customize the service account per service on App Engine
• You want to invoke locally, with your user account, a private Cloud Run or Cloud functions from your code (and not from gcloud CLI)

---

The original article published on Medium.