Billing Alert with Cloud Monitoring Notification Channel

By guillaume blaquiere．May 21, 2021

The cloud cost is one of the most scary aspect: you pay for what you use! To prevent any overcost, one of best practices is to estimate roughly the cost of a project and to set a budget alert on it.

On Google Cloud, you can achieve that on the billing page, in the Budget & Alerts section.

You need to have the Billing Admin role to access it. In an organization, there are different user profiles and all can’t be Billing Admin:

• The cloud billing can be a sensitive information and the Billing Admin role allows a full view on all the billing of all projects.
• Billing Admin can delete/close a billing account and therefore shut down all the attached projects
• The access to the payment method can be confidential/sensitive.

Billing Account User role issue

In a large company with only one billing account and dozens of subsidiaries, it’s mandatory to have them be autonomous for project creation. And, because internal rebill is performed periodically to subsidiaries, the capacity for them to follow and to be alerted on cloud resource consumption is paramount.

The user in the subsidiaries have the Billing Account User role to enable attaching the billing account to their newly created project. But, because they don’t have the Billing Account Admin role, they are unable to create a budget alert.

The wrong solution

One of the first and wrong explored solution was to ask the subsidiaries to send to the Billing Account admin team the budget alert parameter to set on their projects.

Besides the overhead for this team (which could be fixed with a development to automate the creation), the alerts, when raised, are sent to the Billing Account admin users, and not directly to the project managers in the subsidiaries.
Therefore an additional overhead to forward the budget alert, to the right team, with all the latency, mistakes and forgetting that it incurs.

Recipient customization

A feature, released last year (May 2020), allows to change the default notification email recipient (the Billing Account admin users) to custom emails defined in the Cloud Monitoring Notification channels.

The feature solves the previous issue: the budget alert can be sent to the correct recipients instead of the wrong Billing Account Admin users. To configure that automatically and at scale, we designed this application.

The principle is to create a message (sync with direct HTTP call or async with a Pub/Sub message) with basic information:

• ProjectID on which to set the billing alert
• The monthly budget to set
• The list of emails (users or groups) to notify.

It’s arbitrary simple, you can choose to build something more customizable and complex.

The permissions required by the subsidiaries to trigger Pub/Sub (publish a message into Pub/Sub topic) or directly the Cloud Run (Cloud Run Invoker) are managed by IAM and aren’t linked to the billing account; which is safer.

---

When the parameters are received, the process is split in 2 parts:

1. Cloud Monitoring Notification Channel management
2. Billing alert management

Cloud Monitoring Notification Channel API

This API proposes to programmatically manage the notification channels. In this context, the process is the following

From the list of emails (user emails or groups) to notify from the parameters in entry

• Check if the Notification channel exists
• If it exists, get the name reference
• If not, create it, and get the name reference after the creation.

The requester needs to have, at least, the Monitoring NotificationChannel Editor role

https://monitoring.googleapis.com/v3/projects/gdglyon-cloudrun/notificationChannelsBy API , you can get the  notification channels like thatcurl -H "Authorization: Bearer $(gcloud auth print-access-token)" \
https://monitoring.googleapis.com/v3/projects/<PROJECT_ID>/notificationChannels

And to create a notification channel, you can send this request

curl -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  -H "content-type: application/json" \
  -X POST -d @notification-channel.json \ https://monitoring.googleapis.com/v3/projects/<PROJECT_ID>/notificationChannels

along with this notification-channel.jsondata structure

{
  "type": "email",
  "displayName": "<Human Readable Name>",
  "labels": {
    "email_address": "<EMAIL_ADDRESS>"
  }
}

In the response to the API call, you have the name of the created channel. Get it and keep it. We will use it to create the budget alert.

You can check the GO implementation here with the Google API Libraries

Billing Alerts with Budget API

The budget API only allows to manage the budget on the billing account. Billing Account Admin role is required to use this API.

In this context the process in the following

• If no alert exists (based on our own naming convention), create the budget alert with the monthly budget, the projectID and the notification channels
• Else, update the existing budget alert with monthly budget and notification channels.

To create a budget alert by API you can perform this call
Note: you can’t use your user account to invoke this API. A service account is required for that.

curl -H "Authorization: Bearer $(gcloud auth print-access-token)" \
https://billingbudgets.googleapis.com/v1/billingAccounts/<BILLING_ACCOUNT>/budgets

And this one to get an existing budget alerts

curl -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  -H "content-type: application/json" \
  -X POST -d @budget.json \
https://billingbudgets.googleapis.com/v1/billingAccounts/<BILLING_ACCOUNT>/budgets

With this budget.json data structure

{
  "displayName": "<Human Readable Name (naming convention?)>",
  "budgetFilter": {
    "projects": [
      "<PROJECT_ID>"
    ],
    "calendarPeriod": "MONTH"
  },
  "amount": {
    "specifiedAmount": {
      "currencyCode": "EUR",
      "units": 10,
      "nanos": 10000000
    }
  },
  "thresholdRules": [
    {
      "thresholdPercent": 0.5
    },
    {
      "thresholdPercent": 0.9
    },
    {
      "thresholdPercent": 1.0
    }
  ],
  "notificationsRule": {
    "monitoringNotificationChannels": [
      "<Notification Channel Name got previously>"
    ],
    "disableDefaultIamRecipients": true
  }
}

The update is more tricky because you need to provide an update mask

curl -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  -H "content-type: application/json" \
  -X PATCH -d @budget-update.json \
https://billingbudgets.googleapis.com/v1/billingAccounts/<BILLING_ACCOUNT>/budgets/<BUDGET_ID>?updateMask=amount.specified_amount,notifications_rule

With this budget-update.json data structure

{
  "amount": {
    "specifiedAmount": {
      "currencyCode": "EUR",
      "units": 80,
      "nanos": 10000000
    }
  },
  "notificationsRule": {
    "monitoringNotificationChannels": [
      "<Notification Channel Name got previously>"
    ]
  }
}

The GO implementation is in this file.

Budget alerting for everyone

Thanks to the Notification Channel feature in the Budget API, it’s now possible to notify anyone on budget alerts and without too much complexity and overhead.

Of course, the solution isn’t perfect because the users do not have a UI to configure the alerts and the capacity to view/browse the existing ones.

However it is better than having full access to the Billing Account permission, or no budget alerts and therefore being blind on what the projects cost!

---

You’ll find the full open source project here. Follow the readme file to deploy it manually or use terraform. Update and adapt the project as you wish!
Open issue or feature request if you want. I will be happy to help.

---

The original article published on Medium.