Examples of automated cost control responses

Example reference architecture

Figure 1: Illustrates an example of using budget alerts to automate cost control responses using Pub/Sub for programmatic notifications and Cloud Run functions to automate a response.

If you're cost conscious and need to control your environment relative to your budget, then you can use programmatic budget notifications to automate your cost control response based on the budget notification.

Budget notifications use Pub/Sub topics to provide a real-time status of the Cloud Billing budget, using the scalability, flexibility, and reliability of enterprise message-oriented middleware for the cloud.

This document has examples and step-by-step instructions on how to use budget notifications with Cloud Run functions to automate cost management.

Note: Budget notifications are sent to the Pub/Sub topic multiple times per day with the current status of your budget and thresholds. Notifications continue to be sent even after all thresholds are exceeded.

Set up budget notifications

The first step is to enable a Pub/Sub topic for your budget. This is described in detail at Manage programmatic budget alert notifications.

After enabling budget notifications, note the following:

• Pub/Sub Topic: This is the configured notifications endpoint for the budget.
• Budget ID: This is a unique ID for your budget that's included in all notifications. You can locate the budget's ID in your budget under Manage notifications. The ID is displayed after you select Connect a Pub/Sub topic to this budget.

Listen to your notifications

The next step is to listen to your notifications by subscribing to your Pub/Sub topic. If you don't have a subscriber, then Pub/Sub will drop published messages and you can't retrieve them later.

Although there are many ways you can subscribe to your topic, these examples use Cloud Run function triggers.

Create a Cloud Run function

To create a new Cloud Run function:

1. In the Google Cloud console, go to the Cloud Run functions page.

   Go to Cloud Run functions page

2. Click CREATE FUNCTION and give the function a name that's meaningful to your budget.

3. Under Trigger, select Pub/Sub topic.

4. Select the topic that you configured on your budget.

5. Provide source code and dependencies for the function to run.

6. Ensure you set the Function to execute to the correct function name.

Describe your Cloud Run function

To tell your Cloud Run function what you want it to do with the notification, you can either write code using the inline editor or you can upload a file. For details about the notifications your code will receive, see Notification Format.

For example, a function might log received Pub/Sub notifications, attributes, and data when triggered by a budget notification. To learn more, see Pub/Sub Triggers.

View your Cloud Run function events

After you save the Cloud Run function, you can click VIEW LOGS to view your logged budget notifications. This shows the logs from your function invocations.

Test your Cloud Run function

Notifications are sent to Pub/Sub and subscribers receive the messages. To test a sample notification to make sure your function is working as expected, publish a message in Pub/Sub using this object as the message body:

{
    "budgetDisplayName": "name-of-budget",
    "alertThresholdExceeded": 1.0,
    "costAmount": 100.01,
    "costIntervalStart": "2019-01-01T00:00:00Z",
    "budgetAmount": 100.00,
    "budgetAmountType": "SPECIFIED_AMOUNT",
    "currencyCode": "USD"
}

You can also add message attributes such as the billing account ID. See the full notification format documentation for more information.

Send notifications to Slack

Email isn't always the best way to stay up-to-date on your cloud costs, particularly if your budget is critical and time sensitive. With notifications, you can forward your budget messages to other mediums.

In this example we describe how to forward budget notifications to Slack. This way, every time Cloud Billing publishes a budget notification, a Cloud Run function uses a bot to post a message to a Slack channel of the bot's workspace.

Set up a Slack channel and permissions

The first step is to create your Slack workspace and the bot user tokens that are used to call the Slack API. API tokens can be managed at https://api.slack.com/apps. For detailed instructions, see Bot Users on the Slack site.

Write a Cloud Run function

1. Create a new function following the steps in Create a Cloud Run function. Ensure that the trigger is set to the same Pub/Sub topic that your budget is set to use.

2. Add dependencies:

   Node.js

   Copy the following to your package.json:

   {
     "name": "cloud-functions-billing",
     "private": "true",
     "version": "0.0.1",
     "description": "Examples of integrating Cloud Functions with billing",
     "main": "index.js",
     "engines": {
       "node": ">=16.0.0"
     },
     "scripts": {
       "compute-test": "c8 mocha -p -j 2 test/periodic.test.js --timeout=600000",
       "test": "c8 mocha -p -j 2 test/index.test.js --timeout=5000 --exit"
     },
     "author": "Ace Nassri <[e-mail]>",
     "license": "Apache-2.0",
     "dependencies": {
       "@google-cloud/billing": "^4.0.0",
       "@google-cloud/compute": "^4.0.0",
       "google-auth-library": "^9.0.0",
       "googleapis": "^143.0.0",
       "slack": "^11.0.1"
     },
     "devDependencies": {
       "@google-cloud/functions-framework": "^3.0.0",
       "c8": "^10.0.0",
       "gaxios": "^6.0.0",
       "mocha": "^10.0.0",
       "promise-retry": "^2.0.0",
       "proxyquire": "^2.1.0",
       "sinon": "^18.0.0",
       "wait-port": "^1.0.4"
     }
   }
   

   Python

   Copy the following to your requirements.txt:

   slackclient==2.9.4
   google-api-python-client==2.131.0
   

3. Write code or use the following example to post budget notifications to a Slack chat channel using the Slack API.

4. Ensure the following Slack API postMessage parameters are set correctly:

   • Bot User OAuth access token
   • Channel name

Example code:

Node.js

const slack = require('slack');

// TODO(developer) replace these with your own values
const BOT_ACCESS_TOKEN =
  process.env.BOT_ACCESS_TOKEN || 'xxxx-111111111111-abcdefghidklmnopq';
const CHANNEL = process.env.SLACK_CHANNEL || 'general';

exports.notifySlack = async pubsubEvent => {
  const pubsubAttrs = pubsubEvent.attributes;
  const pubsubData = Buffer.from(pubsubEvent.data, 'base64').toString();
  const budgetNotificationText = `${JSON.stringify(
    pubsubAttrs
  )}, ${pubsubData}`;

  await slack.chat.postMessage({
    token: BOT_ACCESS_TOKEN,
    channel: CHANNEL,
    text: budgetNotificationText,
  });

  return 'Slack notification sent successfully';
};

Python

import base64
import json
import os

import slack
from slack.errors import SlackApiError

# See https://api.slack.com/docs/token-types#bot for more info
BOT_ACCESS_TOKEN = "xxxx-111111111111-abcdefghidklmnopq"
CHANNEL = "C0XXXXXX"

slack_client = slack.WebClient(token=BOT_ACCESS_TOKEN)


def notify_slack(data, context):
    pubsub_message = data

    # For more information, see
    # https://cloud.google.com/billing/docs/how-to/budgets-programmatic-notifications#notification_format
    try:
        notification_attr = json.dumps(pubsub_message["attributes"])
    except KeyError:
        notification_attr = "No attributes passed in"

    try:
        notification_data = base64.b64decode(data["data"]).decode("utf-8")
    except KeyError:
        notification_data = "No data passed in"

    # This is just a quick dump of the budget data (or an empty string)
    # You can modify and format the message to meet your needs
    budget_notification_text = f"{notification_attr}, {notification_data}"

    try:
        slack_client.api_call(
            "chat.postMessage",
            json={"channel": CHANNEL, "text": budget_notification_text},
        )
    except SlackApiError:
        print("Error posting to Slack")

Now you can Test your Cloud Run function to see a message show up in Slack.

Cap (disable) billing to stop usage

This example shows you how to cap costs and stop usage for a project by disabling Cloud Billing. Disabling billing on a project causes all Google Cloud services in the project to terminate, including free-tier services.

Warning: This example removes Cloud Billing from your project, shutting down all resources. Resources might not shut down gracefully, and might be irretrievably deleted. There is no graceful recovery if you disable Cloud Billing.
You can re-enable Cloud Billing, but there's no guarantee of service recovery and manual configuration is required.

Why disable billing?

You might cap costs because you have a hard limit on how much money you can spend on Google Cloud. This is typical for students, researchers, or developers working in sandbox environments. In these cases you want to stop the spending and might be willing to shutdown all your Google Cloud services and usage when your budget limit is reached.

Note: There's a delay between incurring costs and receiving budget notifications. Google Cloud products report usage and cost data to Cloud Billing processes at varying intervals, so you might incur additional costs for usage that hasn't arrived at the time that all services are stopped.
Following the steps in this capping example doesn't guarantee that you won't spend more than your budget.
If you have a hard funds limit, set your maximum budget below your available funds to account for billing delays.

For our example, we use acme-backend-dev as a non-production project for which Cloud Billing can be safely disabled.

Before you start working with this example, ensure you've done the following:

• Enable the Cloud Billing API. Your Cloud Run function needs to call the Cloud Billing API to disable Cloud Billing for a project.

• Set up a budget to monitor project costs and enable budget notifications.

Write a Cloud Run function

Next, you need to configure your Cloud Run function to call the Cloud Billing API. This enables the Cloud Run function to disable Cloud Billing for our example project acme-backend-dev.

1. Create a new function following the steps in Create a Cloud Run function. Ensure that the trigger is set to the same Pub/Sub topic that your budget is set to use.

2. Add the following dependencies:

   Node.js

   Copy the following to your package.json:

   {
     "name": "cloud-functions-billing",
     "private": "true",
     "version": "0.0.1",
     "description": "Examples of integrating Cloud Functions with billing",
     "main": "index.js",
     "engines": {
       "node": ">=16.0.0"
     },
     "scripts": {
       "compute-test": "c8 mocha -p -j 2 test/periodic.test.js --timeout=600000",
       "test": "c8 mocha -p -j 2 test/index.test.js --timeout=5000 --exit"
     },
     "author": "Ace Nassri <[e-mail]>",
     "license": "Apache-2.0",
     "dependencies": {
       "@google-cloud/billing": "^4.0.0",
       "@google-cloud/compute": "^4.0.0",
       "google-auth-library": "^9.0.0",
       "googleapis": "^143.0.0",
       "slack": "^11.0.1"
     },
     "devDependencies": {
       "@google-cloud/functions-framework": "^3.0.0",
       "c8": "^10.0.0",
       "gaxios": "^6.0.0",
       "mocha": "^10.0.0",
       "promise-retry": "^2.0.0",
       "proxyquire": "^2.1.0",
       "sinon": "^18.0.0",
       "wait-port": "^1.0.4"
     }
   }
   

   Python

   Copy the following to your requirements.txt:

   slackclient==2.9.4
   google-api-python-client==2.131.0
   

3. Copy the following code into the Cloud Run function.

4. Set the function to execute to stopBilling (Node) or stop_billing (Python).

5. Depending on your runtime, the GOOGLE_CLOUD_PROJECT environment variable might be set automatically. Review the list of environment variables set automatically and determine if you need to manually set the GOOGLE_CLOUD_PROJECT variable to the project for which you want to cap (disable) Cloud Billing.

Note: If the environment variable is automatically set, the Cloud Billing account will be removed on the project where this Cloud Run function is running. Depending on your needs, you can change the code inside your Cloud Run function to specify a different project.

Node.js

const {CloudBillingClient} = require('@google-cloud/billing');
const {InstancesClient} = require('@google-cloud/compute');

const PROJECT_ID = process.env.GOOGLE_CLOUD_PROJECT;
const PROJECT_NAME = `projects/${PROJECT_ID}`;
const billing = new CloudBillingClient();

exports.stopBilling = async pubsubEvent => {
  const pubsubData = JSON.parse(
    Buffer.from(pubsubEvent.data, 'base64').toString()
  );
  if (pubsubData.costAmount <= pubsubData.budgetAmount) {
    return `No action necessary. (Current cost: ${pubsubData.costAmount})`;
  }

  if (!PROJECT_ID) {
    return 'No project specified';
  }

  const billingEnabled = await _isBillingEnabled(PROJECT_NAME);
  if (billingEnabled) {
    return _disableBillingForProject(PROJECT_NAME);
  } else {
    return 'Billing already disabled';
  }
};

/**
 * Determine whether billing is enabled for a project
 * @param {string} projectName Name of project to check if billing is enabled
 * @return {bool} Whether project has billing enabled or not
 */
const _isBillingEnabled = async projectName => {
  try {
    const [res] = await billing.getProjectBillingInfo({name: projectName});
    return res.billingEnabled;
  } catch (e) {
    console.log(
      'Unable to determine if billing is enabled on specified project, assuming billing is enabled'
    );
    return true;
  }
};

/**
 * Disable billing for a project by removing its billing account
 * @param {string} projectName Name of project disable billing on
 * @return {string} Text containing response from disabling billing
 */
const _disableBillingForProject = async projectName => {
  const [res] = await billing.updateProjectBillingInfo({
    name: projectName,
    resource: {billingAccountName: ''}, // Disable billing
  });
  return `Billing disabled: ${JSON.stringify(res)}`;
};

Python

import base64
import json
import os

from googleapiclient import discovery

PROJECT_ID = os.getenv("GCP_PROJECT")
PROJECT_NAME = f"projects/{PROJECT_ID}"
def stop_billing(data, context):
    pubsub_data = base64.b64decode(data["data"]).decode("utf-8")
    pubsub_json = json.loads(pubsub_data)
    cost_amount = pubsub_json["costAmount"]
    budget_amount = pubsub_json["budgetAmount"]
    if cost_amount <= budget_amount:
        print(f"No action necessary. (Current cost: {cost_amount})")
        return

    if PROJECT_ID is None:
        print("No project specified with environment variable")
        return

    billing = discovery.build(
        "cloudbilling",
        "v1",
        cache_discovery=False,
    )

    projects = billing.projects()

    billing_enabled = __is_billing_enabled(PROJECT_NAME, projects)

    if billing_enabled:
        __disable_billing_for_project(PROJECT_NAME, projects)
    else:
        print("Billing already disabled")


def __is_billing_enabled(project_name, projects):
    """
    Determine whether billing is enabled for a project
    @param {string} project_name Name of project to check if billing is enabled
    @return {bool} Whether project has billing enabled or not
    """
    try:
        res = projects.getBillingInfo(name=project_name).execute()
        return res["billingEnabled"]
    except KeyError:
        # If billingEnabled isn't part of the return, billing is not enabled
        return False
    except Exception:
        print(
            "Unable to determine if billing is enabled on specified project, assuming billing is enabled"
        )
        return True


def __disable_billing_for_project(project_name, projects):
    """
    Disable billing for a project by removing its billing account
    @param {string} project_name Name of project disable billing on
    """
    body = {"billingAccountName": ""}  # Disable billing
    try:
        res = projects.updateBillingInfo(name=project_name, body=body).execute()
        print(f"Billing disabled: {json.dumps(res)}")
    except Exception:
        print("Failed to disable billing, possibly check permissions")

Configure service account permissions

Your Cloud Run function is run as an automatically created service account. So that the service account can disable billing, you need to grant it the correct permissions, such as Billing Admin.

Note: Rather than grant full Billing Admin permissions to the service account, you might want to create a custom role that grants fewer permissions. To understand the permissions required, review the permissions necessary to disable billing for a project.

To identify the correct service account, view your Cloud Run function details. The service account is listed at the bottom of the page.

You can manage Billing Admin permissions on the Billing page in the Google Cloud console.

To grant Billing Account Administrator privileges to the service account, select the service account name.

Validate that Cloud Billing is disabled

When the budget sends out a notification, the specified project will no longer have a Cloud Billing account. If you want to test the function, publish a sample message with the preceding test message. The project will no longer be visible under the Cloud Billing account and resources in the project are disabled, including the Cloud Run function if it's in the same project.

You can manually re-enable Cloud Billing for your project in the Google Cloud console.

Selectively control usage

Capping (disabling) Cloud Billing as described in the previous example is binary and terminal. Your project is either enabled or disabled. When the project is disabled, all services are stopped and all resources are eventually deleted.

If you require a more nuanced response, you can selectively control resources. For example, if you want to stop some Compute Engine resources but leave Cloud Storage intact, then you can selectively control usage. This reduces your cost per hour without completely disabling your environment.

You can write as nuanced a policy as you like. However, for our example, our project is running research with a number of Compute Engine virtual machines, and is storing results in Cloud Storage. This Cloud Run function example shuts down all Compute Engine instances, but won't impact our stored results after the budget is exceeded.

Write a Cloud Run function

1. Create a new function following the steps in Create a Cloud Run function. Ensure that the trigger is set to the same Pub/Sub topic that your budget is set to use.

2. Make sure that you've added the dependencies described in Cap (disable) billing to stop usage.

3. Copy the following code into the Cloud Run function.

4. Set the function to execute to limitUse (Node) or limit_use (Python).

5. Depending on your runtime, the GCP_PROJECT environment variable might be set automatically. Review the list of environment variables set automatically and determine if you need to manually set the GCP_PROJECT variable to the project running the virtual machines.

6. Set the ZONE parameter. This is the zone where instances will be stopped for this sample.

Note: This code sample only stops instances in a single zone in the specified project. Depending on your needs, you can change the code inside your Cloud Run function to handle other projects and zones.

Node.js

const {CloudBillingClient} = require('@google-cloud/billing');
const {InstancesClient} = require('@google-cloud/compute');

const PROJECT_ID = process.env.GOOGLE_CLOUD_PROJECT;
const PROJECT_NAME = `projects/${PROJECT_ID}`;
const instancesClient = new InstancesClient();
const ZONE = 'us-central1-a';

exports.limitUse = async pubsubEvent => {
  const pubsubData = JSON.parse(
    Buffer.from(pubsubEvent.data, 'base64').toString()
  );
  if (pubsubData.costAmount <= pubsubData.budgetAmount) {
    return `No action necessary. (Current cost: ${pubsubData.costAmount})`;
  }

  const instanceNames = await _listRunningInstances(PROJECT_ID, ZONE);
  if (!instanceNames.length) {
    return 'No running instances were found.';
  }

  await _stopInstances(PROJECT_ID, ZONE, instanceNames);
  return `${instanceNames.length} instance(s) stopped successfully.`;
};

/**
 * @return {Promise} Array of names of running instances
 */
const _listRunningInstances = async (projectId, zone) => {
  const [instances] = await instancesClient.list({
    project: projectId,
    zone: zone,
  });
  return instances
    .filter(item => item.status === 'RUNNING')
    .map(item => item.name);
};

/**
 * @param {Array} instanceNames Names of instance to stop
 * @return {Promise} Response from stopping instances
 */
const _stopInstances = async (projectId, zone, instanceNames) => {
  await Promise.all(
    instanceNames.map(instanceName => {
      return instancesClient
        .stop({
          project: projectId,
          zone: zone,
          instance: instanceName,
        })
        .then(() => {
          console.log(`Instance stopped successfully: ${instanceName}`);
        });
    })
  );
};

Python

import base64
import json
import os

from googleapiclient import discovery

PROJECT_ID = os.getenv("GCP_PROJECT")
PROJECT_NAME = f"projects/{PROJECT_ID}"
ZONE = "us-west1-b"


def limit_use(data, context):
    pubsub_data = base64.b64decode(data["data"]).decode("utf-8")
    pubsub_json = json.loads(pubsub_data)
    cost_amount = pubsub_json["costAmount"]
    budget_amount = pubsub_json["budgetAmount"]
    if cost_amount <= budget_amount:
        print(f"No action necessary. (Current cost: {cost_amount})")
        return

    compute = discovery.build(
        "compute",
        "v1",
        cache_discovery=False,
    )
    instances = compute.instances()

    instance_names = __list_running_instances(PROJECT_ID, ZONE, instances)
    __stop_instances(PROJECT_ID, ZONE, instance_names, instances)


def __list_running_instances(project_id, zone, instances):
    """
    @param {string} project_id ID of project that contains instances to stop
    @param {string} zone Zone that contains instances to stop
    @return {Promise} Array of names of running instances
    """
    res = instances.list(project=project_id, zone=zone).execute()

    if "items" not in res:
        return []

    items = res["items"]
    running_names = [i["name"] for i in items if i["status"] == "RUNNING"]
    return running_names


def __stop_instances(project_id, zone, instance_names, instances):
    """
    @param {string} project_id ID of project that contains instances to stop
    @param {string} zone Zone that contains instances to stop
    @param {Array} instance_names Names of instance to stop
    @return {Promise} Response from stopping instances
    """
    if not len(instance_names):
        print("No running instances were found.")
        return

    for name in instance_names:
        instances.stop(project=project_id, zone=zone, instance=name).execute()
        print(f"Instance stopped successfully: {name}")

Configure service account permissions

1. Your Cloud Run function is run as an automatically created service account. To control usage, you need to grant the service account permissions to any services on the project that it needs to make changes.
   Note: Security best practice is to grant the least privilege specific to the services it will manage. The Compute Instance Admin (v1) role will work here. However, to be more secure, you might want to configure and use a custom role that grants fewer permissions than an Admin role.
2. To identify the correct service account, view the details of your Cloud Run function. The service account is listed at the bottom of the page.
3. In the Google Cloud console, go to the IAM page to set the appropriate permissions.
   Go to the IAM page
    
   

Validate that instances are stopped

When the budget sends out a notification, the Compute Engine virtual machines are stopped.

To test the function, publish a sample message with the testing message from earlier. To validate that the function ran successfully, check your Compute Engine virtual machines in the Google Cloud console.