Managing Cloud Monitoring dashboards automatically using the Cloud Monitoring API

This tutorial describes how to use the Cloud Monitoring API to treat your dashboards as code. Using the Monitoring API, you can store your dashboards in version control, and automate dashboard creation and modification with Cloud Build. This document is intended for DevOps engineers who create and maintain monitoring infrastructure.

DevOps engineers usually maintain continuous integration/continuous delivery (CI/CD) pipelines to build and deploy software. However, monitoring infrastructure such as monitoring dashboards is often not part of the pipeline and is manually updated.

DevOps engineers want consistent dashboards that are easy to maintain. It is important to know who made changes and which dashboards were changed, to validate that those changes are good, and to be able to undo or repeat changes easily. You also want to push these dashboard changes to production reliably.

This tutorial shows you how to version-control your Monitoring dashboards using the Monitoring API. Version control lets you track who did what to your dashboards, so you can have an audit trail and a way to review and roll back changes. You can also restrict access to the Monitoring API so that teams cannot manually change dashboards, and must use the CI/CD pipeline instead. Restricting access ensures that all changes go through version control and that dashboards remain consistent across projects because only the CI/CD pipeline can apply changes. When you restrict access to the Monitoring API, you help ensure that only validated version-controlled dashboards can be pushed to production.

This tutorial assumes that you are familiar with Monitoring, Cloud Build, and Git, and that you have a GitHub user account. When you finish this tutorial, you have a pipeline that pushes dashboard updates to multiple projects whenever a new change is committed. Figure 1 shows the Monitoring dashboard architecture.

A typical workflow for changing Monitoring dashboard architecture.

Figure 1. Diagram of the Monitoring dashboard architecture when treating the dashboard as code.

Figure 1 shows the following typical workflow for making changes to your Monitoring dashboard architecture:

  1. An engineer develops and tests changes to dashboards in the development Cloud project.
  2. The engineer pushes the dashboard changes to the GitHub repository.
  3. Changes are automatically deployed to the staging Cloud project.
  4. A peer reviews and approves the changes in the staging Cloud project.
  5. Approved changes are automatically deployed to the production Cloud project.

Objectives

  • Version-control your dashboards so that you can:
    • Inspect differences between versions.
    • View an audit trail of changes.
    • Deploy only changes that have been reviewed.
    • Quickly find and roll back changes that are problematic.
  • Test dashboard changes before deploying to production.
  • Deploy a dashboard automatically to production.

About this tutorial

This tutorial uses GitHub to host Git repositories, but you can achieve the same results with Bitbucket or Cloud Source Repositories.

You can access the Monitoring API by using the REST API, gRPC API, or by using the gcloud command-line tool integration. You use the gcloud tool integration in this tutorial.

This tutorial assumes that you are a DevOps engineer who has the following responsibilities:

  • Ensure that all teams in your organization have a consistent view of their production infrastructure through a common set of dashboards.
  • Provide a workflow to review planned dashboard changes without modifying existing dashboards in production.
  • Automatically deploy dashboard changes to production.

This tutorial walks you through the steps needed to fulfill these responsibilities.

Costs

This tutorial uses the following billable components of Google Cloud:

  • Compute Engine
  • Monitoring
  • Cloud Build

To generate a cost estimate based on your projected usage, use the pricing calculator. New Google Cloud users might be eligible for a free trial.

Before you begin

Create projects

For this tutorial, you create the following Google Cloud projects:

  • A development project where you experiment with changes to your dashboard.
  • A staging project where changes are reviewed.
  • A production project where you deploy the reviewed dashboard changes.

Follow the steps below to create each project:

  1. In the Google Cloud Console, create a Google Cloud project.

    Go to the Google Cloud Console

  2. Make sure that billing is enabled for your Cloud project. Learn how to confirm that billing is enabled for your project.

  3. In the Cloud Console, activate Cloud Shell.

    Activate Cloud Shell

    You use Cloud Shell to run all the commands in this tutorial.

  4. Configure Cloud Shell to use your project. Replace PROJECT_ID with your own project ID.

    gcloud config set project PROJECT_ID
    
  5. Enable the required Google Cloud APIs:

    gcloud services enable \
        compute.googleapis.com \
        monitoring.googleapis.com \
        cloudbuild.googleapis.com
    

Repeat the preceding steps to create your staging and production projects and enable required APIs.

When you finish this tutorial, you can avoid continued billing by deleting the resources you created. For more information, see Cleaning up.

Configure gcloud tool

After you create the development, staging, and production projects, follow the steps below to complete your configuration:

  1. Set the gcloud tool default for the Compute Engine region and zone that you want to use in your projects for this tutorial. This tutorial uses the us-central1 region and the us-central1-a zone. You can change the region and zone to suit your needs. For more information, see Geography and regions.

    gcloud config set compute/region us-central1
    gcloud config set compute/zone us-central1-a
    
  2. Save the staging and production project IDs in environment variables. Replace DEV_PROJECT_ID with your development project ID, STAGING_PROJECT_ID with your own staging project ID, and PROD_PROJECT_ID with your own production project ID.

    DEV_PROJECT_ID=DEV_PROJECT_ID
    STAGING_PROJECT_ID=STAGING_PROJECT_ID
    PROD_PROJECT_ID=PROD_PROJECT_ID
    
  3. Configure Git in Cloud Shell with your name and email address. Git uses this information to identify you as the author of the commits you create in Cloud Shell. Replace YOUR_EMAIL_ADDRESS with your email address and YOUR_NAME with your name. If you want to connect to GitHub without supplying your username or password with each connection, you can connect to GitHub using SSH.

    git config --global user.email "YOUR_EMAIL_ADDRESS"
    git config --global user.name "YOUR_NAME"
    

Setting up required components

You begin by creating the required infrastructure and dashboard to simulate an existing environment, as shown in the following diagram:

Project components for the development, staging, and production projects.

Figure 2. Diagram of the components that you create for each project in this tutorial.

Figure 2 shows the following components, which you create for each project—development, staging, and production—in this tutorial:

  • Project components:
    • VM1
    • VM2
    • Workspace
      • Dashboard

Create VM instances

In this section, you create virtual machine (VM) instances in your development, staging, and production projects. The VMs generate monitoring metrics to display in the dashboards that you create later.

  1. In Cloud Shell, create VM instances vm1 and vm2 in your development project:

    gcloud compute instances create vm1 vm2 --project=${DEV_PROJECT_ID}
    
  2. Create new VM instances vm1 and vm2 in your staging project:

    gcloud compute instances create vm1 vm2 --project=${STAGING_PROJECT_ID}
    
  3. Create new VM instances vm1 and vm2 in your production project:

    gcloud compute instances create vm1 vm2 --project=${PROD_PROJECT_ID}
    

Create the dashboard

Next, you create a dashboard in each Cloud project to simulate an existing environment.

  • The development Cloud project lets you safely experiment with changes to your dashboard without affecting dashboards that are already live in production projects.
  • The staging Cloud project lets you deploy changes for review.
  • The production Cloud project lets your team monitor production infrastructure and deploys all reviewed changes.

This tutorial uses predefined dashboards. You can also create a dashboard manually through the Cloud Console, or use the API to create a dashboard using a predefined dashboard.

  1. Configure Cloud Shell to use your development project:

    gcloud config set project $DEV_PROJECT_ID
    
  2. Clone the repository of sample dashboards:

    git clone https://github.com/GoogleCloudPlatform/monitoring-dashboard-samples.git
    
  3. Navigate to the repository that you cloned:

    cd monitoring-dashboard-samples
    
  4. Create an environment variable for your dashboard ID. Replace DASHBOARD_ID with the dashboard ID that you want to create:

    DASHBOARD_ID=DASHBOARD_ID
    
  5. Create the fully qualified development dashboard name from the dashboard ID:

    DASHBOARD_NAME="\"name\": \"projects/$DEV_PROJECT_ID/dashboards/$DASHBOARD_ID\","
    

    When you create a dashboard, the API automatically generates an alphanumeric dashboard ID. The preceding command overrides the default behavior to specify a custom dashboard ID by setting the name field of a Dashboard object. A custom dashboard ID is a meaningful identifier that helps other people understand the purpose of the dashboard when they encounter it in environment variables or your CI/CD pipeline.

  6. Add the dashboard ID to the sample dashboard in the repository that you cloned:

    sed -i '/"displayName":/a'"${DASHBOARD_NAME}" dashboards/compute/gce-vm-instance-monitoring.json
    
  7. To create the dashboard in your development Cloud project, use the gcloud tool integration:

    gcloud monitoring dashboards create \
        --config-from-file=dashboards/compute/gce-vm-instance-monitoring.json
    
  8. Clear your changes:

    git checkout .
    
  9. Change to your staging project:

    gcloud config set project $STAGING_PROJECT_ID
    
  10. Create the fully qualified staging dashboard name from the dashboard ID:

    DASHBOARD_NAME="\"name\": \"projects/$STAGING_PROJECT_ID/dashboards/$DASHBOARD_ID\","
    
  11. Add the dashboard ID to the sample dashboard:

    sed -i '/"displayName":/a'"${DASHBOARD_NAME}" dashboards/compute/gce-vm-instance-monitoring.json
    
  12. To create the dashboard in your staging Cloud project, use the gcloud tool integration:

    gcloud monitoring dashboards create \
        --config-from-file=dashboards/compute/gce-vm-instance-monitoring.json
    
  13. Clear your changes:

    git checkout .
    
  14. Change to your production project:

    gcloud config set project $PROD_PROJECT_ID
    
  15. Create the fully qualified production dashboard name from the dashboard ID:

    DASHBOARD_NAME="\"name\": \"projects/$PROD_PROJECT_ID/dashboards/$DASHBOARD_ID\","
    
  16. Add the dashboard ID to the sample dashboard:

    sed -i '/"displayName":/a'"${DASHBOARD_NAME}" dashboards/compute/gce-vm-instance-monitoring.json
    
  17. To create the dashboard in your production Cloud project, use the gcloud tool integration:

    gcloud monitoring dashboards create \
        --config-from-file=dashboards/compute/gce-vm-instance-monitoring.json
    

Your dashboard is now programmatically created across your Google Cloud projects. Using the Monitoring API programmatically means that the time to deploy your dashboards remains constant even as the number of dashboards grows.

Verify dashboard creation

Check that your dashboard was created in each of your Google Cloud projects in the Cloud Console.

  1. In the Cloud Console, go to the Monitoring > Dashboards page:

    Go to the Dashboards page

  2. Select your development Cloud project.

  3. The dashboard you created appears as GCE VM Instance Monitoring.

  4. Repeat the preceding steps to verify that a dashboard was created for your staging and production projects.

If the dashboard doesn't appear in one of your projects, repeat the steps to Create the dashboard for that project.

In the following sections, you learn how to experiment safely with changes to your dashboard, and have these changes reviewed before deploying them to production.

Creating a GitHub repository

In this section, you create a GitHub repository that you use for the rest of this tutorial:

  1. Create a GitHub repository.
    1. Select Initialize this repository with a README. This lets you clone the repository immediately.

This repository versions your Monitoring dashboard, which lets you do the following:

  • Inspect differences between versions.
  • View an audit trail of changes.
  • Deploy only changes that have been reviewed.
  • Share a common dashboard between teams.

Versioning your dashboard

Next, you save the JSON representation of your dashboard, so that you can version your dashboard and treat it as code.

  1. Configure Cloud Shell to use your development project:

    gcloud config set project $DEV_PROJECT_ID
    
  2. Clone the GitHub repository that you created to your home directory:

    cd ~
    git clone YOUR_REPOSITORY_NAME
    
  3. Navigate to the cloned repository:

    cd YOUR_REPOSITORY_NAME
    
  4. Get the JSON representation of your dashboard and save it to a file:

    gcloud monitoring dashboards describe $DASHBOARD_ID --format=json > dashboard.json
    

    This dashboard.json file includes an ETag, which is an identifier for a specific version of a resource—in this case, your dashboard. When you update a dashboard by using the API, you specify the dashboard ID and the ETag, which identifies a specific version of the dashboard. Each time the dashboard is modified, a new ETag is generated.

    The dashboard.json file includes a name field, which specifies the dashboard ID and the project ID.

  5. To use the JSON file for dashboard creation, remove the ETag and parameterize the name field:

    sed -i '/"etag":/d' dashboard.json
    sed -i 's/projects\/[0-9]\+/projects\/[PROJECT_ID]/g' dashboard.json
    
  6. Check in the dashboard file to your repository:

    git add .
    git commit -m "Add dashboard file"
    git push
    

Your dashboard is now under version control. Version control lets you view a history of changes, and you can configure your environment so that dashboards can be modified only through your CI/CD pipeline.

Following the principle of least privilege, make sure that only your CI/CD pipeline has read-write access to the API to modify your dashboards. Give all other users read access to the API so that they can view the dashboard configuration. Allowing only your CI/CD pipeline to make changes helps to enforce a consistent environment and ensures a single source of truth for your dashboard changes.

Creating a build trigger to push changes to staging

In this section, you create a build trigger that runs when you push a change to the development branch in your version control system. This build trigger deploys your dashboard to staging, using the dashboard stored in your version control system.

  1. In the Cloud Console, select your staging project.
  2. Connect your GitHub repository to your staging project. When you are asked to connect your GitHub repositories to your Cloud project, choose one of the following options based on your business need:
    • All repositories - enable all current and future GitHub repositories for access by using the Cloud Build app.
    • Only select repositories - enable only specific repositories for access by using the Cloud Build app. You can enable additional repositories later. To select a repository for this tutorial, use the Select repositories drop-down to enable the repository you created earlier.
  3. Create a GitHub App trigger with the following trigger settings:

    1. Name: enter the name you want to use for the trigger.
    2. Event: select Push to a branch.
    3. Source: select the GitHub repository that you created earlier.
    4. Branch: enter ^dev$.
    5. Configuration: select Cloud Build configuration file (yaml or json). Keep the default cloudbuild.yaml configuration file location.
    6. Under Substitution variables, click Add variable.
    7. Add the new variable values:

      1. Variable: enter _DASHBOARD_ID.
      2. Value: enter the value of your DASHBOARD_ID

      When the build executes, all occurrences of _DASHBOARD_ID in your cloudbuild.yaml file are replaced with the value you provide.

  4. Click Create.

You now have a build trigger that runs when a commit is pushed to your version control system. You can also create build triggers that run only when there are changes that match certain criteria. For example, you can create triggers that validate your configuration against a set of rules.

Granting permissions to your Cloud Build service account

Cloud Build uses a service account to execute builds for you. When you enabled the Cloud Build API earlier, the Cloud Build service account was automatically created and granted the Cloud Build Service Account role for the project. This role gives the service account permissions to perform several tasks. However, you can grant more permissions to the service account to perform additional tasks.

In this section, you grant the Cloud Build service account in your staging project additional permissions to read and write dashboards to your Cloud project.

  1. Configure Cloud Shell to use your staging project:

    gcloud config set project $STAGING_PROJECT_ID
    
  2. Grant the Cloud Build service account the required permissions:

    PROJECT_NUMBER="$(gcloud projects describe ${STAGING_PROJECT_ID} \
      --format='get(projectNumber)')"
    
    gcloud projects add-iam-policy-binding ${PROJECT_NUMBER} \
      --member=serviceAccount:${PROJECT_NUMBER}@cloudbuild.gserviceaccount.com \
      --role=roles/monitoring.dashboardEditor
    

Changing the dashboard in development

The following sections show you how to safely test changes to your dashboard in a development project, get the changes reviewed, and then automatically deploy your changes to production. Automatic deployment is possible because of the previous steps that you took to treat your dashboards as code and place them under version control.

  1. In the Cloud Console, go to the Monitoring > Dashboards page:

    Go to the Dashboards page

  2. Select your development Cloud project.

  3. Click the dashboard that you created earlier in this tutorial.

  4. On the GCE VM Instance - CPU Utilization chart, click More chart options, and then click Edit.

  5. In the Edit Chart dialog that appears, change the Chart Title value to My first change.

  6. Click Save.

Updating the dashboard with the Monitoring API

To update a dashboard, you can use the Monitoring API to deploy an update to the existing dashboard, or delete the existing dashboard and recreate it with the updated definition.

Deploying an updated dashboard

If you want to deploy an updated dashboard, you store the ETag of the dashboard at the time of creation or when you first obtained the JSON representation. Then, when you want to update the dashboard, you specify that ETag to ensure that the dashboard hasn't changed. If other engineers have changed the dashboard, specifying the ETagensures that you don't overwrite the dashboard with your changes. It also means that you have to maintain state.

However, if you lock down your environment so that only your CI/CD pipeline can change your dashboard, you change who is responsible for maintaining state. Your version control system is now responsible for tracking changes over time, instead of an engineer checking for identical ETags between commits. Any overwritten changes can be viewed in the commit history of your version control system.

Deleting and recreating the dashboard

When you want to update the dashboard, you can delete the current dashboard and create a new dashboard with the desired changes. This approach is simpler to automate, reason, and debug, because you only need to specify the Cloud project that you want to deploy to and the dashboard to delete. When you delete and recreate a dashboard, you don't have to maintain state in the form of ETags. The feasibility of this approach depends on the fact that only your CI/CD pipeline can change your dashboards. This approach treats your dashboards as a form of immutable infrastructure. You can always safely restore your dashboard from your version control system if there are unintended changes, such as accidental modifications.

The following sections show you how to implement this approach to delete the dashboard and then recreate it.

Deploying the development dashboard to staging

When you push your changes to your repository, the build trigger starts a build that deploys your dashboard to your staging Cloud project. The build executes the steps specified in a cloudbuild.yaml file that you provide.

Create a Cloud Build configuration file

Create a Cloud Build YAML configuration file in your local repository with the following content:

cat >cloudbuild.yaml <<EOF
steps:
# Substitute project ID placeholder in dashboard file with project ID supplied by Cloud Build
- name: 'google/cloud-sdk'
  id : 'Substitute project ID'
  entrypoint: 'bash'
  args:
    - '-c' # pass what follows as a command to bash
    - |
      sed -i 's/\[PROJECT_ID\]/'\$PROJECT_ID'/g' dashboard.json

- name: 'google/cloud-sdk'
  id: 'Delete dashboard'
  args:
    [
      'gcloud', 'monitoring', 'dashboards', 'delete',
      '--quiet',
      'projects/\$PROJECT_ID/dashboards/\${_DASHBOARD_ID}'
    ]
- name: 'google/cloud-sdk'
  id: 'Create dashboard'
  args:
    [
      'gcloud', 'monitoring', 'dashboards', 'create',
      '--config-from-file=dashboard.json'
    ]
EOF

You can now automatically push the dashboard in development to your staging project.

Deploy the change automatically

  1. Configure Cloud Shell to use your development project:

    gcloud config set project $DEV_PROJECT_ID
    
  2. Get the new JSON representation of your dashboard from your staging Cloud project:

    gcloud monitoring dashboards describe $DASHBOARD_ID --format=json > dashboard.json
    
  3. Remove the ETag and parameterize the name field from the JSON file so that you can use it for dashboard creation:

    sed -i '/"etag":/d' dashboard.json
    sed -i 's/projects\/[0-9]\+/projects\/[PROJECT_ID]/g' dashboard.json
    
  4. Push the Cloud Build configuration to GitHub:

    git checkout -b dev
    git add .
    git commit -m "Add changes"
    git push --set-upstream origin dev
    
  5. In the Cloud Console, go to the History page.

    Go to the History page

  6. Select your staging project.

  7. The Build history page displays the status of the build trigger that's being executed.

You can also use this architecture to automatically push dashboard updates to other projects. For each project that you want to push updates to, you create a build trigger that starts a build that pushes updates whenever there is a new commit to the version control system.

After the trigger completes, you can check that your development dashboard has been deployed to staging.

Checking deployment to staging

Check that your development dashboard has been deployed to your staging Cloud project in the Cloud Console.

  1. In the Cloud Console, go to the Monitoring > Dashboards page.

    Go to the Dashboards page

  2. Select your staging Cloud project.

  3. Click the dashboard you created earlier in this tutorial.

  4. In the GCE VM Instance Monitoring dashboard, verify that the My first change chart is available.

You have now automatically deployed your dashboard from your development Cloud project to your staging Cloud project without reproducing the dashboard manually. This automation makes it easy to deploy dashboards consistently across Cloud projects, independent of the number of charts in each dashboard, or their complexity.

Creating a build trigger to push changes to production

In this section, you create a build trigger that runs when a pull request is reviewed and merged to main. This build trigger deploys your staging dashboard to production, using the dashboard that's stored in your version control system.

  1. In the Cloud Console, select your production project.
  2. Connect your GitHub repository to your production project and select the repository you created earlier in this tutorial. The repository is available because the Cloud Build GitHub App was already installed into your repository the first time you created a build trigger in your staging project.
  3. Create a GitHub App trigger.

    1. Name: enter the name you want to use for the trigger.
    2. Event: select Push to a branch.
    3. Source: select the GitHub repository that you created earlier.
    4. Branch: enter ^main$.
    5. Configuration: select Cloud Build configuration file (yaml or json).
    6. Under Substitution variables, click Add variable.
    7. Add the new variable values:

      1. Variable: enter _DASHBOARD_ID.
      2. Value: enter the value of your DASHBOARD_ID

      When the build executes, all occurrences of _DASHBOARD_ID in your cloudbuild.yaml file are replaced with the value you provide.

  4. Click Create.

You now have a build trigger that runs when a commit is pushed to your version control system. You can also create build triggers that run only when there are changes that match certain criteria. For example, you can create triggers that validate your configuration against a set of rules.

Granting permissions to your Cloud Build service account

Next, you grant the Cloud Build service account in your production project permission to read and write dashboards to your Cloud project.

  1. Configure Cloud Shell to use your production project:

    gcloud config set project $PROD_PROJECT_ID
    
  2. Grant the Cloud Build service account the required permissions:

    PROJECT_NUMBER="$(gcloud projects describe ${PROD_PROJECT_ID} \
      --format='get(projectNumber)')"
    
    gcloud projects add-iam-policy-binding ${PROJECT_NUMBER} \
      --member=serviceAccount:${PROJECT_NUMBER}@cloudbuild.gserviceaccount.com \
      --role=roles/monitoring.dashboardEditor
    

Creating a pull request

The changes to the dashboard are now in the staging Cloud project and they are ready for review. Changes were pushed to GitHub when you completed the setup to deploy changes automatically earlier in this tutorial. Next, you create a pull request to start the review process.

  1. Go to the GitHub page of the repository you created earlier in this tutorial.
  2. Click Compare & pull request.

    The GitHub interface to compare and pull request.

  3. Enter a title and comment for the pull request. The title should summarize what this pull request is going to do. In this case, the title states that the pull request will modify the title. The comment should provide details about the pull request. In this case, the comment explains where to review the changes, and the results of approving the pull request.

    The GitHub interface to open a pull request.

  4. Click Create pull request.

You have now created a pull request that represents a set of changes to be reviewed.

Reviewing the pull request

Your pull request is reviewed in the staging Cloud project, and then merged to automatically deploy changes to production. Pull requests are usually reviewed by a peer. In this tutorial, you act as your peer to review your own changes. To review changes, do the following to visit your staging Cloud project:

  1. In the Cloud Console, go to the Monitoring > Dashboards page:

    Go to the Dashboards page

  2. Select your staging Cloud project.

  3. Click the dashboard that you created earlier in this tutorial.

  4. In the GCE VM Instance Monitoring dashboard, review your title change.

Merging the pull request

After changes are reviewed, merge the pull request:

  1. Go to the pull request page for the repository you created earlier in this tutorial.
  2. To merge the pull request, click Merge pull request and then click Confirm Merge.

    The GitHub interface with "Merge pull request" highlighted.

To see the status of the build trigger that's being executed, go to the Build history page in the Cloud Console, and then select your production project.

After the trigger completes, you can check that your staging dashboard has been deployed to production.

Checking that your staging dashboard has been deployed to production

  1. In the Cloud Console, go to the Monitoring > Dashboards page:

    Go to the Dashboards page

  2. Select your production Cloud project.

  3. Click the dashboard you created earlier in this tutorial.

  4. In the GCE VM Instance Monitoring dashboard, verify that the My first change chart is available.

You have automatically deployed your dashboard in your staging Cloud project to your production Cloud project without reproducing the dashboard manually. This automation makes it easy to deploy dashboards consistently across Cloud projects, independent of the number of charts in each dashboard, or their complexity.

Clean up

  1. In the Cloud Console, go to the Manage resources page.

    Go to Manage resources

  2. In the project list, select the project that you want to delete, and then click Delete.
  3. In the dialog, type the project ID, and then click Shut down to delete the project.

What's next