Estimate your GKE costs early in the development cycle using GitHub

This tutorial demonstrates the best practice of shifting Google Kubernetes Engine (GKE) cost visibility left to your development team. This shift-left practice generates awareness of costs early in the process, helping you avoid surprises in your Google Cloud bill.

This tutorial is intended for developers, operators, and FinOps practitioners who want to optimize costs in GKE clusters and who use GitHub in production. If you use GitLab instead, see Estimate your GKE costs early in the development cycle using GitLab.

The tutorial assumes that you are familiar with Docker, GitHub Kubernetes, GKE, Cloud Build, and Linux.

Overview

Many teams embracing the public cloud are not used to the pay-as-you-go billing style. Frequently, they don't fully understand the environment their apps are running on—in this case, GKE. The FinOps operating model promotes this culture of financial accountability. A FinOps best practice is to provide teams with real-time information about their spending so that cost issues can be addressed as soon as they arise.

This document shows you how to go one step further by estimating cost before it becomes an expense in your bill. As highlighted in the GitHub website, "On GitHub, lightweight code review tools are built into every pull request." This lets you "evolve projects, propose new features, and discuss implementation details before changing your source code." The best time to estimate costs is early in the process during development and at code review time. This way, practitioners can understand and discuss alternatives for the cost impact of new features and bug fixes before it becomes a problem. The following diagram summarizes such a practice.

Best practice of estimating cost early.

As the diagram shows, a developer estimates GKE costs in their local environment, ideally at build time. This estimate gives them a good understanding of the monthly production workload cost. When the feature or bug fix is code complete, they propose a pull request that triggers Cloud Build to check the difference between the old and the new cost. If there are increases above a predefined threshold, they can request a new code review. This practice helps developers become more aware of their workload capacity and proactively fix application issues instead of adding more resources each time an instability is found in production.

Objectives

  • Build and push the Kubernetes cost-estimator image.
  • Create a GitHub repository.
  • Connect Cloud Build to your GitHub repository.
  • Push the example code to your GitHub repository.
  • Change the code and propose a pull request to see the cost estimation in action.

Costs

This tutorial uses the following billable components of Google Cloud:

To generate a cost estimate based on your projected usage, use the pricing calculator. New Google Cloud users might be eligible for a free trial.

When you finish this tutorial, you can avoid continued billing by deleting the resources you created. For more information, see Clean up.

Before you begin

  1. In the Google Cloud Console, go to the project selector page.

    Go to project selector

  2. Select or create a Google Cloud project.

  3. Make sure that billing is enabled for your Cloud project. Learn how to confirm that billing is enabled for your project.

  4. In the Cloud Console, activate Cloud Shell.

    Activate Cloud Shell

    At the bottom of the Cloud Console, a Cloud Shell session starts and displays a command-line prompt. Cloud Shell is a shell environment with the Cloud SDK already installed, including the gcloud command-line tool, and with values already set for your current project. It can take a few seconds for the session to initialize.

Preparing your environment

  1. In Cloud Shell, clone the gke-shift-left-cost GitHub repository:

    git clone https://github.com/GoogleCloudPlatform/gke-shift-left-cost
    cd gke-shift-left-cost
    

    The code in this repository is structured into the following folders:

    • Root: Contains a Dockerfile file that is used to build the cost-estimator image and the main.go file that implements the command-line logic for the estimator.
    • api/: Contains the Golang API for manipulating Kubernetes objects and making the cost estimation.
    • samples/: Contains examples of Kubernetes manifests so that you can experiment with the process before implementing it in your organization.
  2. Set your Cloud project ID, your GitHub user and email address, and another GitHub user to act as the FinOps reviewer:

    export GCP_PROJECT_ID=YOUR_PROJECT_ID
    export GITHUB_USER=YOUR_GITHUB_USER
    export GITHUB_EMAIL=YOUR_GITHUB_EMAIL_ADDRESS
    export GITHUB_FINOPS_REVIEWER_USER=ANOTHER_GITHUB_USER
    

    Replace the following:

    • YOUR_PROJECT_ID: the Cloud project ID for the project that you're using in this tutorial.
    • YOUR_GITHUB_USER: the user that you use to log in to your GitHub account.
    • YOUR_GITHUB_EMAIL_ADDRESS: the email that you use in your GitHub account.
    • ANOTHER_GITHUB_USER: another GitHub user to act as the FinOps reviewer. Note that this tutorial requires that you add this user as a repository collaborator and therefore it cannot be yourself. To avoid getting stuck during the tutorial steps, make sure this user accepts your invitation as soon as you create the invite.
  3. Set the Cloud project and enable required APIs:

    gcloud config set project $GCP_PROJECT_ID
    
    gcloud services enable cloudbilling.googleapis.com \
        artifactregistry.googleapis.com \
        cloudbuild.googleapis.com
    

Building and pushing the Kubernetes cost-estimator image

The Kubernetes cost-estimator tool that comes with this tutorial is just an example of what can be done. It offers the capability of estimating cost for DaemonSet, Deployment, StatefulSet, ReplicaSet, HorizontalPodAutoScaler, and PersistentVolumeClaim Kubernetes objects. You can also implement your own cost-estimation tool or propose pull requests with the improvements that you want.

  1. In Cloud Shell, allow application-default to use your credentials:

    gcloud auth application-default login
    
  2. Build the Kubernetes cost-estimator binary:

    mkdir ./bin
    go test ./api
    go build -v -o ./bin/k8s-cost-estimator .
    
  3. Test the binary by executing cost estimation in a sample folder:

    ./bin/k8s-cost-estimator \
        --k8s ./samples/k8s-cost-estimator-local/app-v1  \
        --config ./samples/k8s-cost-estimator-local/example-conf.yaml --v trace
    

    In the output, you see a Markdown table detailing the monthly estimated costs for the ./samples/k8s-cost-estimator-local/app-v1/ folder. To better understand the monthly production cost of their applications, developers can run this step before pushing to the remote repository.

    INFO[0000] Starting cost estimation (version v0.0.1)...
    ...
    
    |         KIND          | MIN REQUESTED (USD) | MIN REQ + HPA CPU BUFFER (USD) | MAX REQUESTED (USD) | MIN LIMITED (USD) | MAX LIMITED (USD) |
    |-----------------------|---------------------|--------------------------------|---------------------|-------------------|-------------------|
    | Deployment            |             $133.31 |                        $198.71 |             $266.54 |           $312.83 |           $579.29 |
    | StatefulSet           |              $36.33 |                         $36.33 |              $36.33 |            $72.67 |            $72.67 |
    | DaemonSet             |              $29.68 |                         $29.68 |              $29.68 |            $53.19 |            $53.19 |
    | PersistentVolumeClaim |              $28.88 |                         $28.88 |              $28.88 |            $33.68 |            $33.68 |
    | **TOTAL**             |         **$228.20** |                    **$293.60** |         **$361.43** |       **$472.38** |       **$738.83** |
    
    INFO[0002] Finished cost estimation!
    
  4. Build the Kubernetes cost-estimator container image:

    docker build . -t us-central1-docker.pkg.dev/$GCP_PROJECT_ID/docker-repo/k8s-cost-estimator:v0.0.1
    
  5. Create the Artifact Registry Docker repository to store the image:

    gcloud artifacts repositories create docker-repo \
            --repository-format=docker \
            --location=us-central1 \
            --description="Docker repository"
    
  6. Register gcloud as the credential helper to Docker's configuration file. If prompted, confirm the file update.

    gcloud auth configure-docker us-central1-docker.pkg.dev
    
  7. Push the image to the Artifact Registry:

    docker push us-central1-docker.pkg.dev/$GCP_PROJECT_ID/docker-repo/k8s-cost-estimator:v0.0.1
    

Creating a new GitHub repository

  1. In Cloud Shell, change directories to the GitHub example:

    cd samples/k8s-cost-estimator-github
    
  2. In GitHub, create an access token:

    Go to the GitHub Personal Access Tokens page

    1. In the Note field, enter a token description.
    2. Under Select scopes, select the repo, admin:public_key, and delete_repo checkboxes.
    3. Click Generate token and copy the Your new personal access token value at the top of the page.
  3. In Cloud Shell, save your personal access token in a variable.

    GITHUB_TOKEN=YOUR_NEW_PERSONAL_ACCESS_TOKEN
    

    Replace the following:

    • YOUR_NEW_PERSONAL_ACCESS_TOKEN: the personal access token that you just created.
  4. Create a GitHub repository:

    curl -X POST \
      -H "Accept: application/vnd.github.v3+json" \
      -H "Authorization: Bearer $GITHUB_TOKEN" \
      https://api.github.com/user/repos \
      -d '{"name":"k8s-cost-estimator-github"}' | jq
    

    The output is similar to the following:

    {
      "id": 36099474,
      "node_id": "MDEwOldfsdjA5OTQ3Njc=",
      "name": "k8s-cost-estimator-github",
      ...
    }
    
  5. Add the FinOps reviewer as a collaborator in your repository:

    curl -X PUT \
      -H "Accept: application/vnd.github.v3+json" \
      -H "Authorization: Bearer $GITHUB_TOKEN" \
      https://api.github.com/repos/$GITHUB_USER/k8s-cost-estimator-github/collaborators/$GITHUB_FINOPS_REVIEWER_USER  | jq -r .html_url
    

    The output is similar to the following:

    https://github.com/your-user/k8s-cost-estimator-github/invitations
    
  6. Share the output URL with the user you set in the GITHUB_FINOPS_REVIEWER_USER variable so that they can accept the invitation. Before continuing to the next step, visit the same URL to make sure the invitation was accepted.

    Verify that the invitation was accepted.

Connecting Cloud Build to your GitHub repository

This section shows you how to install the Cloud Build GitHub app. This installation lets you connect your GitHub repository with your Cloud project so that Cloud Build can automatically run the Kubernetes estimator tool at each pull request.

  1. Go to the GitHub Marketplace page for the Cloud Build app:

    Open the Cloud Build app page

  2. Set up GitHub account access for the app:

    1. If this is your first time configuring an app in GitHub, click Setup with Google Cloud Build at the bottom of the page, and then click Grant this app access to your GitHub account.
    2. If you've previously set up an app in GitHub, click Configure access.
  3. In the Applications page that opens, follow these steps:

    1. In the Google Cloud Build row, click Configure.
    2. Select the Only select repositories option.
    3. Select k8s-cost-estimator-github to connect to the repository that you just created.
    4. Click Save—or Install (the button label changes depending on the flow you are executing).
  4. You are now redirected to Google Cloud to continue the installation. Sign in with your Google Cloud account. If prompted, authorize Cloud Build integration with GitHub.

  5. In the Cloud Build page, select your project. A wizard appears.

  6. In the Select repository section of the wizard, select your GitHub account and the k8s-cost-estimator-github repository.

  7. If you agree with the terms and conditions, select the checkbox, and then click Connect.

  8. In the Create a trigger section, click Create a trigger, and then follow these steps:

    1. Enter a trigger name.
    2. In the Event section, select Pull request (GitHub App only).
    3. In the Source section:
      • Make sure the Repository field is auto-filled with your-github-user/k8s-cost-estimator-github (GitHub App).
      • In the Base Branch dropdown, select .*.
    4. In the Configuration Type section, select Cloud Build configuration file (yaml or json).
    5. In the Advanced section, add the following substitution variables:

      • _GITHUB_TOKEN = YOUR_PERSONAL_ACCESS_TOKEN
      • _GITHUB_FINOPS_REVIEWER_USER = THE_GITHUB_FINOPS_REVIEWER_USER
      • _GITHUB_FINOPS_COST_USD_THRESHOLD = 10

      Replace the following:

      • YOUR_PERSONAL_ACCESS_TOKEN: the GitHub personal access token that you created. This token is available in the GITHUB_TOKEN variable in Cloud Shell.
      • THE_GITHUB_FINOPS_REVIEWER_USER: the user that you invited as a collaborator in your GitHub repository. This username is available in the GITHUB_FINOPS_REVIEWER_USER variable in Cloud Shell.
  9. Click Create.

The Cloud Build GitHub app is now configured, and your GitHub repository is linked to your Cloud project. Pull requests to your GitHub repository now trigger Cloud Build executions, which report the results back to GitHub using GitHub Checks.

Pushing the example code to your GitHub repository

  1. Create an SSH keypair to allow you to push the sample code to your GitHub repository:

    mkdir -p ssh && cd ssh
    ssh-keygen -t rsa -b 4096 -N '' -f github-key
    eval `ssh-agent` && ssh-add $(pwd)/github-key
    curl -X POST \
      -H "Accept: application/vnd.github.v3+json" \
      -H "Authorization: Bearer $GITHUB_TOKEN" \
      https://api.github.com/user/keys \
      -d "{\"title\":\"k8s-cost-estimator-key\", \"key\":\"$(cat github-key.pub)\"}" | jq
    cd ..
    

    The output is similar to the following:

    {
      "id": 52356205,
      "key": "ssh-rsa AAAAB3NzaC….wJICyt0yvWjGFZGCWBPUw==",
      "url": "https://api.github.com/user/keys/526205",
      "title": "k8s-cost-estimator-key",
      "verified": true,
      "created_at": "2021-04-23T16:22:58Z",
      "read_only": false
    }
    
  2. Push the content to your new GitHub repository:

    sed "s/GCP_PROJECT_ID/$GCP_PROJECT_ID/g; s/GITHUB_USER/$GITHUB_USER/g; s/GITHUB_EMAIL/$GITHUB_EMAIL/g;" templates/cloudbuild.yaml.tpl > cloudbuild.yaml
    
    GITHUB_SSH_URL_REPO=$(curl -X GET \
      -H "Accept: application/vnd.github.v3+json" \
      -H "Authorization: Bearer $GITHUB_TOKEN" \
      https://api.github.com/repos/$GITHUB_USER/k8s-cost-estimator-github | jq -r .ssh_url)
    [ -z "$GITHUB_SSH_URL_REPO" ] && echo "GITHUB_SSH_URL_REPO is not exported" || echo "GITHUB_SSH_URL_REPO is $GITHUB_SSH_URL_REPO"
    
    git init
    git remote add origin $GITHUB_SSH_URL_REPO
    git add -A .
    git commit -m "Initial commit"
    git checkout -b main
    git push -u origin main
    

Changing the code and proposing a pull request to test the cost estimation

  1. In Cloud Shell, get the GitHub URL to the file wordpress/wordpress_hpa.yaml:

    echo "https://github.com/$GITHUB_USER/k8s-cost-estimator-github/edit/main/wordpress/wordpress_hpa.yaml"
    
  2. Ctrl+click (Cmd+click for Mac users) the output URL to navigate to GitHub and edit the wordpress/wordpress_hpa.yaml file.

  3. In GitHub, change minReplicas to 5.

  4. Select Create a new branch for this commit and start a pull request, and then click Propose Changes.

  5. In the Open a pull request screen, click Create pull request.

    Beyond creating a new pull request, this step triggers a Cloud Build execution based on the cloudbuild.yaml file that you created previously. This Cloud Build execution uses the container image that you built in Building and pushing the Kubernetes cost-estimator image, and it makes the decision when a FinOps reviewer is required.

  6. Wait a minute or so for the pipeline to finish. When it finishes, a comment with cost details is added in the pull request, and because the increase in cost of the code you are proposing exceeded the $10 threshold, a FinOps reviewer is also requested.

    The output is similar to the following:

    A detailed breakdown of costs is added to the pull request.

You now know how to give your developers visibility into their spending early in the development cycle. This setup helps you and your organization avoid surprises in your Google Cloud bill.

Clean up

To avoid incurring charges to your Google Cloud account for the resources used in this tutorial, you can delete your project.

Delete the project

  1. In the Cloud Console, go to the Manage resources page.

    Go to Manage resources

  2. In the project list, select the project that you want to delete, and then click Delete.
  3. In the dialog, type the project ID, and then click Shut down to delete the project.

Delete the GitHub repository

If you don't want to keep your GitHub repository, follow these steps:

  1. In Cloud Shell, delete your GitHub repository:

    curl -X DELETE \
      -H "Accept: application/vnd.github.v3+json" \
      -H "Authorization: Bearer $GITHUB_TOKEN" \
      https://api.github.com/repos/$GITHUB_USER/k8s-cost-estimator-github
    

    If you lose your connection with Cloud Shell, you need to reset the GITHUB_TOKEN and GITHUB_USER variables.

What's next