Estimate your GKE costs early in the development cycle using GitLab

This tutorial demonstrates the best practice of shifting Google Kubernetes Engine (GKE) cost visibility to your development team using GitLab. Creating awareness of costs early in the development process helps you to avoid surprises in your Google Cloud bill. Moving a task or information to an earlier part of a process is sometimes referred to as shifting left.

This tutorial is intended for developers, operators, and FinOps practitioners who want to optimize costs in GKE clusters and who use GitLab in production. If you use GitHub instead, see Estimate your GKE costs early in the development cycle using GitHub.

The tutorial assumes that you are familiar with the following technologies:

Overview

Many teams embracing the public cloud are not used to the pay-as-you-go billing style. Frequently, they don't fully understand the environment their apps are running on—in this case, GKE. The FinOps operating model promotes the culture of financial accountability. A FinOps best practice is to provide teams with real-time information about their spending so that cost issues can be addressed when they arise.

This document shows you how to go one step further by estimating cost before it becomes an expense in your bill. As highlighted on the GitLab website, "Code review is an essential practice of every successful project, and giving your approval once a merge request is in good shape is an important part of the review process." Therefore, the best time to estimate costs is early in the process during development and at code review time. This way, practitioners can understand and discuss alternatives for the cost impact of new features and bug fixes before it becomes a problem. The following diagram summarizes such a practice.

Best practice of estimating cost early.

As the diagram shows, developers can estimate GKE costs in their local environment, ideally at build time. This estimate gives them a good understanding of the monthly production workload cost. When the feature or bug fix is code complete, they can propose a merge request that triggers a GitLab CI/CD pipeline to check the difference between the old and the new cost. If there are increases above a predefined threshold, the pipeline automatically requests a new code review. This practice helps developers become more aware of their workload capacity and proactively fix application issues instead of adding more resources each time an instability is found in production.

Objectives

  • Build and push a Kubernetes cost estimator image.
  • Create a new GitLab project.
  • Configure the GitLab runner to run on a GKE cluster.
  • Push the example code to your GitLab repository.
  • Change the code and propose a merge request to see the cost estimation in action.

Costs

This tutorial uses the following billable components of Google Cloud:

To generate a cost estimate based on your projected usage, use the pricing calculator. New Google Cloud users might be eligible for a free trial.

When you finish this tutorial, you can avoid continued billing by deleting the resources you created. For more information, see Clean up.

Before you begin

  1. In the Google Cloud Console, go to the project selector page.

    Go to project selector

  2. Select or create a Google Cloud project.

  3. Make sure that billing is enabled for your Cloud project. Learn how to confirm that billing is enabled for your project.

  4. In the Cloud Console, activate Cloud Shell.

    Activate Cloud Shell

    At the bottom of the Cloud Console, a Cloud Shell session starts and displays a command-line prompt. Cloud Shell is a shell environment with the Cloud SDK already installed, including the gcloud command-line tool, and with values already set for your current project. It can take a few seconds for the session to initialize.

Preparing your environment

  1. In Cloud Shell, clone the gke-shift-left-cost GitHub repository:

    git clone https://github.com/GoogleCloudPlatform/gke-shift-left-cost
    cd gke-shift-left-cost
    

    The code in this repository is structured into the following folders:

    • Root: Contains a Dockerfile file that's used to build the cost estimator image and the main.go file that implements the command-line logic for the cost estimator.
    • api/: Contains the Go API for manipulating Kubernetes objects and making the cost estimation.
    • samples/: Contains examples of Kubernetes manifests so that you can experiment with the process before implementing it in your organization.
  2. Set your Cloud project ID and your GitLab user account and email address:

    export GCP_PROJECT_ID=YOUR_PROJECT_ID
    export GITLAB_USER=YOUR_GITLAB_USER
    export GITLAB_EMAIL=YOUR_GITLAB_EMAIL_ADDRESS
    
    gcloud config set project $GCP_PROJECT_ID
    
    gcloud services enable cloudbilling.googleapis.com \
        compute.googleapis.com \
        container.googleapis.com \
        iamcredentials.googleapis.com \
        artifactregistry.googleapis.com
    
    gcloud config set compute/region us-central1
    gcloud config set compute/zone us-central1-f
    

    Replace the following:

  • YOUR_PROJECT_ID : the Cloud project ID for the project that you're using in this tutorial.
  • YOUR_GITLAB_USER : the user account that you use to log in to your GitLab account.
  • YOUR_GITLAB_EMAIL_ADDRESS: the email that you use in your GitLab account.

Optionally, you can use a different region and zone for this tutorial.

Building and pushing the Kubernetes cost estimator image

The Kubernetes cost estimator tool that comes with this tutorial is an example of what can be done. It offers the capability of estimating cost for DaemonSet, Deployment, StatefulSet, ReplicaSet, HorizontalPodAutoScaler, and PersistentVolumeClaim Kubernetes objects. You can also implement your own cost-estimation tool or propose pull requests with the improvements that you want.

  1. In Cloud Shell, allow application-default to use your credentials:

    gcloud auth application-default login
    
  2. Build the Kubernetes cost estimator binary:

    mkdir ./bin
    go test ./api
    go build -v -o ./bin/k8s-cost-estimator .
    
  3. Test the binary by executing cost estimation in a sample folder:

    ./bin/k8s-cost-estimator \
        --k8s ./samples/k8s-cost-estimator-local/app-v1  \
        --config ./samples/k8s-cost-estimator-local/example-conf.yaml \
        --v trace
    

    In the output, you see a Markdown table that details the monthly estimated costs for the ./samples/k8s-cost-estimator-local/app-v1/ folder. To better understand the monthly production cost of their applications, developers can run this step before pushing code to the remote repository.

    INFO[0000] Starting cost estimation (version v0.0.1)...
    ...
    
    |         KIND          | MIN REQUESTED (USD) | MIN REQ + HPA CPU BUFFER (USD) | MAX REQUESTED (USD) | MIN LIMITED (USD) | MAX LIMITED (USD) |
    |-----------------------|---------------------|--------------------------------|---------------------|-------------------|-------------------|
    | Deployment            |             $133.31 |                        $198.71 |             $266.54 |           $312.83 |           $579.29 |
    | StatefulSet           |              $36.33 |                         $36.33 |              $36.33 |            $72.67 |            $72.67 |
    | DaemonSet             |              $29.68 |                         $29.68 |              $29.68 |            $53.19 |            $53.19 |
    | PersistentVolumeClaim |              $28.88 |                         $28.88 |              $28.88 |            $33.68 |            $33.68 |
    | **TOTAL**             |         **$228.20** |                    **$293.60** |         **$361.43** |       **$472.38** |       **$738.83** |
    
    INFO[0002] Finished cost estimation!
    
  4. Build the Kubernetes cost estimator container image:

    docker build . -t \
    us-central1-docker.pkg.dev/$GCP_PROJECT_ID/docker-repo/k8s-cost-estimator:v0.0.1
    
  5. Create the Artifact Registry Docker repository to store the image:

    gcloud artifacts repositories create docker-repo \
            --repository-format=docker \
            --location=us-central1 \
            --description="Docker repository"
    
  6. Register gcloud as the credential helper to the Docker configuration file.

    gcloud auth configure-docker us-central1-docker.pkg.dev
    

    If prompted, confirm the file update.

  7. Push the image to Artifact Registry:

    docker push us-central1-docker.pkg.dev/$GCP_PROJECT_ID/docker-repo/k8s-cost-estimator:v0.0.1
    

Creating a new GitLab project

  1. In Cloud Shell, change the directory to the GitLab example:

    cd samples/k8s-cost-estimator-gitlab
    
  2. On the GitLab Personal access tokens page, create an access token:

    Go to the GitLab personal access tokens page

    1. In the Name field, enter the name for the token you're creating.
    2. In the Scope field, select api, and then click Create personal access token.
    3. Copy the Your new personal access token value.
  3. In Cloud Shell, save your personal access token in a variable:

    GITLAB_API_TOKEN=YOUR_NEW_PERSONAL_ACCESS_TOKEN
    

    Replace the following:

    • YOUR_NEW_PERSONAL_ACCESS_TOKEN: the GitLab personal access token that you created.
  4. Create a new GitLab project:

    GITLAB_PROJECT_OUTPUT=$(curl -X POST -H "content-type:application/json" -H "PRIVATE-TOKEN:$GITLAB_API_TOKEN" -d '{"name":"k8s-cost-estimator-gitlab","visibility":"public"}'        https://gitlab.com/api/v4/projects)
    GITLAB_PROJECT_ID=$(echo $GITLAB_PROJECT_OUTPUT | jq ".id")
    GITLAB_FINOPS_REVIEWER_ID=$(echo $GITLAB_PROJECT_OUTPUT | jq ".owner.id")
    
  5. Set the variables for the cost estimator tool to use when a merge request is created:

    curl -X POST -H "content-type:application/json" -H "PRIVATE-TOKEN:$GITLAB_API_TOKEN" -d "{\"key\": \"GITLAB_API_TOKEN\",\"value\": \"$GITLAB_API_TOKEN\", \"masked\":\"true\"}" https://gitlab.com/api/v4/projects/$GITLAB_PROJECT_ID/variables
    
    curl -X POST -H "content-type:application/json" -H "PRIVATE-TOKEN:$GITLAB_API_TOKEN" -d "{\"key\": \"GITLAB_FINOPS_REVIEWER_ID\",\"value\": \"$GITLAB_FINOPS_REVIEWER_ID\"}" https://gitlab.com/api/v4/projects/$GITLAB_PROJECT_ID/variables
    
    curl -X POST -H "content-type:application/json" -H "PRIVATE-TOKEN:$GITLAB_API_TOKEN" -d "{\"key\": \"GITLAB_FINOPS_COST_USD_THRESHOLD\",\"value\": \"10\"}" https://gitlab.com/api/v4/projects/$GITLAB_PROJECT_ID/variables
    
  6. Check that your project and variables were created:

    curl -s --header "PRIVATE-TOKEN:$GITLAB_API_TOKEN" \
    https://gitlab.com/api/v4/projects/$GITLAB_PROJECT_ID/variables | jq
    

    The output resembles the following:

    [
      {
        "variable_type": "env_var",
        "key": "GITLAB_API_TOKEN",
        "value": "Ex...n1",
        "protected": false,
        "masked": true,
        "environment_scope": "*"
      },
      {
        "variable_type": "env_var",
        "key": "GITLAB_FINOPS_REVIEWER_ID",
        "value": "88..87",
        "protected": false,
        "masked": false,
        "environment_scope": "*"
      },
      {
        "variable_type": "env_var",
        "key": "GITLAB_FINOPS_COST_USD_THRESHOLD",
        "value": "10",
        "protected": false,
        "masked": false,
        "environment_scope": "*"
      }
    ]
    

    The variables configured in your GitLab project are used by the ./samples/k8s-cost-estimator-gitlab/templates/.gitlab-ci.yml.tpl file to update merge requests and are as follows:

    • GITLAB_API_TOKEN: Your GitLab personal access token.
    • GITLAB_FINOPS_REVIEWER_ID: The code reviewer required whenever the cost increases above a given threshold. For simplicity, this tutorial sets your own user ID as the reviewer. However, in a production environment, we recommend that you configure a team instead of an individual person.
    • GITLAB_FINOPS_COST_USD_THRESHOLD: The threshold in USD, in this case $10. When the difference between old and new costs passes this threshold, an extraordinary approval is enforced. You can also set thresholds for other values. To explore this feature, you can append the --output parameter when you execute the ./bin/k8s-cost-estimator command in Building and pushing the Kubernetes cost estimator image. This parameter generates a file with a .diff extension that lets you view the available options.

Configuring GitLab runner to run on a GKE cluster

In this section, you install the GitLab runner in your own GKE cluster with Workload Identity to allow the Kubernetes estimator tool to query Google Cloud Price Catalog. The estimator uses gross prices and doesn't take into account preemptible VMs or any discounts.

  1. In Cloud Shell, create a GKE cluster:

    gcloud beta container clusters create gitlab-runners \
        --enable-ip-alias \
        --release-channel=stable \
        --workload-pool=$GCP_PROJECT_ID.svc.id.goog \
        --enable-autoprovisioning --min-cpu 1 --min-memory 1 --max-cpu 4 --max-memory 16 \
        --autoscaling-profile=optimize-utilization \
        --preemptible
    
  2. Get the GitLab runner registration token from the project that you created:

    export GITLAB_RUNNER_TOKEN=$(curl -s --header "PRIVATE-TOKEN:$GITLAB_API_TOKEN" https://gitlab.com/api/v4/projects/$GITLAB_PROJECT_ID | jq -r '.runners_token')
    [ -z "$GITLAB_RUNNER_TOKEN" ] && echo "GITLAB_RUNNER_TOKEN is not exported" || echo "GITLAB_RUNNER_TOKEN is $GITLAB_RUNNER_TOKEN"
    
  3. Install the GitLab runner in your GKE cluster:

    kubectl create namespace gitlab
    
    helm repo add gitlab https://charts.gitlab.io
    
    sed "s/GCP_PROJECT_ID/$GCP_PROJECT_ID/g; s/GITLAB_RUNNER_TOKEN/$GITLAB_RUNNER_TOKEN/g" templates/gitlab-runner-values.yaml.tpl > gitlab-runner-values.yaml
    
    helm install --namespace gitlab --version 0.24.0 gitlab-runner -f gitlab-runner-values.yaml gitlab/gitlab-runner
    
    kubectl -n gitlab wait --for=condition=available deployment gitlab-runner --timeout=5m
    
    gcloud iam service-accounts create gitlab-runner --display-name=gitlab-runner
    gcloud iam service-accounts add-iam-policy-binding \
        --role roles/iam.workloadIdentityUser \
        --member "serviceAccount:$GCP_PROJECT_ID.svc.id.goog[gitlab/gitlab-runner]" \
        gitlab-runner@$GCP_PROJECT_ID.iam.gserviceaccount.com
    
  4. Disable shared runners in your GitLab project:

    curl -s --header "PRIVATE-TOKEN: ${GITLAB_API_TOKEN}" -X PUT "https://gitlab.com/api/v4/projects/$GITLAB_PROJECT_ID" --form "shared_runners_enabled=false"
    
  5. Verify that the runner that you deployed is enabled in your GitLab project:

    curl -s --header "PRIVATE-TOKEN: ${GITLAB_API_TOKEN}" "https://gitlab.com/api/v4/projects/$GITLAB_PROJECT_ID/runners?status=active" | jq '.[] | select(.is_shared==false)'
    

    The output resembles the following:

    {
        "id": 49345561,
        "description": "gitlab-runner-gitlab-runner-788459d488-jlscn",
        "ip_address": "35.178.223.199",
        "active": true,
        "is_shared": false,
        "name": "gitlab-runner",
        "online": true,
        "status": "online"
    }
    

Pushing the example code to your GitLab repository

  1. Create an SSH key pair to push the sample code to your GitLab repository:

    mkdir -p ssh && cd ssh
    ssh-keygen -t rsa -b 4096 -N '' -f gitlab-key
    eval `ssh-agent` && ssh-add $(pwd)/gitlab-key
    curl -s --request POST --header "PRIVATE-TOKEN:$GITLAB_API_TOKEN" https://gitlab.com/api/v4/user/keys --form "title=k8s-cost-estimator-key" --form "key=$(cat gitlab-key.pub)"
    cd ..
    
  2. Push the content to your new GitLab repository:

    sed "s/GCP_PROJECT_ID/$GCP_PROJECT_ID/g; s/GITLAB_USER/$GITLAB_USER/g; s/GITLAB_EMAIL/$GITLAB_EMAIL/g;" templates/.gitlab-ci.yml.tpl > .gitlab-ci.yml
    
    GITLAB_SSH_URL_REPO=$(curl -s --header "PRIVATE-TOKEN:$GITLAB_API_TOKEN" https://gitlab.com/api/v4/users/$GITLAB_FINOPS_REVIEWER_ID/projects | jq '.[] | select(.name=="k8s-cost-estimator-gitlab")' | jq -r '.ssh_url_to_repo')
    [ -z "$GITLAB_SSH_URL_REPO" ] && echo "GITLAB_PROJECT_SSH_URL is not exported" || echo "GITLAB_PROJECT_SSH_URL is $GITLAB_SSH_URL_REPO"
    
    git init
    git remote add origin $GITLAB_SSH_URL_REPO
    git add -A .
    git commit -m "Initial commit"
    git checkout -b main
    git push -u origin main
    

Changing the code and proposing a merge request to see the cost estimation in action

  1. In Cloud Shell, get the URL for the GitLab Web Integrated Development Environment (IDE):

    echo "https://gitlab.com/-/ide/project/$GITLAB_USER/k8s-cost-estimator-gitlab/tree/main/-/wordpress/wordpress_hpa.yaml"
    
  2. Ctrl + click (Cmd + click on macOS) the output URL to navigate to the GitLab Web IDE.

  3. In the GitLab Web IDE, edit the ./wordpress/wordpress_hpa.yaml file as follows:

    1. Change the minReplicas value from 2 to 5.
    2. Click Commit.
  4. As shown in the following screenshot, select Create a new branch and Start a new merge request, and then click Commit.

    Start a merge request.

  5. On the New merge request screen, click Create merge request at the bottom of the page.

    In addition to creating a new merge request, this step triggers a cost estimation pipeline based on the .gitlab-ci.yml file. This pipeline uses the container image that you created in a previous section. This pipeline also determines when a FinOps approval is required. For simplicity, .gitlab-ci.yml adds approvals for each merge request base, but you can define and reuse approvals rules defined at GitLab project level.

  6. Wait about a minute for the pipeline to finish. When it finishes, a comment with cost details is added in the merge request. Because the increase in cost of the code that you're proposing exceeds the $10 threshold, a FinOps reviewer is also requested.

    The output resembles the following:

    Comment with cost detail in merge request.

    In this tutorial, the default configuration is used for merge request approvals. It's possible to select different configurations for your merge requests in Gitlab. For example, to prevent the author from approving merge requests, you go to Settings > General > Merge request (MR) approvals > Approval settings.

Clean up

To avoid incurring charges to your Google Cloud account for the resources used in this tutorial, you can delete your project.

Delete the project

  1. In the Cloud Console, go to the Manage resources page.

    Go to Manage resources

  2. In the project list, select the project that you want to delete, and then click Delete.
  3. In the dialog, type the project ID, and then click Shut down to delete the project.

Delete the GitLab project

If you don't want to keep your GitLab project, do the following:

  1. In Cloud Shell, delete your GitLab project:

     curl -X DELETE -H "content-type:application/json" -H "PRIVATE-TOKEN:$GITLAB_API_TOKEN" https://gitlab.com/api/v4/projects/$GITLAB_PROJECT_ID
    

    The output resembles the following: {"message":"202 Accepted"}

    If you lose your connection with Cloud Shell, you must set the following variables again:

    • GITLAB_API_TOKEN
    • GITLAB_PROJECT_ID

What's next