Estimate your GKE costs early in the development cycle using GitLab

Autopilot Standard

---

This tutorial demonstrates the best practice of shifting Google Kubernetes Engine (GKE) cost visibility to your development team using GitLab. Creating awareness of costs early in the development process helps you to avoid surprises in your Google Cloud bill. Moving a task or information to an earlier part of a process is sometimes referred to as shifting left.

This tutorial is intended for developers, operators, and FinOps practitioners who want to optimize costs in GKE clusters and who use GitLab in production. If you use GitHub instead, see Estimate your GKE costs early in the development cycle using GitHub.

The tutorial assumes that you are familiar with the following technologies:

• Docker
• GitLab
• Kubernetes
• Cloud Build
• GKE
• Linux

Overview

Many teams embracing the public cloud are not used to the pay-as-you-go billing style. Frequently, they don't fully understand the environment their apps are running on—in this case, GKE. The FinOps operating model promotes the culture of financial accountability. A FinOps best practice is to provide teams with real-time information about their spending so that cost issues can be addressed when they arise.

This document shows you how to go one step further by estimating cost before it becomes an expense in your bill. The best time to estimate costs is early in the process during development and at code review time. This way, practitioners can understand and discuss alternatives for the cost impact of new features and bug fixes before it becomes a problem. The following diagram summarizes such a practice.

As the diagram shows, developers can estimate GKE costs in their local environment, ideally at build time. This estimate gives them a good understanding of the monthly production workload cost. When the feature or bug fix is code complete, they can propose a merge request that triggers a GitLab CI/CD pipeline to check the difference between the old and the new cost. If there are increases above a predefined threshold, the pipeline automatically requests a new code review. This practice helps developers become more aware of their workload capacity and proactively fix application issues instead of adding more resources each time an instability is found in production.

Note: if you don't have a code review process in place and you don't plan to start one in your organization, you can still implement a similar approach to the one suggested in this document in your continuous integration pipeline.

Objectives

• Build and push a Kubernetes cost estimator image.
• Create a new GitLab project.
• Configure the GitLab runner to run on a GKE cluster.
• Push the example code to your GitLab repository.
• Change the code and propose a merge request to see the cost estimation in action.

Costs

In this document, you use the following billable components of Google Cloud:

• Artifact Registry
• Cloud Build

To generate a cost estimate based on your projected usage, use the pricing calculator. New Google Cloud users might be eligible for a free trial.

When you finish the tasks that are described in this document, you can avoid continued billing by deleting the resources that you created. For more information, see Clean up.

Before you begin

1. In the Google Cloud console, go to the project selector page.

   Go to project selector

2. Select or create a Google Cloud project.

   Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

3. Make sure that billing is enabled for your Google Cloud project.

4. In the Google Cloud console, activate Cloud Shell.

   Activate Cloud Shell

   At the bottom of the Google Cloud console, a Cloud Shell session starts and displays a command-line prompt. Cloud Shell is a shell environment with the Google Cloud CLI already installed and with values already set for your current project. It can take a few seconds for the session to initialize.

Prepare your environment

1. In Cloud Shell, clone the gke-shift-left-cost GitHub repository:

   git clone https://github.com/GoogleCloudPlatform/kubernetes-engine-samples
   cd kubernetes-engine-samples/cost-optimization/gke-shift-left-cost/
   

   The code in this repository is structured into the following folders:

   • Root: Contains a Dockerfile file that's used to build the cost estimator image and the main.go file that implements the command-line logic for the cost estimator.
   • api/: Contains the Go API for manipulating Kubernetes objects and making the cost estimation.
   • samples/: Contains examples of Kubernetes manifests so that you can experiment with the process before implementing it in your organization.

2. Set your Google Cloud project ID and your GitLab user account and email address:

   export GCP_PROJECT_ID=YOUR_PROJECT_ID
   export GITLAB_USER=YOUR_GITLAB_USER
   export GITLAB_EMAIL=YOUR_GITLAB_EMAIL_ADDRESS
   
   gcloud config set project $GCP_PROJECT_ID
   
   gcloud services enable cloudbilling.googleapis.com \
       compute.googleapis.com \
       container.googleapis.com \
       iamcredentials.googleapis.com \
       artifactregistry.googleapis.com
   
   gcloud config set compute/region us-central1
   gcloud config set compute/zone us-central1-f
   

   Replace the following:

• YOUR_PROJECT_ID : the Google Cloud project ID for the project that you're using in this tutorial.
• YOUR_GITLAB_USER : the user account that you use to log in to your GitLab account.
• YOUR_GITLAB_EMAIL_ADDRESS: the email that you use in your GitLab account.

Optionally, you can use a different region and zone for this tutorial.

Build and push the Kubernetes cost estimator image

The Kubernetes cost estimator tool that comes with this tutorial is an example of what can be done. It offers the capability of estimating cost for DaemonSet, Deployment, StatefulSet, ReplicaSet, HorizontalPodAutoScaler, and PersistentVolumeClaim Kubernetes objects. You can also implement your own cost-estimation tool or propose pull requests with the improvements that you want.

1. In Cloud Shell, allow application-default to use your credentials:

   gcloud auth application-default login
   

2. Build the Kubernetes cost estimator binary:

   mkdir ./bin
   go test ./api
   go build -v -o ./bin/k8s-cost-estimator .
   

3. Test the binary by executing cost estimation in a sample folder:

   ./bin/k8s-cost-estimator \
       --k8s ./samples/k8s-cost-estimator-local/app-v1  \
       --config ./samples/k8s-cost-estimator-local/example-conf.yaml \
       --v trace
   

   In the output, you see a Markdown table that details the monthly estimated costs for the ./samples/k8s-cost-estimator-local/app-v1/ folder. To better understand the monthly production cost of their applications, developers can run this step before pushing code to the remote repository.

   INFO[0000] Starting cost estimation (version v0.0.1)...
   ...
   
   |         KIND          | MIN REQUESTED (USD) | MIN REQ + HPA CPU BUFFER (USD) | MAX REQUESTED (USD) | MIN LIMITED (USD) | MAX LIMITED (USD) |
   |-----------------------|---------------------|--------------------------------|---------------------|-------------------|-------------------|
   | Deployment            |             $133.31 |                        $198.71 |             $266.54 |           $312.83 |           $579.29 |
   | StatefulSet           |              $36.33 |                         $36.33 |              $36.33 |            $72.67 |            $72.67 |
   | DaemonSet             |              $29.68 |                         $29.68 |              $29.68 |            $53.19 |            $53.19 |
   | PersistentVolumeClaim |              $28.88 |                         $28.88 |              $28.88 |            $33.68 |            $33.68 |
   | **TOTAL**             |         **$228.20** |                    **$293.60** |         **$361.43** |       **$472.38** |       **$738.83** |
   
   INFO[0002] Finished cost estimation!
   

4. Build the Kubernetes cost estimator container image:

   docker build . -t \
   us-central1-docker.pkg.dev/$GCP_PROJECT_ID/docker-repo/k8s-cost-estimator:v0.0.1
   

5. Create the Artifact Registry Docker repository to store the image:

   gcloud artifacts repositories create docker-repo \
           --repository-format=docker \
           --location=us-central1 \
           --description="Docker repository"
   

6. Register gcloud as the credential helper to the Docker configuration file.

   gcloud auth configure-docker us-central1-docker.pkg.dev
   

   If prompted, confirm the file update.

7. Push the image to Artifact Registry:

   docker push us-central1-docker.pkg.dev/$GCP_PROJECT_ID/docker-repo/k8s-cost-estimator:v0.0.1
   

Create a new GitLab project

1. In Cloud Shell, change the directory to the GitLab example:

   cd samples/k8s-cost-estimator-gitlab
   

2. On the GitLab Personal access tokens page, create an access token:

   Go to the GitLab personal access tokens page

   1. In the Name field, enter the name for the token you're creating.
   2. In the Scope field, select api, and then click Create personal access token.
   3. Copy the Your new personal access token value.

3. In Cloud Shell, save your personal access token in a variable:

   GITLAB_API_TOKEN=YOUR_NEW_PERSONAL_ACCESS_TOKEN
   

   Replace YOUR_NEW_PERSONAL_ACCESS_TOKEN with the GitLab personal access token that you created.

4. Create a new GitLab project:

   GITLAB_PROJECT_OUTPUT=$(curl -X POST -H "content-type:application/json" -H "PRIVATE-TOKEN:$GITLAB_API_TOKEN" -d '{"name":"k8s-cost-estimator-gitlab","visibility":"public"}'        https://gitlab.com/api/v4/projects)
   GITLAB_PROJECT_ID=$(echo $GITLAB_PROJECT_OUTPUT | jq ".id")
   GITLAB_FINOPS_REVIEWER_ID=$(echo $GITLAB_PROJECT_OUTPUT | jq ".owner.id")
   

5. Set the variables for the cost estimator tool to use when a merge request is created:

   curl -X POST -H "content-type:application/json" -H "PRIVATE-TOKEN:$GITLAB_API_TOKEN" -d "{\"key\": \"GITLAB_API_TOKEN\",\"value\": \"$GITLAB_API_TOKEN\", \"masked\":\"true\"}" https://gitlab.com/api/v4/projects/$GITLAB_PROJECT_ID/variables
   
   curl -X POST -H "content-type:application/json" -H "PRIVATE-TOKEN:$GITLAB_API_TOKEN" -d "{\"key\": \"GITLAB_FINOPS_REVIEWER_ID\",\"value\": \"$GITLAB_FINOPS_REVIEWER_ID\"}" https://gitlab.com/api/v4/projects/$GITLAB_PROJECT_ID/variables
   
   curl -X POST -H "content-type:application/json" -H "PRIVATE-TOKEN:$GITLAB_API_TOKEN" -d "{\"key\": \"GITLAB_FINOPS_COST_USD_THRESHOLD\",\"value\": \"10\"}" https://gitlab.com/api/v4/projects/$GITLAB_PROJECT_ID/variables
   

6. Confirm that your project and variables were created:

   curl -s --header "PRIVATE-TOKEN:$GITLAB_API_TOKEN" \
   https://gitlab.com/api/v4/projects/$GITLAB_PROJECT_ID/variables | jq
   

   The output is similar to the following:

   [
       {
       "variable_type": "env_var",
       "key": "GITLAB_API_TOKEN",
       "value": "Ex...n1",
       "protected": false,
       "masked": true,
       "environment_scope": "*"
       },
       {
       "variable_type": "env_var",
       "key": "GITLAB_FINOPS_REVIEWER_ID",
       "value": "88..87",
       "protected": false,
       "masked": false,
       "environment_scope": "*"
       },
       {
       "variable_type": "env_var",
       "key": "GITLAB_FINOPS_COST_USD_THRESHOLD",
       "value": "10",
       "protected": false,
       "masked": false,
       "environment_scope": "*"
       }
   ]
   

   The variables configured in your GitLab project are used by the ./samples/k8s-cost-estimator-gitlab/templates/.gitlab-ci.yml.tpl file to update merge requests and are as follows:

   • GITLAB_API_TOKEN: Your GitLab personal access token.
   • GITLAB_FINOPS_REVIEWER_ID: The code reviewer required whenever the cost increases above a given threshold. For simplicity, this tutorial sets your own user ID as the reviewer. However, in a production environment, we recommend that you configure a team instead of an individual person.
   • GITLAB_FINOPS_COST_USD_THRESHOLD: The threshold in USD, in this case $10. When the difference between old and new costs passes this threshold, an extraordinary approval is enforced. You can also set thresholds for other values. To explore this feature, you can append the --output parameter when you execute the ./bin/k8s-cost-estimator command in Building and pushing the Kubernetes cost estimator image. This parameter generates a file with a .diff extension that lets you view the available options.

Configure GitLab runner to run on a GKE cluster

In this section, you install the GitLab runner in your own GKE cluster with Workload Identity Federation for GKE to allow the Kubernetes estimator tool to query Google Cloud Price Catalog. The estimator uses gross prices and doesn't take into account preemptible VMs or any discounts.

1. In Cloud Shell, create a GKE cluster:

   gcloud beta container clusters create gitlab-runners \
       --enable-ip-alias \
       --release-channel=stable \
       --workload-pool=$GCP_PROJECT_ID.svc.id.goog \
       --enable-autoprovisioning --min-cpu 1 --min-memory 1 --max-cpu 4 --max-memory 16 \
       --autoscaling-profile=optimize-utilization \
       --preemptible
   

2. Get the GitLab runner registration token from the project that you created:

   export GITLAB_RUNNER_TOKEN=$(curl -s --header "PRIVATE-TOKEN:$GITLAB_API_TOKEN" https://gitlab.com/api/v4/projects/$GITLAB_PROJECT_ID | jq -r '.runners_token')
   [ -z "$GITLAB_RUNNER_TOKEN" ] && echo "GITLAB_RUNNER_TOKEN is not exported" || echo "GITLAB_RUNNER_TOKEN is $GITLAB_RUNNER_TOKEN"
   

3. Install the GitLab runner in your GKE cluster:

   kubectl create namespace gitlab
   
   helm repo add gitlab https://charts.gitlab.io
   
   sed "s/GCP_PROJECT_ID/$GCP_PROJECT_ID/g; s/GITLAB_RUNNER_TOKEN/$GITLAB_RUNNER_TOKEN/g" templates/gitlab-runner-values.yaml.tpl > gitlab-runner-values.yaml
   
   helm install --namespace gitlab --version 0.24.0 gitlab-runner -f gitlab-runner-values.yaml gitlab/gitlab-runner
   
   kubectl -n gitlab wait --for=condition=available deployment gitlab-runner --timeout=5m
   
   gcloud iam service-accounts create gitlab-runner --display-name=gitlab-runner
   gcloud iam service-accounts add-iam-policy-binding \
       --role roles/iam.workloadIdentityUser \
       --member "serviceAccount:$GCP_PROJECT_ID.svc.id.goog[gitlab/gitlab-runner]" \
       gitlab-runner@$GCP_PROJECT_ID.iam.gserviceaccount.com
   

4. Disable shared runners in your GitLab project:

   curl -s --header "PRIVATE-TOKEN: ${GITLAB_API_TOKEN}" -X PUT "https://gitlab.com/api/v4/projects/$GITLAB_PROJECT_ID" --form "shared_runners_enabled=false"
   

5. Verify that the runner that you deployed is enabled in your GitLab project:

   curl -s --header "PRIVATE-TOKEN: ${GITLAB_API_TOKEN}" "https://gitlab.com/api/v4/projects/$GITLAB_PROJECT_ID/runners?status=active" | jq '.[] | select(.is_shared==false)'
   

   The output is similar to the following:

   {
       "id": 49345561,
       "description": "gitlab-runner-gitlab-runner-788459d488-jlscn",
       "ip_address": "35.178.223.199",
       "active": true,
       "is_shared": false,
       "name": "gitlab-runner",
       "online": true,
       "status": "online"
   }
   

Push the example code to your GitLab repository

1. Create an SSH key pair to push the sample code to your GitLab repository:

   mkdir -p ssh && cd ssh
   ssh-keygen -t rsa -b 4096 -N '' -f gitlab-key
   eval `ssh-agent` && ssh-add $(pwd)/gitlab-key
   curl -s --request POST --header "PRIVATE-TOKEN:$GITLAB_API_TOKEN" https://gitlab.com/api/v4/user/keys --form "title=k8s-cost-estimator-key" --form "key=$(cat gitlab-key.pub)"
   cd ..
   

2. Push the content to your new GitLab repository:

   sed "s/GCP_PROJECT_ID/$GCP_PROJECT_ID/g; s/GITLAB_USER/$GITLAB_USER/g; s/GITLAB_EMAIL/$GITLAB_EMAIL/g;" templates/.gitlab-ci.yml.tpl > .gitlab-ci.yml
   
   GITLAB_SSH_URL_REPO=$(curl -s --header "PRIVATE-TOKEN:$GITLAB_API_TOKEN" https://gitlab.com/api/v4/users/$GITLAB_FINOPS_REVIEWER_ID/projects | jq '.[] | select(.name=="k8s-cost-estimator-gitlab")' | jq -r '.ssh_url_to_repo')
   [ -z "$GITLAB_SSH_URL_REPO" ] && echo "GITLAB_PROJECT_SSH_URL is not exported" || echo "GITLAB_PROJECT_SSH_URL is $GITLAB_SSH_URL_REPO"
   
   git config --global user.email $GITLAB_EMAIL
   git config --global user.name $GITLAB_USER
   git init
   git remote add origin $GITLAB_SSH_URL_REPO
   git add -A .
   git commit -m "Initial commit"
   git checkout -b main
   git push -u origin main
   

Change the code and propose a merge request to see the cost estimation in action

1. In Cloud Shell, get the URL for the GitLab Web Integrated Development Environment (IDE):

   echo "https://gitlab.com/-/ide/project/$GITLAB_USER/k8s-cost-estimator-gitlab/tree/main/-/wordpress/wordpress_hpa.yaml"
   

2. Ctrl + click (Cmd + click on macOS) the output URL to navigate to the GitLab Web IDE.

3. In the GitLab Web IDE, edit the ./wordpress/wordpress_hpa.yaml file as follows:

   1. Change the minReplicas value from 2 to 5.
   2. Click Commit.

4. As shown in the following screenshot, select Create a new branch and Start a new merge request, and then click Commit.

   

5. On the New merge request screen, click Create merge request at the bottom of the page.

   In addition to creating a new merge request, this step triggers a cost estimation pipeline based on the .gitlab-ci.yml file. This pipeline uses the container image that you created in a previous section. This pipeline also determines when a FinOps approval is required. For simplicity, .gitlab-ci.yml adds approvals for each merge request base, but you can define and reuse approvals rules defined at GitLab project level.

6. Wait about a minute for the pipeline to finish. When it finishes, a comment with cost details is added in the merge request. Because the increase in cost of the code that you're proposing exceeds the $10 threshold, a FinOps reviewer is also requested.

   The output is similar to the following:

   

   In this tutorial, the default configuration is used for merge request approvals. It's possible to select different configurations for your merge requests in Gitlab. For example, to prevent the author from approving merge requests, you go to Settings > General > Merge request (MR) approvals > Approval settings.

Clean up

To avoid incurring charges to your Google Cloud account for the resources used in this tutorial, you can delete your project.

Delete the project

Caution: Deleting a project has the following effects:

• Everything in the project is deleted. If you used an existing project for the tasks in this document, when you delete it, you also delete any other work you've done in the project.
• Custom project IDs are lost. When you created this project, you might have created a custom project ID that you want to use in the future. To preserve the URLs that use the project ID, such as an appspot.com URL, delete selected resources inside the project instead of deleting the whole project.

If you plan to explore multiple architectures, tutorials, or quickstarts, reusing projects can help you avoid exceeding project quota limits.

1. In the Google Cloud console, go to the Manage resources page.

   Go to Manage resources

2. In the project list, select the project that you want to delete, and then click Delete.
3. In the dialog, type the project ID, and then click Shut down to delete the project.

Delete the GitLab project

If you don't want to keep your GitLab project, do the following:

1. In Cloud Shell, delete your GitLab project:

    curl -X DELETE -H "content-type:application/json" -H "PRIVATE-TOKEN:$GITLAB_API_TOKEN" https://gitlab.com/api/v4/projects/$GITLAB_PROJECT_ID
   ```
   
    The output is similar to the following:
   
    ```none {:.devsite-disable-click-to-copy}
    {"message":"202 Accepted"}
   

   If you lose your connection with Cloud Shell, you must set the following variables again:

   • GITLAB_API_TOKEN
   • GITLAB_PROJECT_ID

What's next

• Learn more about GKE cost optimization in Best practices for running cost-optimized Kubernetes applications on GKE.
• Read about how to estimate your GKE costs early in the development cycle using GitHub.
• Find design recommendations and best practices to optimize the cost of Google Cloud workloads in Google Cloud Architecture Framework: Cost optimization.
• Explore reference architectures, diagrams, and best practices about Google Cloud. Take a look at our Cloud Architecture Center.