View detailed breakdown of cluster costs

Standard

This page shows you how to view a breakdown of your Google Kubernetes Engine (GKE) cluster costs in Cloud Billing BigQuery export using GKE cost allocation.

You can use this information to distribute the costs of a cluster to its users.

Potential data volume increase

Enabling GKE cost allocation can increase your storage and querying costs in BigQuery. The exact increase depends on the number of distinct label and namespace combinations you use across Pods and clusters.

Enabling GKE cost allocation does not change the total cost of your GKE usage. The sum of the cost line items in your export is the same and existing queries and reports you have built on the billing export return the same values.

GKE cost allocation and cluster usage metering

GKE cost allocation is different from cluster usage metering in the following ways:

GKE cost allocation provides an alternative to cluster usage metering for getting cluster information.
GKE cost allocation calculates cluster costs in your Cloud Billing account instead of aggregating the data in a Looker Studio template.
GKE cost allocation lets you view a clusters cost data in the Cloud Billing Google Cloud console and in the Cloud Billing detailed export.

Before you begin

Before you start, make sure you have performed the following tasks:

Enable the Google Kubernetes Engine API.

Enable Google Kubernetes Engine API

If you want to use the Google Cloud CLI for this task, install and then initialize the gcloud CLI. If you previously installed the gcloud CLI, get the latest version by running gcloud components update.
Note: For existing gcloud CLI installations, make sure to set the compute/region and compute/zone properties. By setting default locations, you can avoid errors in gcloud CLI like the following: One of [--zone, --region] must be supplied: Please specify location.

Ensure that you have completed the steps to Export detailed usage cost data to BigQuery. If your organization is already exporting data, you must have access to query the tables.

Limitations and restrictions

You can only view GKE cost allocation data in the Cloud Billing BigQuery detailed usage cost data export.
You must have Google Cloud CLI version 392.0.0 or later
If you enable GKE cost allocation, your billing export starts including additional line items for your GKE resources from that date. The billing export does not backfill data.
If you disable GKE cost allocation, your billing export stops including additional line items, but does not modify or remove previous line items that were generated while the feature was enabled.
GKE cost allocation data is based on resource requests, not resources consumed. To learn more about the differences between resource requests, resource limits, and resource consumption, see Kubernetes best practices: Resource requests and limits.
GKE cost allocation supports the following resource SKU types:
- Compute Engine VM Instance vCPU SKUs
- Compute Engine VM Instance RAM SKUs
- Compute Engine VM Instance GPU SKUs
- Compute Engine PD Capacity SKUs
- Cloud TPU Instance SKUs
If a Pod has more than 50 Kubernetes labels, none of those labels will be available in the Cloud Billing section of the Google Cloud console or the Cloud Billing detailed export.
After you enable GKE cost allocation, it can take up to three days for data to appear in Cloud Billing.
GKE cost allocation includes data from Persistent Disks with the following exceptions:
- Only supports PersistentVolume resources provisioned dynamically through PersistentVolumeClaims or Generic Ephemeral Volumes.
- Only supports PersistentVolume resources provisioned by the Compute Engine persistent disk CSI driver or with the deprecated kubernetes.io/gce-pd storage plugin with ReadWriteOnce or ReadWriteOncePod access mode.
- Persistent Disks with an uptime of less than 30 minutes might not be tracked. Persistent Disks with at least 30 minutes uptime are always tracked.
The Cloud TPU version must be Cloud TPU v4 or later.

View GKE cluster costs

When you enable GKE cost allocation, the cluster name and namespace of your GKE workloads appear in the labels field of the billing export to BigQuery:

	Resource label key (`labels.key`)	Resource label value (`labels.value`)
Description	Kubernetes labels have the following format: `k8s-label/${k8s-label-key}`, where `${k8s-label-key}` is the key of the Pod's Kubernetes label. The billing label `k8s-label/${k8s-label-key}` has the same value as the corresponding Kubernetes Pod label.	If a Kubernetes label key collides with a VM label key, the Kubernetes label value overrides the VM label value. Cluster and node pool resource labels are applied to resources using the following priority from highest to lowest: Individual resource, such as instance or storage Node pool Cluster
Cluster name	`goog-k8s-cluster-name`	[cluster-name]
Namespace	`k8s-namespace`	[namespace]
Workload type	`k8s-workload-type`	[workload-type] (e.g. `apps/v1-Deployment`,`apps/v1-StatefulSet`,`apps/v1-DaemonSet`,`apps/v1-ReplicaSet`,`batch/v1-Job`,`batch/v1-CronJob`,`core/v1-Pod`)
Workload name	`k8s-workload-name`	[workload-name]

For more information, see Schema of the detailed usage cost data.

Additionally, the following namespaces are used to track overhead and unallocated resources:

kube:system-overhead: This accounts for node resources that are not available to Pods. These resources are set aside by the system for running the Kubernetes framework (including but not limited to kubelet, kube-proxy, containerd). This is the difference between the Node's capacity and allocatable resources.
kube:unallocated: Resources are neither requested by workloads nor requested for system overhead.

If GKE cannot determine the cost allocation for a resource, the export includes one of the following values:

goog-k8s-unknown: Cloud Billing couldn't process the SKU. This can occur when a new Compute Engine instance is being provisioned. You can expect some goog-k8s-unknown values during node startup and shutdown, for example when GKE autoscales a cluster.
goog-k8s-unsupported-sku: GKE cost allocation does not support this SKU. Treat this the same as <blank>/NULL. There is no guarantee that a SKU is always labeled with this value.
<blank>/NULL: GKE cost allocation does not track this resource. This can occur when GKE cost allocation is not enabled, or the resource does not belong to a GKE managed cluster.

Costs for Persistent Disks are tracked when the Persistent Disk's backing PersistentVolumeClaims are claimed by Pods. The Persistent Disk costs inherit all of the metadata, such as Pod labels and namespaces, of the claiming Pod, until their PersistentVolumeClaims are deleted. For PersistentVolumeClaims that are deleted but retain PersistentVolumes according to the reclaim policy, costs of the Persistent Disks are tracked as unallocated.

Labels applied directly to PersistentVolumeClaims do not appear in the Cloud Billing section of the Google Cloud console or the Cloud Billing detailed export.

Enable GKE cost allocation

You can enable GKE cost allocation on a new or existing cluster using the gcloud CLI or the Google Cloud console.

Create a new cluster

gcloud

Create a cluster using the --enable-cost-allocation flag:

gcloud container clusters create CLUSTER_NAME \
    --enable-cost-allocation \
    --region=COMPUTE_REGION

Replace the following:

CLUSTER_NAME: the name of the new cluster.
COMPUTE_REGION: the Compute Engine region for your cluster. For zonal clusters, use --zone=COMPUTE_ZONE.

Console

To enable GKE cost allocation when you create a new cluster, do the following:

Go to the Google Kubernetes Engine page in the Google Cloud console.

Go to Google Kubernetes Engine
Click Create.
Configure your cluster.
From the navigation pane, under Cluster, click Features.
In the Other section, select the Enable Cost Allocation checkbox.
Configure the cluster as needed, and then click Create.

Update a cluster

gcloud

Update a cluster using the --enable-cost-allocation flag:

gcloud container clusters update CLUSTER_NAME \
    --enable-cost-allocation

Replace CLUSTER_NAME with the name of the cluster.

Console

To enable GKE cost allocation on an existing cluster, do the following:

Go to the Google Kubernetes Engine page in the Google Cloud console. Go to Google Kubernetes Engine
Click the name of the cluster you want to modify.
On the Clusters page, in the Features section, click next to Cost Allocation.
In the Edit Cost Allocation dialog, select the Enable Cost Allocation checkbox.
Click Save changes.

Verify GKE cost allocation is enabled

gcloud

Describe the cluster:

gcloud container clusters describe CLUSTER_NAME

The output is similar to the following:

...
costManagementConfig:
  enabled: true
...

In this output, costManagementConfig indicates that GKE cost allocation is enabled.

Console

To check if GKE cost allocation is enabled on an existing cluster, do the following:

Go to the Google Kubernetes Engine page in the Google Cloud console. Go to Google Kubernetes Engine
Click the name of the cluster you want to modify.
On the Clusters page, in the Features section, next to Cost Allocation it will show whether GKE cost allocation is enabled.

Filter GKE cost data

The following example queries show you how to filter and group your GKE costs for supported resource types by cluster name, namespace, and label.

Get costs after credits by namespace

SELECT
  labels.value as namespace,
  SUM(cost) + SUM(IFNULL((SELECT SUM(c.amount) FROM UNNEST(credits) c), 0)) AS cost_after_credits,
FROM `project-ID.dataset.gcp_billing_export_resource_v1_XXXXXX-XXXXXX-XXXXXX`
LEFT JOIN UNNEST(labels) as labels
  ON labels.key = "k8s-namespace"
GROUP BY namespace
;

Get cluster costs before credits

SELECT
  SUM(cost) AS cost_before_credits,
  labels.value AS cluster_name
FROM  `project-ID.dataset.gcp_billing_export_resource_v1_XXXXXX-XXXXXX-XXXXXX`
LEFT JOIN UNNEST(labels) as labels
  ON labels.key = "goog-k8s-cluster-name"
GROUP BY labels.value
;

SELECT
  project.id AS project_id,
  labels.value AS cluster_name,
  sku.id AS sku_id,
  sku.description AS sku_description,
  SUM(cost) AS cost
FROM `project-ID.dataset.gcp_billing_export_resource_v1_XXXXXX-XXXXXX-XXXXXX`
JOIN UNNEST(labels) AS labels
  ON labels.key = "goog-k8s-cluster-name"
GROUP BY
  cluster_name, project_id, sku_description, sku_id
;

Replace BILLING_DATASET_TABLE with the name of the dataset that you created in BigQuery.

The table name is similar to gcp_billing_export_resource_v1_<BILLING_ACCOUNT_ID>.

Clean up

To avoid incurring charges to your Google Cloud account for the resources used in this page, disable GKE cost allocation for the cluster:

gcloud container clusters update CLUSTER_NAME \
    --no-enable-cost-allocation

You can also delete the cluster that you created.

What's next

View more Example queries for Cloud Billing data export.
Learn how to Visualize your costs with Looker Studio.

View detailed breakdown of cluster costs

Potential data volume increase

GKE cost allocation and cluster usage metering

Before you begin

Limitations and restrictions

View GKE cluster costs

Enable GKE cost allocation

Create a new cluster

gcloud

Console

Update a cluster

gcloud

Console

Verify GKE cost allocation is enabled

gcloud

Console

Filter GKE cost data

Get costs after credits by namespace

Get cluster costs before credits

Get total GKE-related costs by SKU

Clean up

What's next