Understanding cluster resource usage

Beta

This is a Beta release of Google Kubernetes Engine usage metering. This feature is not covered by any SLA or deprecation policy and might be subject to backward-incompatible changes.

This page explains how to use Google Kubernetes Engine usage metering to understand the usage profiles of GKE clusters and tie usage to individual teams or business units within your organization. Usage metering has no impact on billing for your project; it allows you to understand resource usage at a granular level.

Overview

By default, you can find information about GKE cluster resource usage at the level of the Google Cloud Platform project. Usage metering tracks usage of individual cluster resources such as CPU, GPU, memory, network egress, and storage. You can differentiate resource usage using Kubernetes namespaces, labels, or a combination of both. Data is stored in BigQuery, and you can export the data for analysis using external tools such as Google Data Studio.

Usage metering is helpful for scenarios like the following:

• Tracking per-tenant usage in a multi-tenant cluster where each tenant operates within a given namespace of the cluster.
• Determining the usage of an application running in a given cluster, by assigning a unique label to the Kubernetes objects belonging to the application.

Limitations

You can use the sample BigQuery queries and Google Data Studio template to join GKE usage metering data with exported GCP billing data in BigQuery to estimate a cost breakdown by cluster, namespace and labels. Note that the data generated by GKE usage metering is purely advisory, and doesn't affect your GCP bill. For billing data, your GCP billing invoice is the sole source of truth.

The following limitations apply:

• Any special contract discounts, and credits are not taken into account.
• Resources created outside the scope of GKE are not tracked by namespace or label.
• CPU, GPU, memory, storage, and network egress are tracked.
• Usage metering does not currently work with nodes using custom machine types.

Prerequisites

Before you use usage metering, you must meet the following prerequisites:

• GKE v1.11.4-gke.8 or higher, v1.12.6-gke.8 or higher, or v1.13.0 or higher.
• Version 227.0.0 or higher of the gcloud command. Use gcloud --version to check.
• You must enable the BigQuery API in your Google Cloud Platform project. If you first enabled GKE after July 2018, this is already done.
• You must enable Billing export for BigQuery.

Important: During the Beta, you are charged for BigQuery usage.

Enabling usage metering

To enable usage metering, you first create a BigQuery dataset for either the cluster, multiple clusters in the project, or the entire project. For more information about choosing a mapping between datasets and clusters, see Choosing one or more BigQuery datasets.

Next, you create or modify the cluster to use usage metering.

Creating the BigQuery dataset

To use usage metering for clusters in your Google Cloud Platform project, you first create the BigQuery dataset, and then configure clusters to use it. You can use a single BigQuery dataset to store usage metering information for multiple clusters in the same project.

Visit Creating Datasets for more details. Set the Default table expiration for the dataset to Never so that the table doesn't expire and cause the export to fail.

Warning: If you delete a BigQuery dataset or table that a cluster is using to log usage metering data, Stackdriver Logging shows warnings such as Failed to upload a record to BigQuery. To resolve the warning, you must re-create the dataset and then recreate or update your cluster. Your historical data will be lost.

Enabling usage metering for a cluster

To enable usage metering for a new or existing cluster, use the --resource-usage-bigquery-dataset BIGQUERY_DATASET flag when constructing your gcloud command. You cannot currently enable usage metering using the Google Cloud Platform Console.

Note: By default, network egress data is not collected or exported. Measuring network egress requires a network metering agent (NMA) running on each node. The NMA runs as a privileged pod, consumes some resources on the node (CPU, memory, and disk space), and enables the nf_conntrack_acct sysctl flag on the kernel (for connection tracking flow accounting). If you are comfortable with these caveats, you can enable network egress tracking for use with GKE usage metering. To enable network egress tracking, include the --enable-network-egress-metering option when creating or updating your cluster.

Creating a new cluster:

gcloud

To create a cluster with usage metering enabled, run the following command. Substitute the name of your cluster where you see test-cluster and substitute your BigQuery dataset where you see test_usage_metering_dataset.

To insert your own values into the example before pasting, click any field that is highlighted to edit its value. To reset the values, refresh this page.

gcloud beta container clusters create test-cluster \
  --resource-usage-bigquery-dataset test_usage_metering_dataset

If needed, a table is created within the BigQuery dataset when the cluster starts.

Modifying an existing cluster:

gcloud

To enable usage metering on an existing cluster, run the following command. Substitute the name of your cluster where you see test-cluster and substitute the name of your BigQuery dataset where you see test_usage_metering_dataset.

To insert your own values into the example before pasting, click any field that is highlighted to edit its value. To reset the values, refresh this page.

gcloud beta container clusters update test-cluster \
  --resource-usage-bigquery-dataset test_usage_metering_dataset

Note: You can also change the dataset an existing cluster uses to store its usage metering data to, by changing the value of the --resource-usage-bigquery-dataset flag.

If needed, a table is created within the BigQuery dataset when the cluster is updated.

Disabling usage metering

gcloud

To disable usage metering on a cluster, run the following command. Substitute the name of your cluster where you see test-cluster.

To insert your own values into the example before pasting, click any field that is highlighted to edit its value. To reset the values, refresh this page.

gcloud beta container clusters update test_cluster \
  --clear-resource-usage-bigquery-dataset

Note: The BigQuery dataset is preserved. You can remove it if you no longer need the data and no cluster is using it.

Verifying that usage metering is enabled

To verify that usage metering is enabled on a cluster, and which BigQuery dataset stores the cluster's usage metering data, use the gcloud beta container clusters describe command. In the following example, substitute the name of your cluster where you see test-cluster.

To insert your own values into the example before pasting, click any field that is highlighted to edit its value. To reset the values, refresh this page.

gcloud beta container clusters describe test-cluster \
  --format="value(resourceUsageExportConfig)"

Output is empty if usage metering is not enabled, and otherwise shows the BigQuery dataset used by the cluster, as in the following output:

bigqueryDestination={u'datasetId': u'test_usage_metering_dataset'}

Usage metering data in BigQuery

The following table describes the contents of the usage metering data that is exported to BigQuery.

Field	Type	Description
cluster_location	STRING	The name of the Compute Engine zone or region in which the GKE cluster resides.
cluster_name	STRING	The name of the GKE cluster.
namespace	STRING	The Kubernetes namespace from which the usage is generated.
resource_name	STRING	The name of the resource, such as "cpu", "memory", and "storage".
sku_id	STRING	The SKU ID of the underlying GCP cloud resource.
start_time	TIMESTAMP	The UNIX timestamp of when the usage began.
end_time	TIMESTAMP	The UNIX timestamp of when the usage ended.
fraction	FLOAT	The fraction of a cloud resource used by the usage. For a dedicated cloud resource that is solely used by a single namespace, the fraction is always 1.0. For resources shared among multiple namespaces, the fraction is calculated as the requested amount divided by the total capacity of the underlying cloud resource.
cloud_resource_size	INTEGER	The size of the underlying GCP resource. For example, the size of vCPUs on a n1-standard-2 instances is 2.
labels.key	STRING	The key of a Kubernetes label associated with the usage.
labels.value	STRING	The value of a Kubernetes label associated with the usage.
project.id	STRING	The ID of the project in which the GKE cluster resides.
usage.amount	FLOAT	The quantity of usage.unit used.
usage.unit	STRING	The base unit in which resource usage is measured. For example, the base unit for standard storage is byte-seconds.

Choosing one or more BigQuery datastets

A dataset can hold usage metering data for one or more clusters in your project. Whether you use one or many datasets depends on your security needs:

• A single dataset for the entire project simplifies administration.
• A dataset per cluster allows you to delegate granular access to the datasets.
• A dataset per related group of clusters allows you to find the right mix of simplicity and granularity for your needs.

Viewing usage metering data using BigQuery

To view usage metering data using BigQuery, query the gke_cluster_resource_usage table within the relevant BigQuery dataset. For more information about using queries in Cloud Bigtable, see Running queries. The fields in the schema are stable, though more fields may be added in the future.

This query is only an example. Customize your query to find the data you need.

To insert your own values into the example before pasting, click any field that is highlighted to edit its value. To reset the values, refresh this page.

SELECT
  cluster_name,
  labels,
  fraction
FROM
  `PROJECT-ID.USAGE_METERING.gke_cluster_resource_usage`
WHERE
  namespace="accounting"

Expand the following sections to see more sophisticated examples.

How to query costs, broken down by namespace

The query below discards a cluster's resource usage when the billing information of the associated cloud resource has not yet been exported to the GCP billing export dataset. This happens when the time window of a cloud resource usage is ahead of the latest record in the exported GCP billing data (the latency for billing export can be up to 5 hours).

SELECT
  resource_usage.cluster_name,
  resource_usage.cluster_location,
  resource_usage.namespace,
  resource_usage.resource_name,
  resource_usage.sku_id,
  MIN(resource_usage.start_time) AS usage_start_time,
  MAX(resource_usage.end_time) AS usage_end_time,
  SUM(resource_usage.usage.amount * gcp_billing_export.rate) AS cost
FROM
  `PROJECT_ID.USAGE_METERING_DATASET.gke_cluster_resource_usage` AS resource_usage
LEFT JOIN (
  SELECT
    sku.id AS sku_id,
    SUM(cost) / SUM(usage.amount) AS rate,
    MIN(usage_start_time) AS min_usage_start_time,
    MAX(usage_end_time) AS max_usage_end_time
  FROM
    `PROJECT_ID.BILLING_DATASET.BILLING_TABLE`
  WHERE
    project.id = "CLUSTER_GCP_PROJECT"
  GROUP BY
    sku_id) AS gcp_billing_export
ON
  resource_usage.sku_id = gcp_billing_export.sku_id
WHERE
  resource_usage.start_time >= gcp_billing_export.min_usage_start_time
  AND resource_usage.end_time <= gcp_billing_export.max_usage_end_time
GROUP BY
  resource_usage.cluster_name,
  resource_usage.cluster_location,
  resource_usage.namespace,
  resource_usage.resource_name,
  resource_usage.sku_id
  

How to query costs in a specific time period, broken down by namespace and labels

The query below discards a cluster's resource usage when the billing information of the associated cloud resource has not yet been exported to the GCP billing export dataset. This happens when the time window of a cloud resource usage is ahead of the latest record in the exported GCP billing data (the latency for billing export can be up to 5 hours).

SELECT
  resource_usage.cluster_name,
  resource_usage.cluster_location,
  resource_usage.namespace,
  resource_usage.resource_name,
  resource_usage.sku_id,
  MIN(resource_usage.start_time) AS usage_start_time,
  MAX(resource_usage.end_time) AS usage_end_time,
  SUM(resource_usage.usage.amount * gcp_billing_export.rate) AS cost
FROM (
  SELECT
    *
  FROM
    `PROJECT_ID.USAGE_METERING_DATASET.gke_cluster_resource_usage` AS usage
  WHERE
    (
    SELECT
      COUNT(1)
    FROM
      UNNEST(usage.labels) AS pair
    WHERE
      pair IN (('LABEL_KEY',
          'LABEL_VALUE')) -- key value pairs; can have multiple pairs
      ) >= NUM_PAIRS -- number of key value pairs in the above line
    ) AS resource_usage
LEFT JOIN (
  SELECT
    sku.id AS sku_id,
    SUM(cost) / SUM(usage.amount) AS rate,
    MIN(usage_start_time) AS min_usage_start_time,
    MAX(usage_end_time) AS max_usage_end_time
  FROM
    `PROJECT_ID.BILLING_DATASET.BILLING_TABLE`
  WHERE
    project.id = "CLUSTER_GCP_PROJECT"
  GROUP BY
    sku_id) AS gcp_billing_export
ON
  resource_usage.sku_id = gcp_billing_export.sku_id
WHERE
  resource_usage.start_time >= gcp_billing_export.min_usage_start_time
  AND resource_usage.end_time <= gcp_billing_export.max_usage_end_time
GROUP BY
  resource_usage.cluster_name,
  resource_usage.cluster_location,
  resource_usage.namespace,
  resource_usage.resource_name,
  resource_usage.sku_id
  

Viewing usage metering data using a Data Studio dashboard

You can visualize your usage metering data using a Data Studio dashboard, which allows you to filter your data by cluster name, namespace, or label, and to adjust the reporting period dynamically. If you are an advanced user of Data Studio and BigQuery, you can create a totally customized dashboard, but you can also clone a dashboard that we created specifically for usage metering.

Note: You may see discrepancies between usage metering data and Cloud Billing data, due to upload latency. Batches of Cloud Billing data take up to 5 hours to appear in BigQuery, while usage metering data appears in BigQuery roughly every hour.

After you meet the prerequisites, you create Data Studio data sources, that query the usage metering and Cloud Billing data in BigQuery. Next, you copy a Data Studio dashboard into your project and configure it to use your three data sources. Finally, you can use the dashboard to understand usage metering for your clusters.

Prerequisites

1. Enable Exporting Google Cloud Platform billing data to BigQuery if it is not already enabled. During this process, you create a dataset, but the table within the dataset takes up to 5 hours to appear and start to be populated. When the table appears, note its name. It will be named something similar to gcp_billing_export_v1_<var>BILLING_ACCOUNT_ID</var>.
2. Enable Data Studio if it is not already enabled.

3. Gather the following information, which you will need to modify the queries that drive the dashboard:

   • Cloud Billing export dataset and table name from the first prerequisite
   • Usage metering dataset and table name, created in Create BigQuery dataset
   • GCP Project ID

Create the BigQuery data sources

These queries are complex, but you only need to replace the placeholder values with the values you gathered while completing the prerequisites. In each of the queries, make the following replacements:

• Replace PROJECT_ID with your project's ID (not the project name).
• Replace USAGE_METERING_DATASET with the dataset where your cluster writes its usage metering.
• Replace BILLING_DATASET and BILLING_TABLE with the dataset and table where Cloud Billing data is exported.

For each of the queries below, create a Data Studio data source.

1. Visit Data Sources.
2. Click the + icon near the bottom right of the screen to add a data source.
3. Select BigQuery.
4. Select Custom Query, then select your project ID.
5. Name your data source. Click the words Untitled Data Source at the top of the screen and replace the text with a descriptive name for your query. The headings for each query, below, are appropriate names for the data sources.
6. After replacing all placeholders in the query, paste it into the Query Editor. Be sure that Use Legacy SQL is not selected.
7. Click Connect in the top right corner of the screen.

BreakdownQuery query

To insert your own values into the example before pasting, click any field that is highlighted to edit its value. To reset the values, refresh this page.

SELECT
  resource_usage.cluster_name,
  resource_usage.cluster_location,
  resource_usage.namespace,
  resource_usage.labels,
  resource_usage.resource_name,
  resource_usage.sku_id,
  resource_usage.start_time AS usage_start_time,
  resource_usage.end_time AS usage_end_time,
  resource_usage.usage.amount * gcp_billing_export.rate AS cost
FROM
  `PROJECT_ID.USAGE_METERING_DATASET.gke_cluster_resource_usage` AS resource_usage
LEFT JOIN (
  SELECT
    sku.id AS sku_id,
    SUM(cost) / SUM(usage.amount) AS rate,
    MIN(usage_start_time) AS min_usage_start_time,
    MAX(usage_end_time) AS max_usage_end_time
  FROM
    `PROJECT_ID.BILLING_DATASET.BILLING_TABLE`
  WHERE
    project.id = "PROJECT_ID"
  GROUP BY
    sku_id) AS gcp_billing_export
ON
  resource_usage.sku_id = gcp_billing_export.sku_id
WHERE
  resource_usage.start_time >= gcp_billing_export.min_usage_start_time
  AND resource_usage.end_time <= gcp_billing_export.max_usage_end_time

GKEProjectCostFromGCP query

To insert your own values into the example before pasting, click any field that is highlighted to edit its value. To reset the values, refresh this page.

SELECT
*
FROM
  `PROJECT_ID.BILLING_DATASET.BILLING_TABLE`
WHERE
  project.id = "PROJECT_ID"

Create the Data Studio dashboard

We created a dashboard you can copy into your project. When you copy the dashboard, you are prompted to select three three data sources you just created.

1. Click this dashboard link.
2. Click the Copy button, , in the top right of the screen.
3. For the data sources, select the datasources that you just created. Use BreakdownQuery in the first field and GKEProjectCostFromGCP in the second.
4. Click Create Report.

These steps are shown in an animation in usage metering Dashboard Demonstration.

The dashboard is created, and you can access it at any time in the list of Data Studio reports for your project.

Use the Data Studio dashboard

Each dashboard screen is also demonstrated in the animation in usage metering Dashboard Demonstration.

The dashboard contains three separate pages:

• usage metering Monthly View shows overall cluster usage ratio among all clusters using usage metering in the project, as well as detailed information about resource type (such as CPU or memory) by namespace. You can limit the report data to one or more individual clusters or namespaces.

  

• usage metering Cost Trends by Namespace shows usage trends for all clusters using usage metering, by namespace. You can select one or more individual clusters, namespaces, resources, or SKUs.

  

• usage metering Cost Trends by Label shows usage trends for all clusters using usage metering, by label. You can select one or more individual clusters, namespaces, resources, or SKUs.

  

Note: Because of differences in the frequency of data availability between usage metering and Cloud Billing, data shown in the dashboard is not definitive and is informational only.

You can change pages using the arrows near the top left of the screen. You can change the timeframe for a page using the date picker. To share the report with members of your organization, or to revoke access, click the Share Report link, which looks like a person with a + symbol.

After you copy the report into your project, you can customize it using the Data Studio report editor. Even if the report template provided by Google changes, your copy is unaffected.

usage metering Dashboard Demonstration

The following animation illustrates the steps to create and use the dashboard. If the animation does not play automatically, click anywhere in the animation to start it.

What's next

• Learn about Cloud Billing.
• Learn about Data Studio.
• Learn about namespaces and labels.