Understanding cluster resource usage

は、

This page explains how to use GKE usage metering to understand the usage profiles of GKE clusters and tie usage to individual teams or business units within your organization. GKE usage metering has no impact on billing for your project; it allows you to understand resource usage at a granular level.

Overview

GKE usage metering tracks information about the resource requests and actual resource usage of your cluster's workloads. Currently, GKE usage metering tracks information about CPU, GPU, TPU, memory, storage, and optionally network egress. You can differentiate resource usage using Kubernetes Namespaces, labels, or a combination of both.

Data is stored in BigQuery, where you can query it directly or export it for analysis using external tools such as Google Data Studio.

GKE usage metering is helpful for scenarios like the following:

  • Tracking per-tenant resource requests and actual resource consumption in a multitenant cluster where each tenant operates within a given Namespace.
  • Determining the resource consumption of a workload running in a given cluster, by assigning a unique label to the Kubernetes objects associated with the workload.
  • Identifying workloads whose resource requests differ significantly from their actual resource consumption, so that you can more efficiently allocate resources for each workload.

Changes from the initial Beta

GKE usage metering has the following changes since its initial Beta release:

  • Actual resource consumption is now tracked, in addition to resource requests. Resource consumption is tracked for clusters running v1.12.8-gke.8 and higher, v1.13.6-gke.7 and higher, or 1.14.2-gke.8 and higher. Resource consumption metering is stored in the gke_cluster_resource_consumption table of the BigQuery dataset. Previously, only resource requests were tracked.
  • TPU requests (but not actual resource consumption) are now tracked.
  • You can now track resource requests and actual resource consumption on nodes using custom machine types.
  • You can now enable GKE usage metering when creating or updating a cluster in Google Cloud Platform Console.
  • If the BigQuery dataset is deleted, GKE usage metering now recreates it automatically. Historical data is lost.

Upgrading

All changes are backward-compatible with the initial Beta, and data does not need to be modified or migrated.

When you upgrade a cluster to a GKE version that support resource consumption metering, it is not enabled automatically. You must explicitly enable it by using the --enable-resource-consumption-metering flag. An additional table is created in the BigQuery dataset automatically. Both tables use the same schema.

After upgrading, you only need to create a new Data Studio dashboard to add the information about actual resource consumption.

Limitations

You can use the sample BigQuery queries and Google Data Studio template to join GKE usage metering data with exported GCP billing data in BigQuery to estimate a cost breakdown by cluster, Namespace and labels. GKE usage metering data is purely advisory, and doesn't affect your GCP bill. For billing data, your GCP billing invoice is the sole source of truth.

The following limitations apply:

  • Special contract discounts or credits are not taken into account.
  • Resources created outside the scope of GKE are not tracked by Namespace or label.

Prerequisites

Before you use GKE usage metering, you must meet the following prerequisites:

  • To track actual resource consumption, the cluster must use GKE v1.11.10-gke.5 or higher, v1.12.8-gke.8 or higher, v1.13.6-gke.7 or higher, or 1.14.2-gke.3 or higher.
  • You must enable Billing export for BigQuery Charges are associated with BigQuery usage.
  • Version 250.0.0 or higher of the gcloud command is required. Use gcloud --version to check.
  • You must enable the BigQuery API in your Google Cloud Platform project. If you first enabled GKE after July 2018, this is already done.

Enabling GKE usage metering

To enable GKE usage metering, you first create a BigQuery dataset for either a single cluster, multiple clusters in the project, or the entire project. For more information about choosing a mapping between datasets and clusters, see Choosing one or more BigQuery datasets.

Next, you enable GKE usage metering when creating a new cluster or by modifying an existing cluster.

Optionally, you can create a Google Data Studio dashboard to visualize the resource usage of your clusters.

Creating the BigQuery dataset

To use GKE usage metering for clusters in your Google Cloud Platform project, you first create the BigQuery dataset, and then configure clusters to use it. You can use a single BigQuery dataset to store information about resource usage for multiple clusters in the same project.

Visit Creating Datasets for more details. Set the Default table expiration for the dataset to Never so that the table doesn't expire. However, in the second GKE usage metering Beta, if a table expires, it is recreated automatically (as an empty table).

Enabling GKE usage metering for a cluster

You can enable GKE usage metering on a new or existing cluster, using either the gcloud command or the GCP Console.

For clusters running GKE v1.12.8-gke.8 and higher or v1.13.6-gke.7 and higher, enabling GKE usage metering also enables resource consumption metering by default. To selectively disable resource consumption metering while continuing to track resource requests, see the specific instructions for enabling GKE usage metering using the gcloud command, in this topic.

Network egress metering is disabled by default. To enable it, see the caveats and instructions in Optional: Enabling network egress metering in this topic.

Creating a new cluster

gcloud

To create a cluster with GKE usage metering enabled, run the following command.

To insert your own values into the example before pasting, click any field that is highlighted to edit its value. To reset the values, refresh this page.

gcloud beta container clusters create [CLUSTER-NAME] \
  --resource-usage-bigquery-dataset [RESOURCE-USAGE-DATASET]

Resource consumption metering is enabled by default for clusters running GKE v1.12.8-gke.8 and higher or v1.13.6-gke.7 and higher. To disable it and only track resource requests, add the flag --no-enable-resource-consumption-metering to the command above. You also need to modify the example queries in the rest of this topic so that they do not query for resource consumption.

If needed, the required tables are created within the BigQuery dataset when the cluster starts.

Console

  1. Visit the Google Kubernetes Engine menu in GCP Console.

    Visit the Google Kubernetes Engine menu

  2. Click Create cluster.

  3. Choose the Standard cluster template or choose an appropriate template for your workload.

  4. Click Availability, networking, security, and additional features.

  5. Select Enable GKE usage metering.

  6. Enter the name of your BigQuery dataset.

  7. Optionally enable Enable network egress metering after reviewing the caveats and instructions in Optional: Enabling network egress metering.

  8. Continue configuring your cluster, then click Save.

Configuring an existing cluster

gcloud

To enable GKE usage metering on an existing cluster, run the following command.

To insert your own values into the example before pasting, click any field that is highlighted to edit its value. To reset the values, refresh this page.

gcloud beta container clusters update [CLUSTER-NAME] \
  --resource-usage-bigquery-dataset [RESOURCE-USAGE-DATASET]"

Resource consumption metering is enabled by default for clusters running GKE v1.12.8-gke.8 and higher or v1.13.6-gke.7 and higher. To disable it and only track resource requests, add the flag --no-enable-resource-consumption-metering to the command above. You also need to modify the example queries in the rest of this topic so that they do not query for resource consumption.

You can also change the dataset an existing cluster uses to store its usage metering data by changing the value of the --resource-usage-bigquery-dataset flag.

If needed, a table is created within the BigQuery dataset when the cluster is updated.

Console

  1. Visit the Google Kubernetes Engine menu in GCP Console.

    Visit the Google Kubernetes Engine menu

  2. Click the Edit button for the cluster you want to modify. It looks like a pencil.

  3. Enable GKE usage metering.

  4. Enter the name BigQuery dataset.

  5. Optionally enable Enable network egress metering after reviewing the caveats and instructions in Optional: Enabling network egress metering.

  6. Continue configuring your cluster, then click Save.

Optional: Enabling network egress metering

By default, network egress data is not collected or exported. Measuring network egress requires a network metering agent (NMA) running on each node. The NMA runs as a privileged Pod, consumes some resources on the node (CPU, memory, and disk space), and enables the nf_conntrack_acct sysctl flag on the kernel (for connection tracking flow accounting).

If you are comfortable with these caveats, you can enable network egress tracking for use with GKE usage metering. To enable network egress tracking, include the --enable-network-egress-metering option when creating or updating your cluster, or select Enable network egress metering when enabling GKE usage metering in the Google Cloud Platform Console.

Verifying that GKE usage metering is enabled

To verify that GKE usage metering is enabled on a cluster, and which BigQuery dataset stores the cluster's resource usage data, use the gcloud beta container clusters describe command. In the following example,

To insert your own values into the example before pasting, click any field that is highlighted to edit its value. To reset the values, refresh this page.

gcloud beta container clusters describe [CLUSTER-NAME]\
  --format="value(resourceUsageExportConfig)"

The output is empty if GKE usage metering is not enabled, and otherwise shows the BigQuery dataset used by the cluster, as in the following output:

bigqueryDestination={u'datasetId': u'test_usage_metering_dataset'}

Disabling GKE usage metering

gcloud

To disable GKE usage metering on a cluster, run the following command.

To insert your own values into the example before pasting, click any field that is highlighted to edit its value. To reset the values, refresh this page.

gcloud beta container clusters update [CLUSTER-NAME] \
  --clear-resource-usage-bigquery-dataset

Console

  1. Visit the Google Kubernetes Engine menu in GCP Console.

    Visit the Google Kubernetes Engine menu

  2. Click the Edit button for the cluster you want to modify. It looks like a pencil.

  3. Disable GKE usage metering.

  4. Click Save.

Choosing one or more BigQuery datastets

A dataset can hold GKE usage metering data for one or more clusters in your project. Whether you use one or many datasets depends on your security needs:

  • A single dataset for the entire project simplifies administration.
  • A dataset per cluster allows you to delegate granular access to the datasets.
  • A dataset per related group of clusters allows you to find the right mix of simplicity and granularity for your needs.

Visualizing GKE usage metering data using a Data Studio dashboard

You can visualize your GKE usage metering data using a Data Studio dashboard, which allows you to filter your data by cluster name, Namespace, or label, and to adjust the reporting period dynamically. If you are an advanced user of Data Studio and BigQuery, you can create a totally customized dashboard, but you can also clone a dashboard that we created specifically for GKE usage metering.

You can use the dashboard to visualize resource requests and consumption on your clusters over time.

Prerequisites

  1. Enable Exporting Google Cloud Platform billing data to BigQuery if it is not already enabled. During this process, you create a dataset, but the table within the dataset takes up to 5 hours to appear and start to be populated. When the table appears, its name is gcp_billing_export_v1_[BILLING_ACCOUNT_ID].
  2. Enable GKE usage metering on at least one cluster in the project. Note the name you chose for the BigQuery dataset.
  3. Enable Data Studio if it is not already enabled.
  4. Gather the following information, which is needed to configure the dashboard

    • Cloud Billing export dataset ID and data table
    • GKE usage metering dataset ID

Create the Data Studio dashboard

The process for creating the datasource and dashboard has been streamlined. After you meet the prerequisites, you use a custom Data Studio connector to create the data source and the dashboard automatically.

  1. Access the GKE Usage Metering Data Studio connector.

    You can also find the connector by visiting the Data Studio Connector Gallery and searching for GKE Usage Metering.

  2. Read and accept the prompt to grant the connector the permissions it needs.

  3. When prompted, provide the project ID, Cloud Billing export dataset ID and table, and the GKE usage metering dataset ID.

  4. Enable resource consumption metering if it is enabled for your cluster. This is true for most users.

  5. Click Connect. A new data source is created, with the default name GKE Usage Metering. Optionally, you can click the name in the heading of the page to rename it.

  6. Click Create report to create the Data Studio dashboard.

The dashboard is created, and opens in the dashboard editor. To view and use the dashboard, click View.

You can access the dashboard at any time in the list of Data Studio reports for your project.

Use the Data Studio dashboard

The dashboard contains multiple reports:

Usage breakdown
Overall cluster usage ratio among all clusters sending usage metering data to the same BigQuery data source, as well as detailed information about resource type (such as CPU, memory, or network egress) by Namespace. You can limit the report data to one or more individual clusters or Namespaces.
Usage breadown with unallocated resources
This report is similar to the previous one, but spreads unallocated resources proportionally across all Namespaces. Unallocated resources include idle resources and any resources that are not currently allocated by GKE usage metering to specific tenants.
Cost trends - drill down by namespace
Usage trends among all clusters sending usage metering data to the same BigQuery data source, by namespace. You can select one or more individual clusters, Namespaces, resources, or SKUs.
Cost trends - drill down by label
Cost trends among all clusters sending usage metering data to the same BigQuery data source. You can select one or more individual clusters, resources, label names, or label values.
Consumption-based Metering
Consumption trends among all clusters sending usage metering data to the same BigQuery data source. You can select one or more individual Namespaces, label keys, or label values. This report is only populated if resource consumption metering is enabled on at least one cluster.

You can change pages using the arrows near the top left of the screen. You can change the timeframe for a page using the date picker. To share the report with members of your organization, or to revoke access, click the Share Report link, which looks like a person with a + symbol.

This animation illustrates each screen in the dashboard.

After you copy the report into your project, you can customize it using the Data Studio report editor. Even if the report template provided by Google changes, your copy is unaffected.

Exploring GKE usage metering data using BigQuery

To view data about resource requests using BigQuery, query the gke_cluster_resource_usage table within the relevant BigQuery dataset.

To view data about actual resource consumption, query the gke_cluster_resource_usage_consumed table. Network egress consumption data remains in the gke_cluster_resource_usage because there is no concept of resource requests for egresses.

For more information about using queries in Cloud Bigtable, see Running queries. The fields in the schema are stable, though more fields may be added in the future.

These queries are simple examples. Customize your query to find the data you need.

Query for resource requests

To insert your own values into the example before pasting, click any field that is highlighted to edit its value. To reset the values, refresh this page.

SELECT
  cluster_name,
  labels,
  usage
FROM
  `[CLUSTER_GCP_PROJECT].[USAGE_METERING_DATASET].gke_cluster_resource_usage`
WHERE
  namespace="[NAMESPACE]"

Query for resource consumption

SELECT
  cluster_name,
  labels,
  usage
FROM
  `[CLUSTER_GCP_PROJECT].[USAGE_METERING_DATASET].gke_cluster_resource_usage_consumed`
WHERE
  namespace="[NAMESPACE]"

More examples

Expand the following sections to see more sophisticated examples.

GKE usage metering schema in BigQuery

The following table describes the schema for the GKE usage metering tables in the BigQuery dataset. If your cluster is running a version of GKE that supports resource consumption metering as well as resource requests, an additional table is created with the same schema.

Field Type Description
cluster_location STRING The name of the Compute Engine zone or region in which the GKE cluster resides.
cluster_name STRING The name of the GKE cluster.
namespace STRING The Kubernetes namespace from which the usage is generated.
resource_name STRING The name of the resource, such as "cpu", "memory", and "storage".
sku_id STRING The SKU ID of the underlying GCP cloud resource.
start_time TIMESTAMP The UNIX timestamp of when the usage began.
end_time TIMESTAMP The UNIX timestamp of when the usage ended.
fraction FLOAT The fraction of a cloud resource used by the usage. For a dedicated cloud resource that is solely used by a single namespace, the fraction is always 1.0. For resources shared among multiple namespaces, the fraction is calculated as the requested amount divided by the total capacity of the underlying cloud resource.
cloud_resource_size INTEGER The size of the underlying GCP resource. For example, the size of vCPUs on a n1-standard-2 instances is 2.
labels.key STRING The key of a Kubernetes label associated with the usage.
labels.value STRING The value of a Kubernetes label associated with the usage.
project.id STRING The ID of the project in which the GKE cluster resides.
usage.amount FLOAT The quantity of usage.unit used.
usage.unit STRING The base unit in which resource usage is measured. For example, the base unit for standard storage is byte-seconds.

What's next

このページは役立ちましたか?評価をお願いいたします。

フィードバックを送信...

Kubernetes Engine Documentation