Set up GKE Dataplane V2 observability

Autopilot Standard

This page shows how to configure Google Kubernetes Engine (GKE) clusters with GKE Dataplane V2 observability, starting in GKE versions 1.28 or later. For more information on the benefits and requirements of GKE Dataplane V2 observability, see About GKE Dataplane V2 observability.

Before you begin

Before you start, make sure you have performed the following tasks:

Enable the Google Kubernetes Engine API.

Enable Google Kubernetes Engine API

If you want to use the Google Cloud CLI for this task, install and then initialize the gcloud CLI. If you previously installed the gcloud CLI, get the latest version by running gcloud components update.
Note: For existing gcloud CLI installations, make sure to set the compute/region and compute/zone properties. By setting default locations, you can avoid errors in gcloud CLI like the following: One of [--zone, --region] must be supplied: Please specify location.

Configure GKE Dataplane V2 metrics

To collect metrics, you must configure GKE Dataplane V2 metrics. You can configure GKE Dataplane V2 metrics when you create a cluster or update a cluster running with GKE Dataplane V2. You can enable or disable GKE Dataplane V2 metrics using the gcloud CLI.

We recommend enabling GKE Dataplane V2 metrics and Google Cloud Managed Service for Prometheus on your GKE cluster. Once both are enabled, GKE Dataplane V2 metrics are sent to Google Cloud Managed Service for Prometheus.

Create an Autopilot cluster with GKE Dataplane V2 metrics enabled

When you create new GKE Autopilot clusters, GKE enables GKE Dataplane V2 metrics by default on the cluster without requiring a specific flag.

To use the GKE Autopilot cluster GKE Dataplane V2 metrics with Google Cloud Managed Service for Prometheus, configure the ClusterPodMonitoring resource to scrape the metrics and send them to Google Cloud Managed Service for Prometheus.

Create a ClusterPodMonitoring manifest:

# Copyright 2023 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

apiVersion: monitoring.googleapis.com/v1
kind: ClusterPodMonitoring
metadata:
  name: advanced-datapath-observability-metrics
spec:
  selector:
    matchLabels:
      k8s-app: cilium
  endpoints:
  - port: flowmetrics
    interval: 60s
    metricRelabeling:
    # only keep denormalized pod flow metrics
    - sourceLabels: [__name__]
      regex: 'pod_flow_(ingress|egress)_flows_count'
      action: keep
    # extract pod name
    - sourceLabels: [__name__, destination]
      regex: 'pod_flow_ingress_flows_count;([a-zA-Z0-9-\.]+)/([a-zA-Z0-9-\.]+)'
      replacement: '${2}'
      targetLabel: pod_name
      action: replace
    - sourceLabels: [__name__, source]
      regex: 'pod_flow_egress_flows_count;([a-zA-Z0-9-\.]+)/([a-zA-Z0-9-\.]+)'
      replacement: '${2}'
      targetLabel: pod_name
      action: replace
    # extract workload name by removing 2 last "-XXX" parts
    - sourceLabels: [pod_name]
      regex: '([a-zA-Z0-9-\.]+)((-[a-zA-Z0-9\.]+){2})'
      replacement: '${1}'
      targetLabel: workload_name
      action: replace
    # extract workload name by removing one "-XXX" part when pod name has only 2 parts (eg. daemonset)
    - sourceLabels: [pod_name]
      regex: '([a-zA-Z0-9\.]+)((-[a-zA-Z0-9\.]+){1})'
      replacement: '${1}'
      targetLabel: workload_name
      action: replace
    # extract pod namespace
    - sourceLabels: [__name__, destination]
      regex: 'pod_flow_ingress_flows_count;([a-zA-Z0-9-\.]+)/([a-zA-Z0-9-\.]+)'
      replacement: '${1}'
      targetLabel: namespace_name
      action: replace
    - sourceLabels: [__name__, source]
      regex: 'pod_flow_egress_flows_count;([a-zA-Z0-9-\.]+)/([a-zA-Z0-9-\.]+)'
      replacement: '${1}'
      targetLabel: namespace_name
      action: replace
    # extract remote workload name
    - sourceLabels: [__name__, source]
      regex: 'pod_flow_ingress_flows_count;([a-zA-Z0-9-\.]+)/([a-zA-Z0-9-\.]+)'
      replacement: '${2}'
      targetLabel: remote_workload
      action: replace
    - sourceLabels: [__name__, destination]
      regex: 'pod_flow_egress_flows_count;([a-zA-Z0-9-\.]+)/([a-zA-Z0-9-\.]+)'
      replacement: '${2}'
      targetLabel: remote_workload
      action: replace
    # extract remote workload namespace
    - sourceLabels: [__name__, source]
      regex: 'pod_flow_ingress_flows_count;([a-zA-Z0-9-\.]+)/([a-zA-Z0-9-\.]+)'
      replacement: '${1}'
      targetLabel: remote_namespace
      action: replace
    - sourceLabels: [__name__, destination]
      regex: 'pod_flow_egress_flows_count;([a-zA-Z0-9-\.]+)/([a-zA-Z0-9-\.]+)'
      replacement: '${1}'
      targetLabel: remote_namespace
      action: replace
    # default remote workload class to "pod"
    - replacement: 'pod'
      targetLabel: remote_class
      action: replace
    # extract remote workload class from reserved identity
    - sourceLabels: [__name__, source]
      regex: 'pod_flow_ingress_flows_count;reserved:([^/]*)'
      replacement: '${1}'
      targetLabel: remote_class
      action: replace
    - sourceLabels: [__name__, destination]
      regex: 'pod_flow_egress_flows_count;reserved:([^/]*)'
      replacement: '${1}'
      targetLabel: remote_class
      action: replace
  targetLabels:
    metadata: []

Apply the ClusterPodMonitoring manifest:

kubectl apply -f ClusterPodMonitoring.yaml

Create a Standard cluster with GKE Dataplane V2 metrics enabled

To enable GKE Dataplane V2 metrics, create a cluster with the --enable-dataplane-v2-metrics flag:

gcloud container clusters create CLUSTER_NAME \
    --enable-dataplane-v2 \
    --enable-ip-alias \
    --enable-managed-prometheus \
    --enable-dataplane-v2-metrics

Replace the following:

CLUSTER_NAME: the name of your cluster.

The --enable-managed-prometheus flag instructs GKE to use the metrics with Google Cloud Managed Service for Prometheus.