Configuring Cloud Operations for GKE

Google Kubernetes Engine (GKE) includes integration with Cloud Logging and Cloud Monitoring, including Google Cloud Managed Service for Prometheus. This integration lets you monitor your running GKE clusters, manage your system and debug logs, and analyze your system's performance using advanced profiling and tracing capabilities. This integration also provides a sophisticated GKE dashboard for observing your clusters.

This page describes how to do the following:

• Create a new cluster and configure Cloud Logging, Cloud Monitoring, and Managed Service for Prometheus.

• Select which logs and metrics to collect.

• Disable Cloud Logging, Cloud Monitoring, and Managed Service for Prometheus for a cluster.

Note: Security logs, including basic audit logs, are available for GKE and most other Google Cloud services even when Cloud Logging is not enabled for a GKE cluster. For more information, see Cloud Audit Logs.

Note: For GKE Autopilot clusters, you cannot disable the Cloud Logging and Cloud Monitoring integration.

Before you begin

• Ensure you are an Owner of the project containing your cluster.

• Ensure you have enabled the Cloud Logging API. (You can check the status of the Cloud Logging API from its Overview page).

Note: If your metrics scope is configured with a VPC Service Controls Service Perimeter for the Monitoring API, then Cloud Operations for GKE won't work for GKE clusters outside the perimeter. To learn more about Service Perimeters, see Service perimeter details and configuration.

Warning: If you've disabled Cloud Logging or Cloud Monitoring, GKE customer support is offered on a best-effort basis and might require additional effort from your engineering team.

Available logs

You have a choice whether or not to send logs from your GKE cluster to Cloud Logging. If you choose to send logs to Cloud Logging, you must send system logs, and you may optionally send logs from additional sources.

Learn about Cloud Logging pricing, including best practices and tools for optimizing logging cost.

The following table indicates supported values for the --logging flag for the create and update commands.

Log Source	Enabled by Default When Creating a Cluster?	--logging value	Logs Collected
None	No	NONE	No logs sent to Cloud Logging; no log collection agent installed in the cluster. This value is not supported for GKE Autopilot clusters.
System	Yes	SYSTEM	Collects logs from the following: All Pods running in namespaces kube-system, istio-system, knative-serving, gke-system, and config-management-system. Key services that are not containerized including docker/containerd runtime, kubelet, kubelet-monitor, node-problem-detector, and kube-container-runtime-monitor. The node's serial ports output, if the VM instance metadata serial-port-logging-enable is set to true. Additionally, collects Kubernetes events.
Workload	Yes	WORKLOAD	All logs generated by non-system containers running on user nodes.
API server	No	API_SERVER	All logs generated by kube-apiserver.
Scheduler	No	SCHEDULER	All logs generated by kube-scheduler.
Controller Manager	No	CONTROLLER_MANAGER	All logs generated by kube-controller-manager.

Available metrics

You have a choice whether or not to send metrics from your GKE cluster to Cloud Monitoring. If you choose to send metrics to Cloud Monitoring, you must send system metrics and may optionally send additional metrics.

Learn about Cloud Monitoring pricing, including which metrics are non-chargeable.

The following table indicates supported values for the --monitoring flag for the create and update commands.

Source	Enabled by Default When Creating a Cluster?	--monitoring value	Metrics Collected
None	No	NONE	No metrics sent to Cloud Monitoring; no metric collection agent installed in the cluster. This value is not supported for GKE Autopilot clusters.
System	Yes	SYSTEM	Metrics from essential system components required for Kubernetes functionality. See a complete list of these Kubernetes metrics.
API server	No	API_SERVER	Metrics from kube-apiserver. See a complete list of API server metrics.
Scheduler	No	SCHEDULER	Metrics from kube-scheduler. See a complete list of Scheduler metrics.
Controller Manager	No	CONTROLLER_MANAGER	Metrics from kube-controller-manager. See a complete list of Controller Manager metrics.
Workload	No	WORKLOAD	(Deprecated) Enable a fully managed pipeline capable of collecting Prometheus-style metrics exposed by any GKE workload. You must configure which metrics to collect by deploying a PodMonitor custom resource.

Additionally, you can collect Prometheus-style metrics exposed by any GKE workload by using Google Cloud Managed Service for Prometheus, which lets you monitor and alert on your workloads, using Prometheus, without having to manually manage and operate Prometheus at scale.

Configuring monitoring and logging support for a new cluster

Note: GKE Autopilot clusters always use Google's best practices for telemetry collection, meaning that system and workload logging is always enabled and system monitoring is always enabled. This section only applies to GKE Standard clusters.

The cluster-creation instructions in this section only cover the options relevant to Cloud Logging and Cloud Monitoring. For complete instructions on creating a GKE cluster, see Creating a cluster.

Console

1. Go to the GKE Clusters page for your project. The following button takes you there:

   Go to Kubernetes clusters

2. Click Create.

3. Click Configure for Standard mode.

4. From the navigation pane, under Cluster, click Features.

5. Under Operations, select which logs and metrics you want collected.

   • For new clusters, Cloud Logging and Cloud Monitoring are enabled by default.

   • To change which logs are collected, click the Components menu for Cloud Logging and select the components from which you want to collect logs.

   • To change which metrics are collected, click the Components menu for Cloud Monitoring and select the components from which you want to collect metrics.

   • To disable the Cloud Logging integration (except for audit logs), clear the Enable Cloud Logging checkbox.

   • To disable the Cloud Monitoring integration, clear the Enable Cloud Monitoring checkbox.

   • To enable Managed Service for Prometheus, select the Enable Managed Service for Prometheus checkbox (requires 1.21.4-gke.300 or later). This checkbox enables the managed collector, which must be configured.

   Note: If you've disabled Cloud Logging or Cloud Monitoring, GKE customer support is offered on a best-effort basis and might require additional effort from your engineering team.

6. Click Create.

gcloud

Use gcloud to create your cluster:

1. Open a terminal window with Google Cloud CLI. One way to do this is to use Cloud Shell.

2. In the Google Cloud console, activate Cloud Shell.

   Activate Cloud Shell

   At the bottom of the Google Cloud console, a Cloud Shell session starts and displays a command-line prompt. Cloud Shell is a shell environment with the Google Cloud CLI already installed and with values already set for your current project. It can take a few seconds for the session to initialize.

3. For new clusters, Cloud Logging and Cloud Monitoring are enabled by default. To create your cluster, run the following command:

   gcloud container clusters create [CLUSTER_NAME] \
     --region=[REGION] \
     --project=[PROJECT_ID]
   

   In the previous expression, [CLUSTER_NAME] is your clusters' name, [ZONE] is the compute zone for the cluster, and [PROJECT_ID] is the Google Cloud project ID, not the project name.

   1. Alternatively, you can configure which logs are sent to Cloud Logging by passing a comma-separated list of values to the create command's --logging flag. To collect no logs, pass --logging=NONE. To collect system, API server, Scheduler, and Controller Manager logs, pass --logging=SYSTEM,API_SERVER,SCHEDULER,CONTROLLER_MANAGER. To collect both system and workload logs, pass --logging=SYSTEM,WORKLOAD. For example:

      gcloud container clusters create [CLUSTER_NAME] \
        --region=[REGION] \
        --project=[PROJECT_ID] \
        --logging=SYSTEM,WORKLOAD
      

   2. Similarly, you can configure which metrics are sent to Cloud Monitoring by passing a comma-separated list of values to the create command's --monitoring flag. To collect no metrics, pass --monitoring=NONE. To collect system metrics, pass --monitoring=SYSTEM. To collect system, API server, Scheduler, and Controller Manager metrics, pass --monitoring=SYSTEM,API_SERVER,SCHEDULER,CONTROLLER_MANAGER. For example:

      gcloud container clusters create [CLUSTER_NAME] \
        --region=[REGION] \
        --project=[PROJECT_ID] \
        --monitoring=SYSTEM,API_SERVER,SCHEDULER,CONTROLLER_MANAGER
      

   3. Separately, you can enable Managed Service for Prometheus by using the --enable-managed-prometheus flag. For example:

      gcloud container clusters create [CLUSTER_NAME] \
        --region=[REGION] \
        --project=[PROJECT_ID] \
        --enable-managed-prometheus
      

      The --enable-managed-prometheus flag enables the managed collector, which must be configured.

Terraform

• To configure the collection of logs and metrics using Terraform, see the logging_config and monitoring_config blocks in the Terraform registry for google_container_cluster. Enabling the collection of logs from the API server, scheduler, and controller manager requires Terraform version 4.44.0 or later.

• For general information about using Google Cloud with Terraform, see Terraform with Google Cloud.

Configuring monitoring and logging for an existing cluster

The following section details how to modify the Cloud Logging and Cloud Monitoring integration for an existing GKE cluster.

Note: Changing your monitoring and logging support, and changing the Kubernetes version of your cluster are distinct actions. Changing the Kubernetes version of your cluster doesn't change your configured monitoring and logging support.

Which monitoring and logging support does my cluster use?

To see the Cloud Logging and Cloud Monitoring integration settings for your cluster, follow these steps:

1. In the Google Cloud console, click your cluster's name:

   Go to Kubernetes clusters

2. In the Details panel for your cluster, see the status for Cloud Logging, Cloud Monitoring, and Managed Service for Prometheus.

Modifying your cluster

To change the Cloud Logging or Cloud Monitoring integration settings for an existing cluster, follow these steps:

Console

1. Go to the GKE Kubernetes clusters page for your project:

   Go to Kubernetes clusters

2. Click the name of your cluster.

3. To modify which logs are sent to Cloud Logging, which metrics are sent to Cloud Monitoring, or whether Managed Service for Prometheus is enabled, click Edit next to Cloud Logging, Cloud Monitoring, or Managed Service for Prometheus.

4. After changing your configuration for Cloud Logging, Cloud Monitoring or Managed Service for Prometheus, click Save.

gcloud

The following gcloud instructions cover upgrading your cluster's monitoring and logging support using the gcloud container clusters update command. Notice that you use the update command, not the upgrade command.

1. Open a terminal window with gcloud CLIinstalled. One way to do this is to use Cloud Shell.

2. In the Google Cloud console, activate Cloud Shell.

   Activate Cloud Shell

   At the bottom of the Google Cloud console, a Cloud Shell session starts and displays a command-line prompt. Cloud Shell is a shell environment with the Google Cloud CLI already installed and with values already set for your current project. It can take a few seconds for the session to initialize.

3. If you have not done so, set up default values for your cluster in gcloud CLI:

   gcloud config set project [PROJECT_ID]
   

4. Configure which logs are sent to Cloud Logging by passing a comma-separated list of values to the gcloud container clusters update command's --logging flag. See a full list of available log sources. For example, to collect both system and workload logs, pass --logging=SYSTEM,WORKLOAD. To collect only system logs, pass --logging=SYSTEM. Or, to collect no logs, pass --logging=NONE:

       gcloud container clusters update [CLUSTER_NAME] \
         --region=[REGION] \
         --logging=NONE
   

5. Configure which metrics are sent to Cloud Monitoring by passing a comma-separated list of values to the gcloud container clusters update command's --monitoring flag. See a full list of available metric sources. For example, to collect system metrics, pass --monitoring=SYSTEM. Or, to collect no metrics, pass --monitoring=NONE:

       gcloud container clusters update [CLUSTER_NAME] \
         --region=[REGION] \
         --monitoring=NONE
   

6. Configure whether Managed Service for Prometheus is enabled by using the --enable-managed-prometheus or --disable-managed-prometheus flags. For example:

       gcloud container clusters update [CLUSTER_NAME] \
         --region=[REGION] \
         --enable-managed-prometheus
   

Terraform

• To configure the collection of logs and metrics using Terraform, see the logging_config and monitoring_config blocks in the Terraform registry for google_container_cluster. Enabling the collection of logs from the API server, scheduler, and controller manager requires Terraform version 4.44.0 or later.

• For general information about using Google Cloud with Terraform, see Terraform with Google Cloud.

Deprecated Configuration Parameters

If you have previously been using the old configuration parameters to configure logging and monitoring support for your GKE cluster, those parameters are deprecated. The table below shows the equivalent configuration parameters to replace the deprecated flags.

Old Configuration	Old create Arguments	Old update Arguments	New create and update Arguments
Disabled	--no-enable-stackdriver-kubernetes	--no-enable-stackdriver-kubernetes	--logging=NONE --monitoring=NONE
System monitoring only (Logging disabled)	--enable-stackdriver-kubernetes --no-enable-cloud-logging	--logging-service=none --monitoring-service=monitoring.googleapis.com/kubernetes	--logging=NONE --monitoring=SYSTEM
System and workload logging only (Monitoring disabled)	--enable-stackdriver-kubernetes --no-enable-cloud-monitoring	--logging-service=logging.googleapis.com/kubernetes --monitoring-service=none	--logging=SYSTEM,WORKLOAD --monitoring=NONE
System logging and monitoring only (beta)	--enable-logging-monitoring-system-only	--enable-logging-monitoring-system-only	--logging=SYSTEM --monitoring=SYSTEM
System and workload logging and monitoring	--enable-stackdriver-kubernetes	--enable-stackdriver-kubernetes	--logging=SYSTEM,WORKLOAD --monitoring=SYSTEM

What's next

• Learn about the costs associated with Cloud Logging, Cloud Monitoring, and Managed Service for Prometheus by reading the Pricing page.
• Learn about viewing your GKE logs in Cloud Logging.