Configuring Cloud Operations for GKE

Google Kubernetes Engine (GKE) includes integration with Cloud Monitoring and Cloud Logging. This integration lets you monitor your running GKE clusters, manage your system and debug logs, and analyze your system's performance using advanced profiling and tracing capabilities. This integration also provides a sophisticated GKE dashboard for observing your clusters.

This page describes how to do the following:

  • Create a new cluster and configure Cloud Operations for GKE.

  • Select whether to collect application logs or not.

  • Disable Cloud Monitoring and Cloud Logging for a cluster.

Before you begin

  • Ensure you are an Owner of the project containing your cluster.

  • Ensure you have enabled the Cloud Logging API.

    • You can check the status of the Cloud Logging API from its Overview page.

Available logs

You have a choice whether or not to send logs from your GKE cluster to Cloud Logging. If you choose to send logs to Cloud Logging, you must send system logs, and you may optionally send logs from additional sources.

Learn about Cloud Logging pricing, including best practices and tools for optimizing logging cost.

The following table indicates supported values for the --logging flag for the create and update commands.

Log Source Enabled by Default When Creating a Cluster? --logging value Logs Collected
None No NONE No logs sent to Cloud Logging; no log collection agent installed in the cluster. This value is not supported for GKE Autopilot clusters.
System Yes SYSTEM Collects logs from the following:
  • All Pods running in namespaces kube-system, istio-system, knative-serving, gke-system, and config-management-system.
  • Key services that are not containerized including docker/containerd runtime, kubelet, kubelet-monitor, node-problem-detector, and kube-container-runtime-monitor.
  • The node's serial ports output, if the VM instance metadata serial-port-logging-enable is set to true.
Workload Yes WORKLOAD All logs generated by non-system containers running on user nodes.

Available metrics

You have a choice whether or not to send metrics from your GKE cluster to Cloud Monitoring. If you choose to send metrics to Cloud Monitoring, you must send system metrics and may optionally send additional metrics.

Learn about Cloud Monitoring pricing, including which metrics are non-chargeable.

The following table indicates supported values for the --monitoring flag for the create and update commands.

Source Enabled by Default When Creating a Cluster? --monitoring value Metrics Collected
None No NONE No metrics sent to Cloud Monitoring; no metric collection agent installed in the cluster. This value is not supported for GKE Autopilot clusters.
System Yes SYSTEM Metrics from essential system components required for Kubernetes functionality. See a complete list of these Kubernetes metrics.
Workload No WORKLOAD Enable a fully managed pipeline capable of collecting Prometheus-style metrics exposed by any GKE workload. You must configure which metrics to collect by deploying a PodMonitor custom resource.

Configuring monitoring and logging support for a new cluster

The cluster-creation instructions in this section only cover the options relevant to Cloud Logging and Cloud Monitoring. For complete instructions on creating a GKE cluster, see Creating a cluster.

CONSOLE

  1. Go to the GKE Clusters page for your project. The following button takes you there:

    Go to Kubernetes clusters

  2. Click Create Cluster.

  3. From the navigation pane, under Cluster, click Features.

  4. Under Operations, select which logs and metrics you want collected.

    • For new clusters, Cloud Logging and Cloud Monitoring are enabled by default.

    • To disable Cloud Logging, clear the Enable Cloud Logging checkbox.

    • To disable Cloud Monitoring, clear the Enable Cloud Monitoring checkbox.

  5. Click Create.

GCLOUD

Use gcloud to create your cluster:

  1. Open a terminal window with Cloud SDK and gcloud installed. One way to do this is to use Cloud Shell.

  2. In the Cloud Console, activate Cloud Shell.

    Activate Cloud Shell

    At the bottom of the Cloud Console, a Cloud Shell session starts and displays a command-line prompt. Cloud Shell is a shell environment with the Cloud SDK already installed, including the gcloud command-line tool, and with values already set for your current project. It can take a few seconds for the session to initialize.

  3. For new clusters, Cloud Logging and Cloud Monitoring are enabled by default. To create your cluster, run the following command:

    gcloud container clusters create [CLUSTER_NAME] \
      --zone=[ZONE] \
      --project=[PROJECT_ID]
    

    In the previous expression, [CLUSTER_NAME] is your clusters' name, [ZONE] is the compute zone for the cluster, and [PROJECT_ID] is the Google Cloud project ID, not the project name.

    1. Alternatively, you can configure which logs are sent to Cloud Logging by passing a comma-separated list of values to the create command's --logging flag. To collect no logs, pass --logging=NONE. To collect both system and workload logs, pass --logging=SYSTEM,WORKLOAD. To collect only system logs, pass --logging=SYSTEM. For example:

      gcloud container clusters create [CLUSTER_NAME] \
        --zone=[ZONE] \
        --project=[PROJECT_ID] \
        --logging=SYSTEM
      
    2. Similarly, you can configure which metrics are sent to Cloud Monitoring by passing a comma-separated list of values to the create command's --monitoring flag. To collect no metrics, pass --monitoring=NONE. To collect system metrics, pass --monitoring=SYSTEM. For example:

      gcloud container clusters create [CLUSTER_NAME] \
        --zone=[ZONE] \
        --project=[PROJECT_ID] \
        --monitoring=SYSTEM
      

Configuring monitoring and logging for an existing cluster

The following section details how to modify the Cloud Logging and Cloud Monitoring integration for an existing GKE cluster.

Which monitoring and logging support does my cluster use?

To see the Cloud Logging and Cloud Monitoring integration settings for your cluster, follow these steps:

  1. In the Google Cloud Console, click your cluster's name:

    Go to Kubernetes clusters

  2. In the Details panel for your cluster, see the status for Cloud Logging and Cloud Monitoring.

Modifying your cluster

To change the Cloud Logging or Cloud Monitoring integration settings for an existing cluster, follow these steps:

CONSOLE

  1. Go to the GKE Kubernetes clusters page for your project:

    Go to Kubernetes clusters

  2. Click the name of your cluster.

  3. To modify which logs are sent to Cloud Logging or which metrics are sent to Cloud Monitoring, click Edit next to Cloud Logging or Cloud Monitoring.

  4. In the dialog that appears, select which logs or metrics you want sent to Cloud Logging or Cloud Monitoring.

  5. Click Save.

GCLOUD

The following gcloud instructions cover upgrading your cluster's monitoring and logging support using the gcloud container clusters update command. Notice that you use the update command, not the upgrade command.

  1. Open a terminal window with Cloud SDK and gcloud installed. One way to do this is to use Cloud Shell.

  2. In the Cloud Console, activate Cloud Shell.

    Activate Cloud Shell

    At the bottom of the Cloud Console, a Cloud Shell session starts and displays a command-line prompt. Cloud Shell is a shell environment with the Cloud SDK already installed, including the gcloud command-line tool, and with values already set for your current project. It can take a few seconds for the session to initialize.

  3. If you have not done so, set up default values for your cluster in Cloud SDK:

    gcloud config set project [PROJECT_ID]
    
  4. Configure which logs are sent to Cloud Logging by passing a comma-separated list of values to the gcloud container clusters update command's --logging flag. See a full list of available log sources. For example, to collect both system and workload logs, pass --logging=SYSTEM,WORKLOAD. To collect only system logs, pass --logging=SYSTEM. Or, to collect no logs, pass --logging=NONE:

        gcloud container clusters update [CLUSTER_NAME] \
          --zone=[ZONE] \
          --logging=NONE
    
  5. Configure which metrics are sent to Cloud Monitoring by passing a comma-separated list of values to the gcloud container clusters update command's --monitoring flag. See a full list of available metric sources. For example, to collect system metrics, pass --monitoring=SYSTEM. Or, to collect no metrics, pass --monitoring=NONE:

        gcloud container clusters update [CLUSTER_NAME] \
          --zone=[ZONE] \
          --monitoring=NONE
    

Deprecated Configuration Parameters

If you have previously been using the old configuration parameters to configure logging and monitoring support for your GKE cluster, those parameters are deprecated. The table below shows the equivalent configuration parameters to replace the deprecated flags.

Old Configuration Old create Arguments Old update Arguments New create and update Arguments
Disabled --no-enable-stackdriver-kubernetes --no-enable-stackdriver-kubernetes --logging=NONE
--monitoring=NONE
System monitoring only (Logging disabled) --enable-stackdriver-kubernetes
--no-enable-cloud-logging
--logging-service=none
--monitoring-service=monitoring.googleapis.com/kubernetes
--logging=NONE
--monitoring=SYSTEM
System and workload logging only (Monitoring disabled) --enable-stackdriver-kubernetes
--no-enable-cloud-monitoring
--logging-service=logging.googleapis.com/kubernetes
--monitoring-service=none
--logging=SYSTEM,WORKLOAD
--monitoring=NONE
System logging and monitoring only (beta) --enable-logging-monitoring-system-only --enable-logging-monitoring-system-only --logging=SYSTEM
--monitoring=SYSTEM
System and workload logging and monitoring --enable-stackdriver-kubernetes --enable-stackdriver-kubernetes --logging=SYSTEM,WORKLOAD
--monitoring=SYSTEM

What's next

  • Learn about the costs associated with Cloud Logging and Cloud Monitoring by reading the Pricing page.

  • Learn more about Using Prometheus, a monitoring tool often used with Kubernetes.