Installing monitoring and logging support

Google Kubernetes Engine (GKE) includes integration with Cloud Monitoring and Cloud Logging. This integration lets you monitor your running GKE clusters, manage your system and debug logs, and analyze your system's performance using advanced profiling and tracing capabilities.

This page describes how to do the following:

• Create a new cluster and configure Kubernetes Engine Monitoring or Legacy Logging and Monitoring.

• Select whether to collect application logs or not.

• Disable Cloud Monitoring and Cloud Logging for a cluster.

Kubernetes Engine Monitoring provides a single option that enables improved support for both Cloud Monitoring and Cloud Logging in your clusters. With Legacy Logging and Monitoring, you could enable Monitoring without Logging, but this option is not supported in Kubernetes Engine Monitoring.

Note: Security logs, including basic audit logs, are available for GKE and most other Google Cloud services without going through this procedure. For more information, see Cloud Audit Logs.

Before you begin

• Determine which monitoring and logging integration option you want to use.

• Determine whether application logs should be collected.

• Ensure you are an Owner of the project containing your cluster. Your project must be associated with a Workspace.

Note: If your Workspace is configured with a VPC Service Controls Service Perimeter for the Monitoring API, then Kubernetes Engine Monitoring will not work for GKE clusters outside the perimeter. To learn more about Service Perimeters, see the VPC Service Controls Service Perimeter configuration documentation.

Controlling the collection of application logs

Beta

This product or feature is in a pre-release state and might change or have limited support. For more information, see the product launch stages.

By default, GKE collects logs for workloads deployed to the cluster. This is a powerful tool for instrumenting your application; however, this automated collection may not be preferred. Beginning with GKE version 1.15.7, you can configure Kubernetes Engine Monitoring to only capture system logs and not collect application logs.

When the system-only option is selected, the following logs are collected:

• All pods running in namespaces kube-system, istio-system, knative-serving,gke-system, and config-management-system.

• Key services that are not containerized including docker/containerd runtime, kubelet, kubelet-monitor, node-problem-detector, and kube-container-runtime-monitor.

• The node’s serial ports output, if the VM instance metadata serial-port-logging-enable is set to true.

Configuring monitoring and logging support for a new cluster

When you create a new GKE cluster, Kubernetes Engine Monitoring is enabled by default. Alternatively, you can select Legacy Logging and Monitoring integration for clusters running GKE version 1.14 or lower, or you can disable Google Cloud's operations suite all together.

Note: If Kubernetes Engine Monitoring and Legacy Cloud Logging is disabled, support is offered on best-effort basis and might require additional effort from your engineering team.

The cluster-creation instructions in this section only cover the options relevant to monitoring and logging. For complete instructions on creating a GKE cluster, see Creating a cluster.

CONSOLE

1. Go to the GKE Kubernetes clusters page for your project. The following button takes you there:

   Go to Kubernetes clusters

2. Click Create Cluster.

3. Select a Master version for your cluster. The version you select changes the Cloud Monitoring and Cloud Logging options available to you.

4. From the navigation pane, under Cluster, click Features.

5. Under Telemetry, select the logging and monitoring support for your cluster:

   • For new clusters, Kubernetes Engine Monitoring is enabled by default.

     • If your Master version is 1.15.7 or later, you can select one of the following options from the drop-down list:

       • System and workload logging and monitoring (recommended)
       • System logging and monitoring only (beta)

     • If your Master version is 1.14 or earlier, you can enable Legacy Logging and Monitoring support:

       1. From the drop-down list, select Legacy Stackdriver.
       2. Select Enable legacy Stackdriver Logging or Enable legacy Stackdriver Monitoring, or both. Legacy Monitoring and Legacy Logging can be enabled or disabled separately.

   • To disable Kubernetes Engine Monitoring, clear the Enable Stackdriver Kubernetes Engine Monitoring checkbox.

6. Configure the cluster as you desire.

7. Click Create to create the cluster.

GCLOUD

Use gcloud to create your cluster:

1. Open a terminal window with Cloud SDK and gcloud installed. One way to do this is to use Cloud Shell.

2. In the Cloud Console, go to Cloud Shell.

   Go to Cloud Shell

   At the bottom of the Cloud Console, a Cloud Shell session opens and displays a command-line prompt. Cloud Shell is a shell environment with the Cloud SDK already installed, including the gcloud command-line tool, and with values already set for your current project. It can take a few seconds for the session to initialize.

3. To select Kubernetes Engine Monitoring support for your new cluster, include the --enable-stackdriver-kubernetes option in the create command:

   gcloud beta container clusters create [CLUSTER_NAME] \
     --zone=[ZONE] \
     --project=[PROJECT_ID] \
     --cluster-version=[CLUSTER_VERSION] \
     --enable-stackdriver-kubernetes
   

   In the previous expression, [CLUSTER_NAME] is your clusters' name, [ZONE] is the compute zone for the cluster, and [PROJECT_ID] is the Google Cloud project ID, not the project name.

   [CLUSTER_VERSION] is the GKE version to install on the cluster. You can enter the specific patch release to install, for example 1.12.8-gke.10. Otherwise, you can enter the minor version to install, for example, 1.12.8, and your cluster is created with the latest patch version. For more information on GKE versioning, see Versioning and upgrades.

4. Alternatively, to select Legacy Logging and Monitoring for your a GKE cluster version 1.14 and earlier, create your cluster and include the --enable-cloud-logging and/or --enable-cloud-monitoring options in the create command. For example, a simple cluster creation might look like the following:

   gcloud container clusters create [CLUSTER_NAME] \
     --zone=[ZONE] \
     --project=[PROJECT_ID] \
     --cluster-version=[CLUSTER_VERSION]
   

Configuring monitoring and logging support for an existing cluster

The following section details how to modify the monitoring and logging integration for an existing GKE cluster.

Note: Changing your monitoring and logging support, and changing the Kubernetes version of your cluster are distinct actions. Changing the Kubernetes version of your cluster doesn't change your configured monitoring and logging support.

Which monitoring and logging support does my cluster use?

To see which monitoring and logging support your cluster is using, complete the following steps:

1. In the Google Cloud Console, click your cluster's name:

   Go to Kubernetes clusters

2. In the Details panel for your cluster, see the status for Legacy Stackdriver Monitoring, Legacy Stackdriver Logging, and Kubernetes Engine Monitoring.

   

Modifying your cluster

To change the monitoring and logging configuration for an existing cluster, follow the instructions below:

CONSOLE

1. Go to the GKE Kubernetes clusters page for your project:

   Go to Kubernetes clusters

2. Upgrade your cluster to version 1.12.7 or higher. For instructions, see Upgrading clusters. Allow the upgrade to complete.

3. Click Edit for your cluster:

   

4. Select the appropriate integration option for your cluster.

   • To change to Kubernetes Engine Monitoring, in the Stackdriver Kubernetes Engine Monitoring drop-down box, select System and workload logging and monitoring (recommended).

     

   • To change to Kubernetes Engine Monitoring with system logs only, in the Stackdriver Kubernetes Engine Monitoring drop-down box, select System logging and monitoring only.

     

   • To change to Legacy Logging and Monitoring support (for GKE version 1.14 and below, do the following:

     1. In the Kubernetes Engine Monitoring drop-down box, select Legacy Stackdriver.

     2. In the Legacy Stackdriver Logging drop-down box, select Enabled.

     3. In the Legacy Stackdriver Monitoring drop-down box, select Enabled.

        

   • To completely disable Google Cloud's operations suite integration, select Disabled in the Kubernetes Engine Monitoring drop-down box.

5. Click Save.

GCLOUD

The following gcloud instructions cover upgrading your cluster's monitoring and logging support using the gcloud container clusters update command. Notice that you use the update command, not the upgrade command, even though we sometimes refer to this process as "upgrading."

1. Open a terminal window with Cloud SDK and gcloud installed. One way to do this is to use Cloud Shell.

2. In the Cloud Console, go to Cloud Shell.

   Go to Cloud Shell

   At the bottom of the Cloud Console, a Cloud Shell session opens and displays a command-line prompt. Cloud Shell is a shell environment with the Cloud SDK already installed, including the gcloud command-line tool, and with values already set for your current project. It can take a few seconds for the session to initialize.

3. If you have not done so, set up default values for your cluster in Cloud SDK:

   gcloud config set project [PROJECT_ID]
   gcloud config set compute/zone [ZONE]
   

4. Update your cluster to the desired Kubernetes version and select the monitoring and logging support you want in the cluster. Kubernetes Engine Monitoring is supported in all versions of GKE. Legacy Logging and Monitoring is supported in GKE 1.14 and earlier.

5. The following update command that enables Kubernetes Engine Monitoring only shows the options needed for Google Cloud's operations suite:

   gcloud beta container clusters update [CLUSTER_NAME] \
     --zone=[ZONE]  --region=[REGION]  \
     --enable-stackdriver-kubernetes
     

   The field [REGION] is the compute region of the cluster.

   Note: You must use Cloud SDK version 248.0.0 or higher.

6. Alternatively, the following update command that enables Legacy Logging and Monitoring only shows the options needed for Google Cloud's operations suite:

   gcloud beta container clusters update [CLUSTER_NAME] \
     --zone=[ZONE]  --region=[REGION]  \
     --logging-service logging.googleapis.com \
     --monitoring-service monitoring.googleapis.com
     

   Note: The system-only logging option is not available in gcloud.

What's next

• If you are using Legacy Logging and Monitoring, you should begin to become familiar with the changes in Kubernetes Engine Monitoring by reading Migrating to Kubernetes Engine Monitoring.

• You can deploy your application software to your cluster as usual. For more information, see Overview of deploying workloads.

• To use Prometheus with this release of Kubernetes Engine Monitoring, see Using Prometheus.