Logging

This page explains how to use Stackdriver Logging to collect, query, and analyze logs from your Kubernetes Engine clusters.

Overview

You can enable Stackdriver Logging to have Kubernetes Engine automatically collect, process, and store your container and system logs in a dedicated, persistent datastore. Container logs are collected from your containers. System logs are collected from the cluster's components, such as docker and kubelet. Events are logs about activity in the cluster, such as the scheduling of Pods.

While Kubernetes Engine itself stores logs, these logs are not stored permanently. For example, Kubernetes Engine container logs are removed when their host Pod is removed, when the disk on which they are stored runs out of space, or when they are replaced by newer logs. System logs are periodically removed to free up space for new logs. Cluster events are removed after one hour.

For container and system logs, Kubernetes Engine deploys a per-node logging agent that reads container logs, adds helpful metadata, and then stores them. The logging agent checks for container logs in the following sources:

  • Standard output and standard error logs from containerized processes
  • kubelet and container runtime logs
  • Logs for system components, such as VM startup scripts

For events, Kubernetes Engine uses a Deployment in the kube-system namespace which automatically collects events and sends them to Stackdriver Logging.

Stackdriver Logging is compatible with JSON and glog formats. Logs are stored for up to 30 days.

Before you begin

To prepare for this task, perform the following steps:

  • Ensure that you have installed the Cloud SDK.
  • Set your default project ID:
    gcloud config set project [PROJECT_ID]
  • Set your default compute zone:
    gcloud config set compute/zone [COMPUTE_ZONE]
  • Update all gcloud commands to the latest version:
    gcloud components update

Enabling Stackdriver Logging

The following sections explain how to enable Stackdriver Logging.

Creating a cluster with logging

You can create a cluster with Stackdriver Logging enabled, or enable Stackdriver Logging in an existing cluster.

Console

  1. Visit the Kubernetes Engine menu in GCP Console.

    Visit the Kubernetes Engine menu

  2. Click Create cluster.

  3. Configure the cluster as desired. Ensure that Turn on Stackdriver Logging is selected.
  4. Click Create.

gcloud

When you create a cluster, the --enable-cloud-logging flag is automatically set, which enables Stackdriver Logging in the cluster.

To disable this default behavior, set the --no-enable-cloud-logging flag.

Enabling logging for an existing cluster

To enable logging for an existing cluster, run the following command, where [CLUSTER_NAME] is the name of the cluster.

gcloud beta container clusters update [CLUSTER_NAME] --logging-service logging.googleapis.com

Viewing logs

You can view logs in the in the Google Cloud Platform Console's Stackdriver Logs Viewer.

To learn more about viewing logs, including how to construct basic and advanced queries, refer to Viewing Logs in the Stackdriver Logging documentation.

Example queries

Query logs from Pods labelled nginx-deployment:

resource.type="container"
resource.labels.pod_id:"nginx-deployment-"

Query logs in the prod namespace with severity WARNING or above:

resource.type="container"
resource.labels.namespace_id="prod"
severity>WARNING

Query logs from a specific container, my-container, in a specific Pod, my-pod, with a text payload:

resource.type="container"
resource.labels.container_name="my_container"
resource.labels.pod_id="my_pod"
textPayload:"abracadabra"

Limitations

Multi-line entries (entries with line feed characters) might not be processed correctly. To avoid this issue, wrap your logs in single-line JSON strings.

Best practices

  • Structured logging: Single-line JSON strings written to standard output or standard error will be read into Stackdriver as structured log entries. You can use advanced logs filters to filter logs based on their fields.
  • Severities: By default, logs written to the standard output are on the INFO level and logs written to the standard error are on the ERROR level. Structured logs can include a severity field, which defines the log's severity. glog-formatted logs' severity is set automatically.
  • Exporting to BigQuery: You can export logs to external services, such as BigQuery or Pub/Sub, for additional analysis. Logs exported to BigQuery retain their format and structure.
  • Alerting: You can use logs-based metrics to set up altering policies when Stackdriver Logging logs unexpected behavior.
  • Error Reporting: You can use Stackdriver Error Reporting to collect errors produced in your clusters.

Disabling logging

To disable logging for an existing cluster, run the following command, where [CLUSTER_NAME] is the name of the cluster.

gcloud beta container clusters update [CLUSTER_NAME] --logging-service none

What's next

Was this page helpful? Let us know how we did:

Send feedback about...

Kubernetes Engine