Stackdriver Logging

This page explains how to use Stackdriver Logging to collect and query logs from Google Kubernetes Engine (GKE) clusters.

Overview

When Logging is enabled in your cluster, your logs are stored in a dedicated, persistent datastore. Your Google Cloud project has several logs that are relevant to a GKE cluster. These include the Admin Activity log, the Data Access log, and the Events log.

A log is a collection of log entries, and each log entry applies to a certain type of resource. These are the resource types that are most relevant to GKE clusters:

Resource type Display name
k8s_cluster Kubernetes Cluster
gke_cluster GKE Cluster Operations
gke_container GKE Container
gke_nodepool GKE Node Pool Operations

For detailed information about log entries that apply to the Kubernetes Cluster and GKE Cluster Operations resource types, see Audit logging.

While GKE itself stores logs, these logs are not stored permanently. For example, GKE container logs are removed when their host Pod is removed, when the disk on which they are stored runs out of space, or when they are replaced by newer logs. System logs are periodically removed to free up space for new logs. Cluster events are removed after one hour.

For container and system logs, GKE deploys a per-node logging agent that reads container logs, adds helpful metadata, and then stores them. The logging agent checks for container logs in the following sources:

  • Standard output and standard error logs from containerized processes
  • kubelet and container runtime logs
  • Logs for system components, such as VM startup scripts

For events, GKE uses a deployment in the kube-system namespace which automatically collects events and sends them to Logging.

Logging is compatible with JSON format. Logs are stored for up to 30 days.

Before you begin

To prepare for this task, perform the following steps:

  • Ensure that you have enabled the Google Kubernetes Engine API.
  • Enable Google Kubernetes Engine API
  • Ensure that you have installed the Cloud SDK.
  • Set your default project ID:
    gcloud config set project [PROJECT_ID]
  • If you are working with zonal clusters, set your default compute zone:
    gcloud config set compute/zone [COMPUTE_ZONE]
  • If you are working with regional clusters, set your default compute region:
    gcloud config set compute/region [COMPUTE_REGION]
  • Update gcloud to the latest version:
    gcloud components update

Enabling Logging

You can create a cluster with Logging enabled, or enable Logging in an existing cluster.

Creating a cluster with Logging

gcloud

When you create a cluster, the --enable-cloud-logging flag is automatically set, which enables Logging in the cluster.

To disable this default behavior, set the --no-enable-cloud-logging flag.

Console

  1. In the Cloud Console, go to the Kubernetes Engine > Kubernetes clusters page:

    Go to Kubernetes clusters

  2. Click Create cluster.

  3. Configure the cluster as needed.

  4. Click Advanced options. In the Additional features section, enable Enable Stackdriver Logging service.

  5. Click Create.

Enabling Logging for an existing cluster

gcloud

To enable Logging for an existing cluster, run the following command, where [CLUSTER_NAME] is the name of the cluster.

gcloud container clusters update [CLUSTER_NAME] --logging-service logging.googleapis.com

console

  1. In the Cloud Console, go to the Kubernetes Engine > Kubernetes clusters page:

    Go to Kubernetes clusters

  2. Click Edit.

  3. Set the Stackdriver Logging drop-down value to Enabled.

  4. Click Save.

Viewing logs

You can view logs in the in the Google Cloud Console's Stackdriver Logs Viewer.

To learn more about viewing logs, including how to construct basic and advanced queries, refer to Viewing logs in the Logging documentation.

Example queries

  • Query logs from Pods labelled nginx-deployment:

    resource.type="container"
    resource.labels.pod_id:"nginx-deployment-"
  • Query logs in the prod namespace with severity WARNING or above:

    resource.type="container"
    resource.labels.namespace_id="prod"
    severity>WARNING

    For more infromation about severity levels, see Log Severity.

  • Query logs from a specific container, my-container, in a specific Pod, my-pod, with a text payload:

    resource.type="container"
    resource.labels.container_name="my_container"
    resource.labels.pod_id="my_pod"
    textPayload:"abracadabra"

Limitations

Multi-line entries (entries with line feed characters) might not be processed correctly. To avoid this issue, wrap your logs in single-line JSON strings.

Best practices

  • Structured logging: Single-line JSON strings written to standard output or standard error will be read into Stackdriver as structured log entries. See Structured logging for more details. You can use Advanced logs filters to filter logs based on their fields.
  • Severities: By default, logs written to the standard output are on the INFO level and logs written to the standard error are on the ERROR level. Structured logs can include a severity field, which defines the log's severity.
  • Exporting to BigQuery: You can export logs to external services, such as BigQuery or Pub/Sub, for additional analysis. Logs exported to BigQuery retain their format and structure. See Overview of logs exports for further information.
  • Alerting: You can use logs-based metrics to set up alerting policies when Logging logs unexpected behavior. For an example, see Creating a simple alerting policy on a counter metric. For detailed information on logs-based metrics, see Overview of logs-based metrics.
  • Error reporting: You can use Stackdriver Error Reporting to collect errors produced in your clusters.

Disabling logging

gcloud

To disable Legacy Stackdriver for an existing cluster, run the following command, where [CLUSTER_NAME] is the name of the cluster.

gcloud beta container clusters update [CLUSTER_NAME] --logging-service none

If you are running Stackdriver Kubernetes Engine Monitoring in your cluster, you must disable both monitoring and logging by using gcloud beta to set the following flags in your cluster:

gcloud beta container clusters update [CLUSTER_NAME] --logging-service none --monitoring-service none

Console

  1. In the Cloud Console, go to the Kubernetes Engine > Kubernetes clusters page:

    Go to Kubernetes clusters

  2. Click Click Edit for the cluster you want to change.

  3. Set the Stackdriver Logging drop-down value to Disabled.

  4. Click Save.

What's next

Was this page helpful? Let us know how we did:

Send feedback about...

Stackdriver Monitoring