This page explains how to use Stackdriver Logging to collect, query, and analyze logs from your Kubernetes Engine clusters.
You can enable Stackdriver Logging to have Kubernetes Engine
automatically collect, process, and store your container and system logs in a
dedicated, persistent datastore. Container logs are collected from
your containers. System logs are collected from the cluster's
components, such as
kubelet. Events are logs about
activity in the cluster, such as the scheduling of Pods.
While Kubernetes Engine itself stores logs, these logs are not stored permanently. For example, Kubernetes Engine container logs are removed when their host Pod is removed, when the disk on which they are stored runs out of space, or when they are replaced by newer logs. System logs are periodically removed to free up space for new logs. Cluster events are removed after one hour.
For container and system logs, Kubernetes Engine deploys a per-node logging agent that reads container logs, adds helpful metadata, and then stores them. The logging agent checks for container logs in the following sources:
- Standard output and standard error logs from containerized processes
kubeletand container runtime logs
- Logs for system components, such as VM startup scripts
For events, Kubernetes Engine uses a Deployment in the
kube-system namespace which automatically collects events and sends them
to Stackdriver Logging.
Stackdriver Logging is compatible with JSON and glog formats. Logs are stored for up to 30 days.
Before you begin
To prepare for this task, perform the following steps:
- Ensure that you have installed the Cloud SDK.
- Set your default project ID:
gcloud config set project [PROJECT_ID]
- Set your default compute zone:
gcloud config set compute/zone [COMPUTE_ZONE]
- Update all
gcloudcommands to the latest version:
gcloud components update
- Follow the Quickstart for Stackdriver Logging.
Enabling Stackdriver Logging
The following sections explain how to enable Stackdriver Logging.
Creating a cluster with logging
You can create a cluster with Stackdriver Logging enabled, or enable Stackdriver Logging in an existing cluster.
Visit the Kubernetes Engine menu in GCP Console.
Click Create cluster.
- Configure the cluster as desired. Ensure that Turn on Stackdriver Logging is selected.
- Click Create.
When you create a cluster, the
--enable-cloud-logging flag is
automatically set, which enables Stackdriver Logging in the cluster.
To disable this default behavior, set the
Enabling logging for an existing cluster
To enable logging for an existing cluster, run the following command, where
[CLUSTER_NAME] is the name of the cluster.
gcloud beta container clusters update [CLUSTER_NAME] --logging-service logging.googleapis.com
You can view logs in the in the Google Cloud Platform Console's Stackdriver Logs Viewer.
To learn more about viewing logs, including how to construct basic and advanced queries, refer to Viewing Logs in the Stackdriver Logging documentation.
Query logs from Pods labelled
Query logs in the
prod namespace with severity
WARNING or above:
resource.type="container" resource.labels.namespace_id="prod" severity>WARNING
Query logs from a specific container,
my-container, in a specific Pod,
my-pod, with a text payload:
resource.type="container" resource.labels.container_name="my_container" resource.labels.pod_id="my_pod" textPayload:"abracadabra"
Multi-line entries (entries with line feed characters) might not be processed correctly. To avoid this issue, wrap your logs in single-line JSON strings.
- Structured logging: Single-line JSON strings written to standard output or standard error will be read into Stackdriver as structured log entries. You can use advanced logs filters to filter logs based on their fields.
- Severities: By default, logs written to the standard output are on the
INFO level and logs written to the standard error are on the ERROR level.
Structured logs can include a
severityfield, which defines the log's severity. glog-formatted logs' severity is set automatically.
- Exporting to BigQuery: You can export logs to external services, such as BigQuery or Pub/Sub, for additional analysis. Logs exported to BigQuery retain their format and structure.
- Alerting: You can use logs-based metrics to set up altering policies when Stackdriver Logging logs unexpected behavior.
- Error Reporting: You can use Stackdriver Error Reporting to collect errors produced in your clusters.
To disable logging for an existing cluster, run the following command, where
[CLUSTER_NAME] is the name of the cluster.
gcloud beta container clusters update [CLUSTER_NAME] --logging-service none