Using Cloud Logging with Container-Optimized OS

Starting with milestone 67, Container-Optimized OS includes Google Cloud's operations suite's legacy Logging agent which exports system and container logs to Cloud Logging. The logging agent is configured by default to send logs from certain system critical services and user application containers to the Cloud Logging backend. For more details, see Container-Optimized OS specific configuration source or Google Cloud's operations suite documentation. This document focuses on how to use the logging agent on Container-Optimized OS and how to access logs.

Note: Logging is automatically available when you choose Container-Optimized OS as the node image for running your Google Kubernetes Engine clusters. These instructions are for when you are using Container-Optimized OS outside of GKE, such as in Compute Engine.

Enabling the logging agent

The logging agent is disabled by default. You can enable this feature when creating a new instance or by updating an existing instance.

Creating a new instance with the logging agent enabled

Console

To run a Compute Engine instance on Container-Optimized OS with logging agent enabled, perform the following:

1. Open the Compute Engine instance creation page on Google Cloud Console.

   Create a new Compute Engine instance

2. Specify a Name for your instance.

3. In the Boot disk section, select a Container-Optimized OS image.

4. Click on Management, security, disks, networking, sole tenancy to expand additional options.

5. Under the Management tab, scroll to the Metadata section. Add a new metadata entry, with Key as google-logging-enabled and Value as true.

6. Optionally, specify any other options for your use case. See Creating and configuring instances for more details.

7. Click Create to create and boot the instance.

gcloud

To run a Compute Engine instance on Container-Optimized OS with logging agent enabled, use the gcloud compute instances create command, and include google-logging-enabled=true in the metadata. For example:

gcloud compute instances create instance-name \
    --image-family cos-stable \
    --image-project cos-cloud \
    --zone compute-zone \
    --metadata google-logging-enabled=true

Replace the following:

• instance-name: the name of your VM instance.
• compute-zone: the compute zone for your instance.

For more information on the gcloud command, see the gcloud compute instances create reference documentation. For more details on creating Container-Optimized OS instances, see Creating and configuring instances.

Updating an existing instance to enable or disable the logging agent

Console

1. Go to the VM instances page.

   Go to the VM instances page

2. Click on the name of the Container-Optimized OS instance you want to update.

3. Click Edit from the VM instance details top bar.

4. Under the Custom metadata section, add a new metadata entry, with Key as google-logging-enabled and Value as true to enable or false to disable.

5. Click Save.

gcloud

To enable or disable logging agent on an existing instance, use the gcloud compute instances add-metadata command:

gcloud compute instances add-metadata instance-name \
    --zone compute-zone \
    --metadata google-logging-enabled=enabled-value

Replace the following:

• enabled-value: true to enable, false to disable.
• instance-name: the name of your VM instance.
• compute-zone: the compute zone for your instance.

Accessing logs

Console

1. Go to the VM instances page.

   Go to the VM instances page

2. Click on the name of the Container-Optimized OS instance you want to access logs from.

3. Under the Logs section, click Cloud Logging.

4. This opens the Logs Explorer for the given instance. For more information, see Using the Logs Explorer.

Note: In the query builder, you can filter by log name. Click on Log name, and select the given log name. For example, cos_audit for Audit logs or cos_containers for docker container logs.

gcloud

To access logs, use the gcloud logging read command. For example:

gcloud logging read \
"resource.type=gce_instance AND resource.labels.instance_id=instance-id" \
    --limit 10 \
    --format json \
    --freshness 30d

Replace the following:

• instance-id: the id of your VM instance.

This command attempts to read logs from the VM instance with instance-id, limiting to 10 logs, in JSON format, from the last 30 days.

For more information on the gcloud command, see the gcloud logging read reference documentation.

Known Limitations

Compatibility with gcplogs driver

Starting with milestone 89, if the logging agent included with Container-Optimized OS is enabled and Docker gcplogs logging driver is enabled for one or more containers, excessive warning logs may be output by the included logging agent. This may create log noise and/or increase Cloud Logging related charges.

A workaround is to not use gcplogs as a Docker logging driver, and to instead use the default driver as configured by Container-Optimized OS. Note that the logging agent included with Container-Optimized OS will export container logs to Cloud Logging, so using both solutions simultaneously is not necessary.