Using Cloud Logging with Container-Optimized OS

Container-Optimized OS includes Google Cloud's operations suite's containerized legacy Logging agent which exports some system and container logs to Cloud Logging. This document focuses on how to use the logging agent on Container-Optimized OS and how to access logs.

Enabling the logging agent

The logging agent is disabled by default. You can enable this feature when creating a new instance or by updating an existing instance.

Creating a new instance with the logging agent enabled

Console

To run a Compute Engine instance on Container-Optimized OS with logging agent enabled, perform the following:

  1. Open the Compute Engine instance creation page on Google Cloud console.

    Create a new Compute Engine instance

  2. Specify a Name for your instance.

  3. In the Boot disk section, select a Container-Optimized OS image.

  4. Click on Management, security, disks, networking, sole tenancy to expand additional options.

  5. Under the Management tab, scroll to the Metadata section. Add a new metadata entry, with Key as google-logging-enabled and Value as true.

  6. Optionally, specify any other options for your use case. See Creating and configuring instances for more details.

  7. Click Create to create and boot the instance.

gcloud

To run a Compute Engine instance on Container-Optimized OS with logging agent enabled, use the gcloud compute instances create command, and include google-logging-enabled=true in the metadata. For example:

gcloud compute instances create instance-name \
    --image-family cos-stable \
    --image-project cos-cloud \
    --zone compute-zone \
    --metadata google-logging-enabled=true

Replace the following:

  • instance-name: the name of your VM instance.
  • compute-zone: the compute zone for your instance.

For more information on the gcloud command, see the gcloud compute instances create reference documentation. For more details on creating Container-Optimized OS instances, see Creating and configuring instances.

Updating an existing instance to enable or disable the logging agent

Console

  1. Go to the VM instances page.

    Go to the VM instances page

  2. Click on the name of the Container-Optimized OS instance you want to update.

  3. Click Edit from the VM instance details top bar.

  4. Under the Custom metadata section, add a new metadata entry, with Key as google-logging-enabled and Value as true to enable or false to disable.

  5. Click Save.

gcloud

To enable or disable logging agent on an existing instance, use the gcloud compute instances add-metadata command:

gcloud compute instances add-metadata instance-name \
    --zone compute-zone \
    --metadata google-logging-enabled=enabled-value

Replace the following:

  • enabled-value: true to enable, false to disable.
  • instance-name: the name of your VM instance.
  • compute-zone: the compute zone for your instance.

Enabling the logging agent in project metadata

Starting in milestone 97, logging can be enabled in project metadata:

  gcloud compute project-info add-metadata \
    --metadata google-logging-enabled=true

Accessing logs

Console

  1. Go to the VM instances page.

    Go to the VM instances page

  2. Click on the name of the Container-Optimized OS instance you want to access logs from.

  3. Under the Logs section, click Cloud Logging.

  4. This opens the Logs Explorer for the given instance. For more information, see Using the Logs Explorer.

gcloud

To access logs, use the gcloud logging read command. For example:

gcloud logging read \
"resource.type=gce_instance AND resource.labels.instance_id=instance-id" \
    --limit 10 \
    --format json \
    --freshness 30d

Replace the following:

  • instance-id: the id of your VM instance.

This command attempts to read logs from the VM instance with instance-id, limiting to 10 logs, in JSON format, from the last 30 days.

For more information on the gcloud command, see the gcloud logging read reference documentation.

How does it work?

The logging agent is configured by default to send logs from certain system critical services and user application containers to the Cloud Logging backend. For example, logs from docker containers, selected systemd services, audit logs, journal log errors, etc. See Container-Optimized OS specific configuration source for complete default logging configuration. The docker command that starts the logging agent is defined in the Container-Optimized OS source for stackdriver-logging systemd service. The version of the containerized agent being run is defined in the Container-Optimized OS source app-admin/stackdriver directory.

Known Limitations

Compatibility with gcplogs driver

Starting with milestone 89, if the logging agent included with Container-Optimized OS is enabled and Docker gcplogs logging driver is enabled for one or more containers, excessive warning logs may be output by the included logging agent. This may create log noise and/or increase Cloud Logging related charges.

A workaround is to not use gcplogs as a Docker logging driver, and to instead use the default driver as configured by Container-Optimized OS. Note that the logging agent included with Container-Optimized OS will export container logs to Cloud Logging, so using both solutions simultaneously is not necessary.

References

  • Google Cloud's operations suite's legacy Logging agent documentation. The containerized logging agent included with Container-Optimized OS is a subset of the legacy Logging agent, so this documentation may provide context on the agent more generally, outside of the scope of Container-Optimized OS.
  • Google Cloud's operations suite documentation. Homepage for Google Cloud's operations suite documentation; may be useful for context.