Using Cloud Logging with Container-Optimized OS

Container-Optimized OS includes a logging agent which exports some system and container logs to Cloud Logging. For x86-based Container-Optimized OS images, the agent is a Google Cloud's operations suite's containerized legacy Logging agent. For Arm-based Container-Optimized OS images, the agent is a native OS package called fluent-bit. This document focuses on how to use the logging agent on Container-Optimized OS and how to access logs.

Note: Logging is automatically available when you choose Container-Optimized OS as the node image for running your Google Kubernetes Engine clusters. These instructions are for when you are using Container-Optimized OS outside of GKE, such as in Compute Engine.

Enable the logging agent

The logging agent is disabled by default. You can enable this feature when creating a new instance or by updating an existing instance.

Create an instance with the logging agent enabled

Console

To run a Compute Engine instance on Container-Optimized OS with logging agent enabled, perform the following:

1. Open the Compute Engine instance creation page on Google Cloud console.

   Create a new Compute Engine instance

2. Specify a Name for your instance.

3. In the Boot disk section, select a Container-Optimized OS image.

4. Click on Management, security, disks, networking, sole tenancy to expand additional options.

5. Under the Management tab, scroll to the Metadata section. Add a new metadata entry, with Key as google-logging-enabled and Value as true.

6. Optionally, specify any other options for your use case. See Creating and configuring instances for more details.

7. Click Create to create and boot the instance.

gcloud

To run a Compute Engine instance on Container-Optimized OS with logging agent enabled, use the gcloud compute instances create command, and include google-logging-enabled=true in the metadata. For example:

gcloud compute instances create instance-name \
    --image-family cos-stable \
    --image-project cos-cloud \
    --zone compute-zone \
    --metadata google-logging-enabled=true

Replace the following:

• instance-name: the name of your VM instance.
• compute-zone: the compute zone for your instance.

For more information on the gcloud command, see the gcloud compute instances create reference documentation. For more details on creating Container-Optimized OS instances, see Creating and configuring instances.

Update an existing instance to enable or disable the logging agent

Console

1. Go to the VM instances page.

   Go to the VM instances page

2. Click on the name of the Container-Optimized OS instance you want to update.

3. Click Edit from the VM instance details top bar.

4. Under the Custom metadata section, add a new metadata entry, with Key as google-logging-enabled and Value as true to enable or false to disable.

5. Click Save.

gcloud

To enable or disable logging agent on an existing instance, use the gcloud compute instances add-metadata command:

gcloud compute instances add-metadata instance-name \
    --zone compute-zone \
    --metadata google-logging-enabled=enabled-value

Replace the following:

• enabled-value: true to enable, false to disable.
• instance-name: the name of your VM instance.
• compute-zone: the compute zone for your instance.

Enable the logging agent in project metadata

Starting in milestone 97, logging can be enabled in project metadata:

  gcloud compute project-info add-metadata \
    --metadata google-logging-enabled=true

Note: Metadata flags defined at instance level takes precedence over metadata flags defined at project level.

Access logs

Console

1. Go to the VM instances page.

   Go to the VM instances page

2. Click on the name of the Container-Optimized OS instance you want to access logs from.

3. Under the Logs section, click Cloud Logging.

4. This opens the Logs Explorer for the given instance. For more information, see Using the Logs Explorer.

Note: In the query builder, you can filter by log name. Click on Log name, and select the given log name. For example, cos_audit for Audit logs or cos_containers for docker container logs.

gcloud

To access logs, use the gcloud logging read command. For example:

gcloud logging read \
"resource.type=gce_instance AND resource.labels.instance_id=instance-id" \
    --limit 10 \
    --format json \
    --freshness 30d

Replace the following:

• instance-id: the id of your VM instance.

This command attempts to read logs from the VM instance with instance-id, limiting to 10 logs, in JSON format, from the last 30 days.

For more information on the gcloud command, see the gcloud logging read reference documentation.

How does it work?

The logging agent is configured by default to send logs from certain system critical services and user application containers to the Cloud Logging backend. For example, logs from docker containers, selected systemd services, audit logs, journal log errors, etc. See Container-Optimized OS specific configuration source (x86 images and Arm images) for complete default logging configuration.

For x86 images, the logging agent is a Google Cloud's operations suite's containerized legacy Logging agent. The docker command that starts the logging agent is defined in the Container-Optimized OS source for stackdriver-logging systemd service. The version of the containerized agent being run is defined in the Container-Optimized OS source app-admin/stackdriver directory.

For Arm images, the logging agent is a native OS package called fluent-bit. The agent is built into the OS and updated along with the OS images.

Known Limitations

Compatibility with gcplogs driver

Starting with milestone 89, if the logging agent included with Container-Optimized OS is enabled and Docker gcplogs logging driver is enabled for one or more containers, excessive warning logs may be output by the included logging agent. This may create log noise and/or increase Cloud Logging related charges.

A workaround is to not use gcplogs as a Docker logging driver, and to instead use the default driver as configured by Container-Optimized OS. Note that the logging agent included with Container-Optimized OS will export container logs to Cloud Logging, so using both solutions simultaneously is not necessary.

Logging agent configuration compatibility

The logging agent is a fluentd-based container on x86-based Container-Optimized OS images and a fluent-bit binary on Arm-based Container-Optimized OS images. The configurations of the two agents are not compatible. This is not a problem if you only rely on the default logging configuration built into the OS images. However, if you have a custom logging configuration, you might experience breakage when migrating workloads to the images that are based on a different architecture.

References

• Google Cloud's operations suite's legacy Logging agent documentation. The containerized logging agent included with Container-Optimized OS is a subset of the legacy Logging agent, so this documentation may provide context on the agent more generally, outside of the scope of Container-Optimized OS.
• Google Cloud's operations suite documentation. Homepage for Google Cloud's operations suite documentation; may be useful for context.