Container-Optimized OS includes Google Cloud's operations suite's containerized legacy Logging agent which exports some system and container logs to Cloud Logging. This document focuses on how to use the logging agent on Container-Optimized OS and how to access logs.
Enabling the logging agent
The logging agent is disabled by default. You can enable this feature when creating a new instance or by updating an existing instance.
Creating a new instance with the logging agent enabled
Console
To run a Compute Engine instance on Container-Optimized OS with logging agent enabled, perform the following:
Open the Compute Engine instance creation page on Google Cloud console.
Specify a Name for your instance.
In the Boot disk section, select a Container-Optimized OS image.
Click on Management, security, disks, networking, sole tenancy to expand additional options.
Under the Management tab, scroll to the Metadata section. Add a new metadata entry, with Key as
google-logging-enabled
and Value astrue
.Optionally, specify any other options for your use case. See Creating and configuring instances for more details.
Click Create to create and boot the instance.
gcloud
To run a Compute Engine instance on Container-Optimized OS with logging
agent enabled, use the gcloud compute instances create
command, and include
google-logging-enabled=true
in the metadata. For example:
gcloud compute instances create instance-name \ --image-family cos-stable \ --image-project cos-cloud \ --zone compute-zone \ --metadata google-logging-enabled=true
Replace the following:
- instance-name: the name of your VM instance.
- compute-zone: the compute zone for your instance.
For more information on the gcloud
command, see the
gcloud compute instances create
reference documentation.
For more details on creating Container-Optimized OS instances, see
Creating and configuring instances.
Updating an existing instance to enable or disable the logging agent
Console
Go to the VM instances page.
Click on the name of the Container-Optimized OS instance you want to update.
Click Edit from the VM instance details top bar.
Under the Custom metadata section, add a new metadata entry, with Key as
google-logging-enabled
and Value astrue
to enable orfalse
to disable.Click Save.
gcloud
To enable or disable logging agent on an existing instance, use the gcloud
compute instances add-metadata
command:
gcloud compute instances add-metadata instance-name \ --zone compute-zone \ --metadata google-logging-enabled=enabled-value
Replace the following:
- enabled-value:
true
to enable,false
to disable. - instance-name: the name of your VM instance.
- compute-zone: the compute zone for your instance.
Enabling the logging agent in project metadata
Starting in milestone 97, logging can be enabled in project metadata:
gcloud compute project-info add-metadata \
--metadata google-logging-enabled=true
Accessing logs
Console
Go to the VM instances page.
Click on the name of the Container-Optimized OS instance you want to access logs from.
Under the Logs section, click Cloud Logging.
This opens the Logs Explorer for the given instance. For more information, see Using the Logs Explorer.
gcloud
To access logs, use the gcloud logging read
command. For example:
gcloud logging read \ "resource.type=gce_instance AND resource.labels.instance_id=instance-id" \ --limit 10 \ --format json \ --freshness 30d
Replace the following:
- instance-id: the id of your VM instance.
This command attempts to read logs from the VM instance with instance-id, limiting to 10 logs, in JSON format, from the last 30 days.
For more information on the gcloud
command, see the
gcloud logging read reference
documentation.
How does it work?
The logging agent is configured by default to send logs from certain system critical services and user application containers to the Cloud Logging backend. For example, logs from docker containers, selected systemd services, audit logs, journal log errors, etc. See Container-Optimized OS specific configuration source for complete default logging configuration. The docker command that starts the logging agent is defined in the Container-Optimized OS source for stackdriver-logging systemd service. The version of the containerized agent being run is defined in the Container-Optimized OS source app-admin/stackdriver directory.
Known Limitations
Compatibility with gcplogs
driver
Starting with
milestone 89,
if the logging agent included with Container-Optimized OS is enabled and Docker
gcplogs
logging driver is enabled for one or more containers, excessive
warning logs may be output by the included logging agent. This may create log
noise and/or increase Cloud Logging related charges.
A workaround is to not use gcplogs
as a Docker logging driver, and to
instead use the default driver as configured by Container-Optimized OS. Note
that the logging agent included with Container-Optimized OS will export
container logs to Cloud Logging, so using both solutions simultaneously is
not necessary.
References
- Google Cloud's operations suite's legacy Logging agent documentation. The containerized logging agent included with Container-Optimized OS is a subset of the legacy Logging agent, so this documentation may provide context on the agent more generally, outside of the scope of Container-Optimized OS.
- Google Cloud's operations suite documentation. Homepage for Google Cloud's operations suite documentation; may be useful for context.