This guide explains how to use the Logging API to export your logs out of Cloud Logging.
To instead use the Logs Explorer to set up exports,
see Logs Explorer: Exporting logs. To
use the gcloud
command to set up exports, see
Command-line interface: Exporting logs. To find and use exported logs, see
Using exported logs.
Introduction to sinks
You control the export of log entries by creating sinks in Logging. Sinks contain the following information:
A sink identifier that names the sink within the parent resource. For example:
"my-first-sink"
A parent resource—a project, folder, billing account, or organization where in which the sink is created. The sink can only export logs belonging to the parent resource. The parents have the following formats:
"projects/[PROJECT_ID]" "folders/[FOLDER_ID]" "billingAccounts/[BILLING_ACCOUNT_ID]" "organizations/[ORGANIZATION_ID]"
An include children parameter. This is only relevant to sinks within organizations. See Aggregated sinks.
A filter to select which log entries to export. Filters are written in the logs query language. For more information, see Advanded logs queries. The following example matches all log entries with severity WARNING or higher from a specified Compute Engine VM instance:
"resource.type = gce_instance AND \ resource.labels.instance_id = "1428064241541024269" AND \ severity >= WARNING"
A destination to which the log entries are exported. The destination can be a Cloud Storage bucket, a BigQuery dataset, a Pub/Sub topic, or a Cloud Logging bucket all of which must belong to a Google Cloud project. The destinations have the following formats:
"storage.googleapis.com/[BUCKET_ID]" "bigquery.googleapis.com/projects/[PROJECT_ID]/datasets/[DATASET_ID]" "pubsub.googleapis.com/projects/[PROJECT_ID]/topics/[TOPIC_ID]" "logging.googleapis.com/projects/[PROJECT_ID]/locations/[REGION]/buckets/[BUCKET_ID]"
You can export the log entries to a custom destination in any Google Cloud project. The sink's source and the destination don't have to be within the same Google Cloud organization. A custom export destination still has to be in Cloud Storage, BigQuery, Pub/Sub, or Cloud Logging but allows you to send logs to a sink in a different Google Cloud project.
It isn't required that you have writer access to the export destination. Only the sink's writer service account requires access.
A writer identity that is a service account name. The destination's owner must give this service account permission to write to the destination. When exporting logs, Logging adopts this identity for authorization. For increased security, new sinks get their own unique service account:
[GENERATED_ID_1]@[GENERATED_ID_2].iam.gserviceaccount.com
Older sinks might use a shared service account for this writer:
cloud-logs@system.gserviceaccount.com
For more information on managing the writer identity, refer to Destination permissions.
For more information on sinks, see the LogSink type and the projects.sinks.create API method.
How sinks work
When a log entry arrives in a project, folder, billing account, or organization, Logging compares the log entry with the queries in the sinks created in that parent. Each sink whose filter matches the log entry writes a copy of the log entry to the sink's destination.
Organization and folder sinks can optionally be permitted to export log entries from their projects. See Aggregated sinks.
Troubleshooting: A sink might not export matching log entries for the following reasons:
- The sink's writer identity does not have permission to write logs to the destination. Errors occur until the sink's writer has permission to write log entries to the destination.
For information about delays in finding exported logs at your destination, see Using exported logs.
Creating sinks
This section explains how to create and modify sinks. Sinks can only export logs from the same resource that contains the sink—a particular project, folder, billing account, or organization.
Using a unique writer identity
You can use either a unique writer identity (a generated service account) or the backward-compatible shared writer identity (a common group) for your sink. For better security, use a unique writer for all new sinks. With a unique writer, the destination administrator knows that granting write permission to the identity will not allow any other sinks to write to the destination.
The following steps show you how to create or modify a sink using a unique writer identity:
Create the sink's destination, if needed.
Call the API method projects.sinks.create or projects.sinks.update to create or modify the sink. The following parameters are important for these instructions:
Set
uniqueWriterIdentity
totrue
. When updating a sink, you can change from using a shared writer to a unique writer. If the existing sink already uses a unique writer, the updated sink uses the same writer.If you are updating a sink that already uses a unique writer, do not default the
uniqueWriterIdentity
parameter. The default value isfalse
, which is not permitted in this case.
The methods return the new sink, which contains the new writer identity.
Add the new sink's writer identity to the destination's permission list, giving the writer permission to write to the destination.
Using the shared writer identity
You can use the backward-compatible, shared writer identity only if your sink is in the same project as your destination. Otherwise, the shared writer cannot get permission to write to the destination.
To create or update a sink with the shared writer, do the following:
Create the sink's destination, if needed.
Add one of the following writers—they are equivalent—to your destination's permission list, giving the writer permission to write to the destination:
- the group,
cloud-logs@google.com
- the service account ,
cloud-logs@system.gserviceaccount.com
- the group,
Call the API method projects.sinks.create or projects.sinks.update to create or modify your sink. The following parameter is relevant to these instructions:
- Set
uniqueWriterIdentity
tofalse
. If you are updating a sink, the existing sink must already be using the shared writer; you cannot change a sink from a unique writer to a shared writer.
The new sink begins exporting log entries immediately.
- Set
Access control
There are three issues relevant to controlling access to exported logs:
- Permitting users or applications to create and modify sinks.
- Permitting a sink's writer identity to write exported logs to a destination.
- Permitting users or applications to read or modify exported logs.
Managing sinks
The ability to create and view sinks is controlled by Logging IAM roles in the sink's parent:
To create or modify sinks, you or your application must have the IAM role Logging/Logs Configuration Writer or be an Owner of the sink's parent project or other parent resource.
To view existing sinks, you or your application must have the IAM roll Logging/Logs Viewer or be a Viewer of the sink's parent project or other parent resource.
Writing to the destination
When Logging is exporting logs through a sink, it can run under different writer identities:
Writer identity | Type | When creating the sink |
---|---|---|
cloud-logs@google.com |
group | uniqueWriterIdentity =false |
cloud-logs@system.gserviceaccount.com |
service account | uniqueWriterIdentity =false |
xxxxxxx@yyyyyy.gserviceaccount.com |
service account | uniqueWriterIdentity =true |
The first two writers are equivalent. The third identity is a unique writer for each sink, which has the advantage of letting you export logs to a destination in a project other than the sink's parent. If the sink's parent is not a project, you must use a unique writer, since all destinations are in projects.
Once you choose a kind of writer, use it in a way appropriate to each kind of destination:
In Cloud Storage, add the writer to your destination bucket's permission list and give the writer the Storage Object Creator role.
In BigQuery, add the writer to your destination dataset's permission list and give the writer the BigQuery Data Editor role.
In Pub/Sub, add the writer to your destination topic's permission list and give the writer the Pub/Sub Publisher role.
Reading from the destination
It is the destination's responsibility to control access to exported log entries. For example, Logging prevents the deletion or modification of audit logs that are held in Logging, and can restrict access to private audit logs. When audit logs are exported, the exported copies are no longer under Logging's control.
Code samples
List your sinks
List the sinks in a project to keep track of where your logs are being exported.
C#
To learn how to install and use the client library for Logging, see the Logging Client Libraries.
Go
To learn how to install and use the client library for Logging, see the Logging Client Libraries.
Java
To learn how to install and use the client library for Logging, see the Logging Client Libraries.
Node.js
To learn how to install and use the client library for Logging, see the Logging Client Libraries.
PHP
To learn how to install and use the client library for Logging, see the Logging Client Libraries.
Python
To learn how to install and use the client library for Logging, see the Logging Client Libraries.
Ruby
To learn how to install and use the client library for Logging, see the Logging Client Libraries.
Create a sink
Create a sink in your project to establish a destination to which you can
send your log entries. Before creating the sink, be sure you have granted
cloud-logs@system.gserviceaccount.com
permission to write logs to the
destination. For more information, see
Permissions and roles.
C#
To learn how to install and use the client library for Logging, see the Logging Client Libraries.
Go
To learn how to install and use the client library for Logging, see the Logging Client Libraries.
Java
To learn how to install and use the client library for Logging, see the Logging Client Libraries.
Node.js
To learn how to install and use the client library for Logging, see the Logging Client Libraries.
PHP
To learn how to install and use the client library for Logging, see the Logging Client Libraries.
Python
To learn how to install and use the client library for Logging, see the Logging Client Libraries.
Ruby
To learn how to install and use the client library for Logging, see the Logging Client Libraries.
Update a sink
Update a sink in a project to change its destination. For example, you might want to change the Cloud Storage bucket to which you are exporting logs monthly.
To update the sink, supply the sink's name and a new LogSink
object
containing the new destination. Before updating the sink, be sure you have
given the sink's writer identity permission to write logs to the destination.
For more information, see
Permissions and roles.
C#
To learn how to install and use the client library for Logging, see the Logging Client Libraries.
Go
To learn how to install and use the client library for Logging, see the Logging Client Libraries.
Java
To learn how to install and use the client library for Logging, see the Logging Client Libraries.
Node.js
To learn how to install and use the client library for Logging, see the Logging Client Libraries.
PHP
To learn how to install and use the client library for Logging, see the Logging Client Libraries.
Python
To learn how to install and use the client library for Logging, see the Logging Client Libraries.
Ruby
To learn how to install and use the client library for Logging, see the Logging Client Libraries.
Delete a sink
Delete a sink from your project if you no longer want to export log entries through that sink.
C#
To learn how to install and use the client library for Logging, see the Logging Client Libraries.
Go
To learn how to install and use the client library for Logging, see the Logging Client Libraries.
Java
To learn how to install and use the client library for Logging, see the Logging Client Libraries.
Node.js
To learn how to install and use the client library for Logging, see the Logging Client Libraries.
PHP
To learn how to install and use the client library for Logging, see the Logging Client Libraries.
Python
To learn how to install and use the client library for Logging, see the Logging Client Libraries.
Ruby
To learn how to install and use the client library for Logging, see the Logging Client Libraries.