Exporting logs in the API

This guide explains how to use the Logging API to export your logs out of Cloud Logging.

To instead use the Logs Explorer to set up exports, see Logs Explorer: Exporting logs. To use the gcloud command to set up exports, see Command-line interface: Exporting logs. To find and use exported logs, see Using exported logs.

Introduction to sinks

You control the export of log entries by creating sinks in Logging. Sinks contain the following information:

A sink identifier that names the sink within the parent resource. For example:
```
"my-first-sink"
```
A parent resource—a project, folder, billing account, or organization where in which the sink is created. The sink can only export logs belonging to the parent resource. The parents have the following formats:
```
"projects/[PROJECT_ID]"
"folders/[FOLDER_ID]"
"billingAccounts/[BILLING_ACCOUNT_ID]"
"organizations/[ORGANIZATION_ID]"
```
An include children parameter. This is only relevant to sinks within organizations. See Aggregated sinks.
A filter to select which log entries to export. Filters are written in the logs query language. For more information, see Advanded logs queries. The following example matches all log entries with severity WARNING or higher from a specified Compute Engine VM instance:
```
"resource.type = gce_instance AND \
   resource.labels.instance_id = "1428064241541024269" AND \
   severity >= WARNING"
```
A destination to which the log entries are exported. The destination can be a Cloud Storage bucket, a BigQuery dataset, a Pub/Sub topic, or a Cloud Logging bucket all of which must belong to a Google Cloud project. The destinations have the following formats:
```
"storage.googleapis.com/[BUCKET_ID]"
"bigquery.googleapis.com/projects/[PROJECT_ID]/datasets/[DATASET_ID]"
"pubsub.googleapis.com/projects/[PROJECT_ID]/topics/[TOPIC_ID]"
"logging.googleapis.com/projects/[PROJECT_ID]/locations/[REGION]/buckets/[BUCKET_ID]"
```
You can export the log entries to a custom destination in any Google Cloud project. The sink's source and the destination don't have to be within the same Google Cloud organization. A custom export destination still has to be in Cloud Storage, BigQuery, Pub/Sub, or Cloud Logging but allows you to send logs to a sink in a different Google Cloud project.

It isn't required that you have writer access to the export destination. Only the sink's writer service account requires access.
A writer identity that is a service account name. The destination's owner must give this service account permission to write to the destination. When exporting logs, Logging adopts this identity for authorization. For increased security, new sinks get their own unique service account:
```
[GENERATED_ID_1]@[GENERATED_ID_2].iam.gserviceaccount.com
```
Older sinks might use a shared service account for this writer:
```
cloud-logs@system.gserviceaccount.com
```
For more information on managing the writer identity, refer to Destination permissions.

For more information on sinks, see the LogSink type and the projects.sinks.create API method.

How sinks work

When a log entry arrives in a project, folder, billing account, or organization, Logging compares the log entry with the queries in the sinks created in that parent. Each sink whose filter matches the log entry writes a copy of the log entry to the sink's destination.

Organization and folder sinks can optionally be permitted to export log entries from their projects. See Aggregated sinks.

Troubleshooting: A sink might not export matching log entries for the following reasons:

The sink's writer identity does not have permission to write logs to the destination. Errors occur until the sink's writer has permission to write log entries to the destination.

For information about delays in finding exported logs at your destination, see Using exported logs.

Creating sinks

This section explains how to create and modify sinks. Sinks can only export logs from the same resource that contains the sink—a particular project, folder, billing account, or organization.

Using a unique writer identity

You can use either a unique writer identity (a generated service account) or the backward-compatible shared writer identity (a common group) for your sink. For better security, use a unique writer for all new sinks. With a unique writer, the destination administrator knows that granting write permission to the identity will not allow any other sinks to write to the destination.

The following steps show you how to create or modify a sink using a unique writer identity:

Create the sink's destination, if needed.
Call the API method projects.sinks.create or projects.sinks.update to create or modify the sink. The following parameters are important for these instructions:
- Set uniqueWriterIdentity to true. When updating a sink, you can change from using a shared writer to a unique writer. If the existing sink already uses a unique writer, the updated sink uses the same writer.
  
  If you are updating a sink that already uses a unique writer, do not default the uniqueWriterIdentity parameter. The default value is false, which is not permitted in this case.
The methods return the new sink, which contains the new writer identity.
Add the new sink's writer identity to the destination's permission list, giving the writer permission to write to the destination.

Using the shared writer identity

You can use the backward-compatible, shared writer identity only if your sink is in the same project as your destination. Otherwise, the shared writer cannot get permission to write to the destination.

To create or update a sink with the shared writer, do the following:

Create the sink's destination, if needed.
Add one of the following writers—they are equivalent—to your destination's permission list, giving the writer permission to write to the destination:
- the group, cloud-logs@google.com
- the service account ,cloud-logs@system.gserviceaccount.com
Call the API method projects.sinks.create or projects.sinks.update to create or modify your sink. The following parameter is relevant to these instructions:
- Set uniqueWriterIdentity to false. If you are updating a sink, the existing sink must already be using the shared writer; you cannot change a sink from a unique writer to a shared writer.
The new sink begins exporting log entries immediately.

Access control

There are three issues relevant to controlling access to exported logs:

Permitting users or applications to create and modify sinks.
Permitting a sink's writer identity to write exported logs to a destination.
Permitting users or applications to read or modify exported logs.

Managing sinks

The ability to create and view sinks is controlled by Logging IAM roles in the sink's parent:

To create or modify sinks, you or your application must have the IAM role Logging/Logs Configuration Writer or be an Owner of the sink's parent project or other parent resource.
To view existing sinks, you or your application must have the IAM roll Logging/Logs Viewer or be a Viewer of the sink's parent project or other parent resource.

Writing to the destination

When Logging is exporting logs through a sink, it can run under different writer identities:

Writer identity	Type	When creating the sink
`cloud-logs@google.com`	group	`uniqueWriterIdentity`=false
`cloud-logs@system.gserviceaccount.com`	service account	`uniqueWriterIdentity`=false
`xxxxxxx@yyyyyy.gserviceaccount.com`	service account	`uniqueWriterIdentity`=true

The first two writers are equivalent. The third identity is a unique writer for each sink, which has the advantage of letting you export logs to a destination in a project other than the sink's parent. If the sink's parent is not a project, you must use a unique writer, since all destinations are in projects.

Once you choose a kind of writer, use it in a way appropriate to each kind of destination:

In Cloud Storage, add the writer to your destination bucket's permission list and give the writer the Storage Object Creator role.
In BigQuery, add the writer to your destination dataset's permission list and give the writer the BigQuery Data Editor role.
In Pub/Sub, add the writer to your destination topic's permission list and give the writer the Pub/Sub Publisher role.

Reading from the destination

It is the destination's responsibility to control access to exported log entries. For example, Logging prevents the deletion or modification of audit logs that are held in Logging, and can restrict access to private audit logs. When audit logs are exported, the exported copies are no longer under Logging's control.

Code samples

List your sinks

List the sinks in a project to keep track of where your logs are being exported.

C#

To learn how to install and use the client library for Logging, see the Logging Client Libraries.

View on GitHub Feedback

private void ListSinks()
{
    var sinkClient = ConfigServiceV2Client.Create();
    ProjectName projectName = new ProjectName(s_projectId);
    var listOfSinks = sinkClient.ListSinks(projectName, callSettings: _retryAWhile);
    foreach (var sink in listOfSinks)
    {
        Console.WriteLine($"{sink.Name} {sink.ToString()}");
    }
}

Go

To learn how to install and use the client library for Logging, see the Logging Client Libraries.

View on GitHub Feedback

ctx := context.Background()

var sinks []string
it := client.Sinks(ctx)
for {
	sink, err := it.Next()
	if err == iterator.Done {
		break
	}
	if err != nil {
		return nil, err
	}
	sinks = append(sinks, sink.ID)
}

Java

To learn how to install and use the client library for Logging, see the Logging Client Libraries.

View on GitHub Feedback

Page<Sink> sinks = logging.listSinks(ListOption.pageSize(100));
for (Sink sink : sinks.iterateAll()) {
  // do something with the sink
}

Node.js

To learn how to install and use the client library for Logging, see the Logging Client Libraries.

View on GitHub Feedback

// Imports the Google Cloud client library
const {Logging} = require('@google-cloud/logging');

// Creates a client
const logging = new Logging();

async function printSinkMetadata() {
  // See https://googleapis.dev/nodejs/logging/latest/Logging.html#getSinks
  const [sinks] = await logging.getSinks();
  console.log('Sinks:');
  sinks.forEach(sink => {
    console.log(sink.name);
    console.log(`  Destination: ${sink.metadata.destination}`);
    console.log(`  Filter: ${sink.metadata.filter}`);
  });
}
printSinkMetadata();

PHP

To learn how to install and use the client library for Logging, see the Logging Client Libraries.

View on GitHub Feedback

/**
 * List log sinks.
 *
 * @param string $projectId
 * @return ItemIterator<Google\Cloud\Logging\Sink>
 */
function list_sinks($projectId)
{
    $logging = new LoggingClient(['projectId' => $projectId]);
    $sinks = $logging->sinks();
    foreach ($sinks as $sink) {
        /* @var $sink \Google\Cloud\Logging\Sink */
        foreach ($sink->info() as $key => $value) {
            printf('%s:%s' . PHP_EOL,
                $key,
                is_string($value) ? $value : var_export($value, true)
            );
        }
        print PHP_EOL;
    }
}

Python

To learn how to install and use the client library for Logging, see the Logging Client Libraries.

View on GitHub Feedback

def list_sinks():
    """Lists all sinks."""
    logging_client = logging.Client()

    sinks = list(logging_client.list_sinks())

    if not sinks:
        print("No sinks.")

    for sink in sinks:
        print("{}: {} -> {}".format(sink.name, sink.filter_, sink.destination))

Ruby

To learn how to install and use the client library for Logging, see the Logging Client Libraries.

View on GitHub Feedback

require "google/cloud/logging"

logging = Google::Cloud::Logging.new

logging.sinks.each do |sink|
  puts "#{sink.name}: #{sink.filter} -> #{sink.destination}"
end

Create a sink

Create a sink in your project to establish a destination to which you can send your log entries. Before creating the sink, be sure you have granted cloud-logs@system.gserviceaccount.com permission to write logs to the destination. For more information, see Permissions and roles.

C#

To learn how to install and use the client library for Logging, see the Logging Client Libraries.

View on GitHub Feedback

        private void CreateSink(string sinkId, string logId)
        {
            var sinkClient = ConfigServiceV2Client.Create();
            CreateSinkRequest sinkRequest = new CreateSinkRequest();
            LogSink myLogSink = new LogSink();
            myLogSink.Name = sinkId;

            // This creates a sink using a Google Cloud Storage bucket 
            // named the same as the projectId.
            // This requires editing the bucket's permissions to add the Entity Group 
            // named 'cloud-logs@google.com' with 'Owner' access for the bucket.
            // If this is being run with a Google Cloud service account,
            // that account will need to be granted 'Owner' access to the Project.
            // In Powershell, use this command:
            // PS > Add-GcsBucketAcl <your-bucket-name> -Role OWNER -Group cloud-logs@google.com
            myLogSink.Destination = "storage.googleapis.com/" + s_projectId;
            LogName logName = new LogName(s_projectId, logId);
            myLogSink.Filter = $"logName={logName.ToString()}AND severity<=ERROR";
            ProjectName projectName = new ProjectName(s_projectId);
            sinkRequest.Sink = myLogSink;
            sinkClient.CreateSink(projectName, myLogSink, _retryAWhile);
            Console.WriteLine($"Created sink: {sinkId}.");
        }

Go

To learn how to install and use the client library for Logging, see the Logging Client Libraries.

View on GitHub Feedback

ctx := context.Background()
_, err := client.CreateSink(ctx, &logadmin.Sink{
	ID:          "severe-errors-to-gcs",
	Destination: "storage.googleapis.com/logsinks-bucket",
	Filter:      "severity >= ERROR",
})

Java

To learn how to install and use the client library for Logging, see the Logging Client Libraries.

View on GitHub Feedback

SinkInfo sinkInfo = SinkInfo.of(sinkName, DatasetDestination.of(datasetName));
Sink sink = logging.create(sinkInfo);

Node.js

To learn how to install and use the client library for Logging, see the Logging Client Libraries.

View on GitHub Feedback

// Imports the Google Cloud client libraries
const {Logging} = require('@google-cloud/logging');
const {Storage} = require('@google-cloud/storage');

// Creates clients
const logging = new Logging();
const storage = new Storage();

/**
 * TODO(developer): Uncomment the following lines to run the code.
 */
// const sinkName = 'Name of your sink, e.g. my-sink';
// const bucketName = 'Desination bucket, e.g. my-bucket';
// const filter = 'Optional log filer, e.g. severity=ERROR';

// The destination can be a Cloud Storage bucket, a Cloud Pub/Sub topic,
// or a BigQuery dataset. In this case, it is a Cloud Storage Bucket.
// See https://cloud.google.com/logging/docs/api/tasks/exporting-logs for
// information on the destination format.
const destination = storage.bucket(bucketName);
const sink = logging.sink(sinkName);

/**
 * The filter determines which logs this sink matches and will be exported
 * to the destination. For example a filter of 'severity>=INFO' will send
 * all logs that have a severity of INFO or greater to the destination.
 * See https://cloud.google.com/logging/docs/view/advanced_filters for more
 * filter information.
 */
const config = {
  destination: destination,
  filter: filter,
};

async function createSink() {
  // See https://googleapis.dev/nodejs/logging/latest/Sink.html#create
  await sink.create(config);
  console.log(`Created sink ${sinkName} to ${bucketName}`);
}
createSink();

PHP

To learn how to install and use the client library for Logging, see the Logging Client Libraries.

View on GitHub Feedback

/** Create a log sink.
 *
 * @param string $projectId The Google project ID.
 * @param string $sinkName The name of the sink.
 * @param string $destination The destination of the sink.
 * @param string $filterString The filter for the sink.
 */
function create_sink($projectId, $sinkName, $destination, $filterString)
{
    $logging = new LoggingClient(['projectId' => $projectId]);
    $logging->createSink(
        $sinkName,
        $destination,
        ['filter' => $filterString]
    );
    printf("Created a sink '%s'." . PHP_EOL, $sinkName);
}

Python

To learn how to install and use the client library for Logging, see the Logging Client Libraries.

View on GitHub Feedback

def create_sink(sink_name, destination_bucket, filter_):
    """Creates a sink to export logs to the given Cloud Storage bucket.

    The filter determines which logs this sink matches and will be exported
    to the destination. For example a filter of 'severity>=INFO' will send
    all logs that have a severity of INFO or greater to the destination.
    See https://cloud.google.com/logging/docs/view/advanced_filters for more
    filter information.
    """
    logging_client = logging.Client()

    # The destination can be a Cloud Storage bucket, a Cloud Pub/Sub topic,
    # or a BigQuery dataset. In this case, it is a Cloud Storage Bucket.
    # See https://cloud.google.com/logging/docs/api/tasks/exporting-logs for
    # information on the destination format.
    destination = "storage.googleapis.com/{bucket}".format(bucket=destination_bucket)

    sink = logging_client.sink(sink_name, filter_=filter_, destination=destination)

    if sink.exists():
        print("Sink {} already exists.".format(sink.name))
        return

    sink.create()
    print("Created sink {}".format(sink.name))

Ruby

To learn how to install and use the client library for Logging, see the Logging Client Libraries.

View on GitHub Feedback

require "google/cloud/logging"

logging = Google::Cloud::Logging.new
storage = Google::Cloud::Storage.new
# bucket_name = "name-of-my-storage-bucket"
bucket  = storage.create_bucket bucket_name

# Grant owner permission to Cloud Logging service
email = "cloud-logs@google.com"
bucket.acl.add_owner "group-#{email}"

# sink_name = "name-of-my-sink"
sink = logging.create_sink sink_name, "storage.googleapis.com/#{bucket.id}"
puts "#{sink.name}: #{sink.filter} -> #{sink.destination}"

Update a sink

Update a sink in a project to change its destination. For example, you might want to change the Cloud Storage bucket to which you are exporting logs monthly.

To update the sink, supply the sink's name and a new LogSink object containing the new destination. Before updating the sink, be sure you have given the sink's writer identity permission to write logs to the destination. For more information, see Permissions and roles.

C#

To learn how to install and use the client library for Logging, see the Logging Client Libraries.

View on GitHub Feedback

private void UpdateSinkLog(string sinkId, string logId)
{
    var sinkClient = ConfigServiceV2Client.Create();
    LogName logName = new LogName(s_projectId, logId);
    LogSinkName sinkName = new LogSinkName(s_projectId, sinkId);
    var sink = sinkClient.GetSink(sinkName, _retryAWhile);
    sink.Filter = $"logName={logName.ToString()}AND severity<=ERROR";
    sinkClient.UpdateSink(sinkName, sink, _retryAWhile);
    Console.WriteLine($"Updated {sinkId} to export logs from {logId}.");
}

Go

To learn how to install and use the client library for Logging, see the Logging Client Libraries.

View on GitHub Feedback

ctx := context.Background()
_, err := client.UpdateSink(ctx, &logadmin.Sink{
	ID:          "severe-errors-to-gcs",
	Destination: "storage.googleapis.com/logsinks-new-bucket",
	Filter:      "severity >= INFO",
})

Java

To learn how to install and use the client library for Logging, see the Logging Client Libraries.

View on GitHub Feedback

SinkInfo sinkInfo =
    SinkInfo.newBuilder(sinkName, DatasetDestination.of(datasetName))
        .setVersionFormat(SinkInfo.VersionFormat.V2)
        .setFilter("severity>=ERROR")
        .build();
Sink sink = logging.update(sinkInfo);

Node.js

To learn how to install and use the client library for Logging, see the Logging Client Libraries.

View on GitHub Feedback

// Imports the Google Cloud client library
const {Logging} = require('@google-cloud/logging');

// Creates a client
const logging = new Logging();

/**
 * TODO(developer): Uncomment the following lines to run the code.
 */
// const sinkName = 'Name of sink to update, e.g. my-sink';
// const filter = 'New filter for the sink, e.g. severity >= WARNING';

const sink = logging.sink(sinkName);

/**
 * The filter determines which logs this sink matches and will be exported
 * to the destination. For example a filter of 'severity>=INFO' will send
 * all logs that have a severity of INFO or greater to the destination.
 * See https://cloud.google.com/logging/docs/view/advanced_filters for more
 * filter information.
 */
const metadataInfo = {
  filter: filter,
};

async function updateSink() {
  // See https://googleapis.dev/nodejs/logging/latest/Sink.html#setMetadata
  const [metadata] = await sink.setMetadata(metadataInfo);
  console.log(`Sink ${sinkName} updated.`, metadata);
}
updateSink();

PHP

To learn how to install and use the client library for Logging, see the Logging Client Libraries.

View on GitHub Feedback

/**
 * Update a log sink.
 *
 * @param string $projectId
 * @param string sinkName
 * @param string $filterString
 */
function update_sink($projectId, $sinkName, $filterString)
{
    $logging = new LoggingClient(['projectId' => $projectId]);
    $sink = $logging->sink($sinkName);
    $sink->update(['filter' => $filterString]);
    printf("Updated a sink '%s'." . PHP_EOL, $sinkName);
}

Python

To learn how to install and use the client library for Logging, see the Logging Client Libraries.

View on GitHub Feedback

def update_sink(sink_name, filter_):
    """Changes a sink's filter.

    The filter determines which logs this sink matches and will be exported
    to the destination. For example a filter of 'severity>=INFO' will send
    all logs that have a severity of INFO or greater to the destination.
    See https://cloud.google.com/logging/docs/view/advanced_filters for more
    filter information.
    """
    logging_client = logging.Client()
    sink = logging_client.sink(sink_name)

    sink.reload()

    sink.filter_ = filter_
    print("Updated sink {}".format(sink.name))
    sink.update()

Ruby

To learn how to install and use the client library for Logging, see the Logging Client Libraries.

View on GitHub Feedback

require "google/cloud/logging"

logging = Google::Cloud::Logging.new
storage = Google::Cloud::Storage.new
# bucket_name = "name-of-my-storage-bucket"
bucket  = storage.create_bucket bucket_name
# sink_name = "name-of-my-sink"
sink    = logging.sink sink_name

sink.destination = "storage.googleapis.com/#{bucket.id}"

sink.save
puts "Updated sink destination for #{sink.name} to #{sink.destination}"

Delete a sink

Delete a sink from your project if you no longer want to export log entries through that sink.

C#

To learn how to install and use the client library for Logging, see the Logging Client Libraries.

View on GitHub Feedback

private void DeleteSink(string sinkId)
{
    var sinkClient = ConfigServiceV2Client.Create();
    LogSinkName sinkName = new LogSinkName(s_projectId, sinkId);
    sinkClient.DeleteSink(sinkName, _retryAWhile);
    Console.WriteLine($"Deleted {sinkId}.");
}

Go

To learn how to install and use the client library for Logging, see the Logging Client Libraries.

View on GitHub Feedback

ctx := context.Background()
if err := client.DeleteSink(ctx, "severe-errors-to-gcs"); err != nil {
	return err
}

Java

To learn how to install and use the client library for Logging, see the Logging Client Libraries.

View on GitHub Feedback

boolean deleted = logging.deleteSink(sinkName);
if (deleted) {
  // the sink was deleted
} else {
  // the sink was not found
}

Node.js

To learn how to install and use the client library for Logging, see the Logging Client Libraries.

View on GitHub Feedback

// Imports the Google Cloud client library
const {Logging} = require('@google-cloud/logging');

// Creates a client
const logging = new Logging();

/**
 * TODO(developer): Uncomment the following line to run the code.
 */
// const sinkName = 'Name of sink to delete, e.g. my-sink';

const sink = logging.sink(sinkName);

async function deleteSink() {
  // See https://googleapis.dev/nodejs/logging/latest/Sink.html#delete
  await sink.delete();
  console.log(`Sink ${sinkName} deleted.`);
}
deleteSink();

PHP

To learn how to install and use the client library for Logging, see the Logging Client Libraries.

View on GitHub Feedback

/** Delete a log sink.
 *
 * @param string $projectId The Google project ID.
 * @param string $sinkName The name of the sink.
 */
function delete_sink($projectId, $sinkName)
{
    $logging = new LoggingClient(['projectId' => $projectId]);
    $logging->sink($sinkName)->delete();
    printf("Deleted a sink '%s'." . PHP_EOL, $sinkName);
}

Python

To learn how to install and use the client library for Logging, see the Logging Client Libraries.

View on GitHub Feedback

def delete_sink(sink_name):
    """Deletes a sink."""
    logging_client = logging.Client()
    sink = logging_client.sink(sink_name)

    sink.delete()

    print("Deleted sink {}".format(sink.name))

Ruby

To learn how to install and use the client library for Logging, see the Logging Client Libraries.

View on GitHub Feedback

require "google/cloud/logging"

logging = Google::Cloud::Logging.new

# sink_name = "name-of-my-sink"
sink = logging.sink sink_name
sink.delete
puts "Deleted sink: #{sink.name}"