Exporting Logs in the API

This guide explains how to use the Stackdriver Logging API to export your logs out of Stackdriver Logging. To instead use the Logs Viewer to set up exports, see Logs Viewer: Exporting Logs. To use the command-line SDK to set up exports, see SDK: Exporting Logs. To find and use exported logs, see Using Exported Logs.

Introduction to sinks

You control the export of log entries by creating sinks in Stackdriver Logging. Sinks contain the following information:

• A sink identifier that names the sink within the parent. Examples

  "my-first-sink"
  

• A parent resource—a project, folder, billing account, or organization where in which sink is created. The sink can only export logs belonging to the parent resource. The parents have the following formats:

  "projects/[PROJECT_ID]"
  "organizations/[ORGANIZATION_ID]"
  "folders/[FOLDER_ID]"
  "billingAccounts/[BILLING_ACCOUNT_ID]"
  

• An include children parameter. This is only relevant to sinks within organizations. See Aggregated Exports.

• A logs filter to select which log entries to export. For more information, see Advanced Logs Filters. The following example matches all log entries with severity WARNING or higher from a specified Compute Engine VM instance:

  "resource.type = gce_instance AND \
     resource.labels.instance_id = "1428064241541024269" AND \
     severity >= WARNING"
  

• A destination to which the log entries are exported. This can be a Cloud Storage bucket, a BigQuery dataset, or a Cloud Pub/Sub topic, all of which must belong to a GCP project. The destinations have the following format:

  "storage.googleapis.com/[BUCKET_ID]"
  "bigquery.googleapis.com/projects/[PROJECT_ID]/datasets/[DATASET_ID]"
  "pubsub.googleapis.com/projects/[PROJECT_ID]/topics/[TOPIC_ID]"
  

  You can export the log entries to a destination in any GCP project. The sink's source and the destination do not have to be within the same organization.
  
  It is not required that you have writer access to the export destination. Only the sink's writer service account, explained next, requires access.

• A writer identity that is a service account name. The destination's owner must give this service account permission to write to the destination. When exporting logs, Stackdriver Logging adopts this identity for authorization. For increased security, new sinks get their own unique service account:

  [GENERATED_ID_1]@[GENERATED_ID_2].iam.gserviceaccount.com
  

  Older sinks might use a shared service account for this writer:

  [e-mail]
  

  For more information on managing the writer identity, see Destination permissions.

For more information on sinks, see the LogSink type and the projects.sinks.create API method.

How sinks work

When a log entry arrives in a project, folder, billing account, or organization, Stackdriver Logging compares the log entry with the filters in the sinks created in that parent. Each sink whose filter matches the log entry writes a copy of the log entry to the sink's destination.

Organization and folder sinks can optionally be permitted to export log entries from their projects. See Aggregated Exports.

Caution: A sink can export any kind of log entry, including Admin Activity audit logs and private Data Access audit logs. For more information, see Access control on this page.

Troubleshooting: A sink might not export matching log entries for the following reasons:

• The sink's writer identity does not have permission to write logs to the destination. Errors occur until the sink's writer has permission to write log entries to the destination.

For information about delays in finding exported logs at your destination, see Using exported logs.

Creating sinks

This section explains how to create and modify sinks. Sinks can only export logs from the same resource that contains the sink—a particular project, folder, billing account, or organization.

Using a unique writer identity

You can use either a unique writer identity (a generated service account) or the backward-compatible shared writer identity (a common group) for your sink. For better security, you should prefer to use a unique writer for all new sinks. With a unique writer, the destination administrator knows that granting write permission to the identity will not allow any other sinks to write to the destination.

You must use a unique writer if your sink's parent is not a project or if you are exporting to a destination in a different project than the one containing the sink. The shared writer works only when the destination and the sink are in the same project.

The following steps show you how to create or modify a sink:

1. Create the sink's destination, if needed.

2. Call the API method projects.sinks.create or projects.sinks.update to create or modify the sink. The following parameters are important for these instructions:

   • Set uniqueWriterIdentity to true. When updating a sink, you can change from using a shared writer to a unique writer. If the existing sink already uses a unique writer, the updated sink uses the same writer.

     If you are updating a sink that already uses a unique writer, do not default the uniqueWriterIdentity parameter. The default value is false, which is not permitted in this case.

   The methods return the new sink, which contains the new writer identity.

3. Add the new sink's writer identity to the destination's permission list, giving the writer permission to write to the destination.

Using the shared writer identity

In new sinks, using the shared writer identity is discouraged.

You can use the backward-compatible, shared writer identity only if your sink is in the same project as your destination. Otherwise, the shared writer cannot get permission to write to the destination.

To create or update a sink with the shared writer, do the following:

1. Create the sink's destination, if needed.

2. Add one of the following writers—they are equivalent—to your destination's permission list, giving the writer permission to write to the destination:

   • the group, [e-mail]
   • the service account ,[e-mail]

3. Call the API method projects.sinks.create or projects.sinks.update to create or modify your sink. The following parameter is relevant to these instructions:

   • Set uniqueWriterIdentity to false. If you are updating a sink, the existing sink must already be using the shared writer; you cannot change a sink from a unique writer to a shared writer.

   The new sink begins exporting log entries immediately.

Access control

There are three issues relevant to controlling access to exported logs:

• Permitting users or applications to create and modify sinks.
• Permitting a sink's writer identity to write exported logs to a destination.
• Permitting users or applications to read or modify exported logs.

Managing sinks

The ability to create and view sinks is controlled by Logging IAM roles in the sink's parent:

• To create or modify sinks, you or your application must have the IAM role Logging/Logs Configuration Writer or be an Owner of the sink's parent project or other parent resource.

• To view existing sinks, you or your application must have the IAM roll Logging/Logs Viewer or be a Viewer of the sink's parent project or other parent resource.

Caution: All sinks can export audit logs, including private Data Access audit logs, without extra permissions.

Writing to the destination

When Stackdriver Logging is exporting logs through a sink, it can run under different writer identities:

Writer identity	Type	When creating the sink
[e-mail]	group	uniqueWriterIdentity=false
[e-mail]	service account	uniqueWriterIdentity=false
[e-mail]	service account	uniqueWriterIdentity=true

The first two writers are equivalent. The third identity is a unique writer for each sink, which has the advantage of letting you export logs to a destination in a project other than the sink's parent. If the sink's parent is not a project, you must use a unique writer, since all destinations are in projects.

Once you choose a kind of writer, use it in a way appropriate to each kind of destination:

• In Cloud Storage, add the writer to your destination bucket's permission list and give the writer the Storage Object Creator role.

• In BigQuery, add the writer to your destination dataset's permission list and give the writer the BigQuery Data Editor role.

• In Cloud Pub/Sub, add the writer to your destination topic's permission list and give the writer the Pub/Sub Publisher role.

Reading from the destination

It is the destination's responsibility to control access to exported log entries. For example, Stackdriver Logging prevents the deletion or modification of audit logs that are held in Stackdriver Logging, and can restrict access to private audit logs. When audit logs are exported, the exported copies are no longer under Stackdriver Logging's control.

Code samples

List your sinks

List the sinks in a project to keep track of where your logs are being exported.

C#

View on GitHub Feedback

private void ListSinks()
{
    var sinkClient = ConfigServiceV2Client.Create();
    ProjectName projectName = new ProjectName(s_projectId);
    var listOfSinks = sinkClient.ListSinks(ParentNameOneof.From(projectName),
        callSettings: _retryAWhile);
    foreach (var sink in listOfSinks)
    {
        Console.WriteLine($"{sink.Name} {sink.ToString()}");
    }
}

Go

View on GitHub Feedback

ctx := context.Background()

var sinks []string
it := client.Sinks(ctx)
for {
	sink, err := it.Next()
	if err == iterator.Done {
		break
	}
	if err != nil {
		return nil, err
	}
	sinks = append(sinks, sink.ID)
}

Java

View on GitHub Feedback

Page<Sink> sinks = logging.listSinks(ListOption.pageSize(100));
for (Sink sink : sinks.iterateAll()) {
  // do something with the sink
}

Node.js

View on GitHub Feedback

// Imports the Google Cloud client library
const {Logging} = require('@google-cloud/logging');

// Creates a client
const logging = new Logging();

// See
// https://googlecloudplatform.github.io/google-cloud-node/#/docs/logging/latest/logging?method=getSinks
const [sinks] = await logging.getSinks();
console.log('Sinks:');
sinks.forEach(sink => {
  console.log(sink.name);
  console.log(`  Destination: ${sink.metadata.destination}`);
  console.log(`  Filter: ${sink.metadata.filter}`);
});

PHP

View on GitHub Feedback

/**
 * List log sinks.
 *
 * @param string $projectId
 * @return ItemIterator<Google\Cloud\Logging\Sink>
 */
function list_sinks($projectId)
{
    $logging = new LoggingClient(['projectId' => $projectId]);
    $sinks = $logging->sinks();
    foreach ($sinks as $sink) {
        /* @var $sink \Google\Cloud\Logging\Sink */
        foreach ($sink->info() as $key => $value) {
            print "$key:$value\n";
        }
        print PHP_EOL;
    }
}

Python

View on GitHub Feedback

def list_sinks():
    """Lists all sinks."""
    logging_client = logging.Client()

    sinks = list(logging_client.list_sinks())

    if not sinks:
        print('No sinks.')

    for sink in sinks:
        print('{}: {} -> {}'.format(sink.name, sink.filter_, sink.destination))

Ruby

View on GitHub Feedback

require "google/cloud/logging"

logging = Google::Cloud::Logging.new project: "my-gcp-project-id"

logging.sinks.each do |sink|
  puts "#{sink.name}: #{sink.filter} -> #{sink.destination}"
end

Create a sink

Create a sink in your project to establish a destination to which you can send your log entries. Before creating the sink, be sure you have granted [e-mail] permission to write logs to the destination. For more information, see Permissions and Roles.

C#

View on GitHub Feedback

        private void CreateSink(string sinkId, string logId)
        {
            var sinkClient = ConfigServiceV2Client.Create();
            CreateSinkRequest sinkRequest = new CreateSinkRequest();
            LogSink myLogSink = new LogSink();
            myLogSink.Name = sinkId;

            // This creates a sink using a Google Cloud Storage bucket 
            // named the same as the projectId.
            // This requires editing the bucket's permissions to add the Entity Group 
            // named '[e-mail]' with 'Owner' access for the bucket.
            // If this is being run with a Google Cloud service account,
            // that account will need to be granted 'Owner' access to the Project.
            // In Powershell, use this command:
            // PS > Add-GcsBucketAcl <your-bucket-name> -Role OWNER -Group [e-mail]
            myLogSink.Destination = "storage.googleapis.com/" + s_projectId;
            LogName logName = new LogName(s_projectId, logId);
            myLogSink.Filter = $"logName={logName.ToString()}AND severity<=ERROR";
            ProjectName projectName = new ProjectName(s_projectId);
            sinkRequest.Sink = myLogSink;
            sinkClient.CreateSink(ParentNameOneof.From(projectName), myLogSink, _retryAWhile);
            Console.WriteLine($"Created sink: {sinkId}.");
        }

Go

View on GitHub Feedback

ctx := context.Background()
_, err := client.CreateSink(ctx, &logadmin.Sink{
	ID:          "severe-errors-to-gcs",
	Destination: "storage.googleapis.com/logsinks-bucket",
	Filter:      "severity >= ERROR",
})

Java

View on GitHub Feedback

SinkInfo sinkInfo = SinkInfo.of(sinkName, DatasetDestination.of(datasetName));
Sink sink = logging.create(sinkInfo);

Node.js

View on GitHub Feedback

// Imports the Google Cloud client libraries
const {Logging} = require('@google-cloud/logging');
const {Storage} = require('@google-cloud/storage');

// Creates clients
const logging = new Logging();
const storage = new Storage();

/**
 * TODO(developer): Uncomment the following lines to run the code.
 */
// const sinkName = 'Name of your sink, e.g. my-sink';
// const bucketName = 'Desination bucket, e.g. my-bucket';
// const filter = 'Optional log filer, e.g. severity=ERROR';

// The destination can be a Cloud Storage bucket, a Cloud Pub/Sub topic,
// or a BigQuery dataset. In this case, it is a Cloud Storage Bucket.
// See https://cloud.google.com/logging/docs/api/tasks/exporting-logs for
// information on the destination format.
const destination = storage.bucket(bucketName);
const sink = logging.sink(sinkName);

/**
 * The filter determines which logs this sink matches and will be exported
 * to the destination. For example a filter of 'severity>=INFO' will send
 * all logs that have a severity of INFO or greater to the destination.
 * See https://cloud.google.com/logging/docs/view/advanced_filters for more
 * filter information.
 */
const config = {
  destination: destination,
  filter: filter,
};

// See
// https://googlecloudplatform.github.io/google-cloud-node/#/docs/logging/latest/logging/sink?method=create
await sink.create(config);
console.log(`Created sink ${sinkName} to ${bucketName}`);

PHP

View on GitHub Feedback

/** Create a log sink.
 *
 * @param string $projectId The Google project ID.
 * @param string $sinkName The name of the sink.
 * @param string $destination The destination of the sink.
 * @param string $filterString The filter for the sink.
 */
function create_sink($projectId, $sinkName, $destination, $filterString)
{
    $logging = new LoggingClient(['projectId' => $projectId]);
    $logging->createSink(
        $sinkName,
        $destination,
        ['filter' => $filterString]
    );
    printf("Created a sink '%s'." . PHP_EOL, $sinkName);
}

Python

View on GitHub Feedback

def create_sink(sink_name, destination_bucket, filter_):
    """Creates a sink to export logs to the given Cloud Storage bucket.

    The filter determines which logs this sink matches and will be exported
    to the destination. For example a filter of 'severity>=INFO' will send
    all logs that have a severity of INFO or greater to the destination.
    See https://cloud.google.com/logging/docs/view/advanced_filters for more
    filter information.
    """
    logging_client = logging.Client()

    # The destination can be a Cloud Storage bucket, a Cloud Pub/Sub topic,
    # or a BigQuery dataset. In this case, it is a Cloud Storage Bucket.
    # See https://cloud.google.com/logging/docs/api/tasks/exporting-logs for
    # information on the destination format.
    destination = 'storage.googleapis.com/{bucket}'.format(
        bucket=destination_bucket)

    sink = logging_client.sink(
        sink_name,
        filter_,
        destination)

    if sink.exists():
        print('Sink {} already exists.'.format(sink.name))
        return

    sink.create()
    print('Created sink {}'.format(sink.name))

Ruby

View on GitHub Feedback

require "google/cloud/logging"

logging = Google::Cloud::Logging.new project: "my-gcp-project-id"
storage = Google::Cloud::Storage.new project: "my-gcp-project-id"
bucket  = storage.create_bucket "my-logs-bucket"

# Grant owner permission to Cloud Logging service
email = "[e-mail]"
bucket.acl.add_owner "group-#{email}"

sink = logging.create_sink "my-sink", "storage.googleapis.com/#{bucket.id}"

Update a sink

Update a sink in a project to change its destination. For example, you might want to change the Cloud Storage bucket to which you are exporting logs monthly.

To update the sink, supply the sink's name and a new LogSink object containing the new destination. Before updating the sink, be sure you have given the sink's writer identity permission to write logs to the destination. For more information, see Permissions and Roles.

C#

View on GitHub Feedback

private void UpdateSinkLog(string sinkId, string logId)
{
    var sinkClient = ConfigServiceV2Client.Create();
    LogName logName = new LogName(s_projectId, logId);
    SinkName sinkName = new SinkName(s_projectId, sinkId);
    var sink = sinkClient.GetSink(SinkNameOneof.From(sinkName), _retryAWhile);
    sink.Filter = $"logName={logName.ToString()}AND severity<=ERROR";
    sinkClient.UpdateSink(SinkNameOneof.From(sinkName), sink, _retryAWhile);
    Console.WriteLine($"Updated {sinkId} to export logs from {logId}.");
}

Go

View on GitHub Feedback

ctx := context.Background()
_, err := client.UpdateSink(ctx, &logadmin.Sink{
	ID:          "severe-errors-to-gcs",
	Destination: "storage.googleapis.com/logsinks-new-bucket",
	Filter:      "severity >= INFO",
})

Java

View on GitHub Feedback

SinkInfo sinkInfo =
    SinkInfo.newBuilder(sinkName, DatasetDestination.of(datasetName))
        .setVersionFormat(SinkInfo.VersionFormat.V2)
        .setFilter("severity>=ERROR")
        .build();
Sink sink = logging.update(sinkInfo);

Node.js

View on GitHub Feedback

// Imports the Google Cloud client library
const {Logging} = require('@google-cloud/logging');

// Creates a client
const logging = new Logging();

/**
 * TODO(developer): Uncomment the following lines to run the code.
 */
// const sinkName = 'Name of sink to update, e.g. my-sink';
// const filter = 'New filter for the sink, e.g. severity >= WARNING';

const sink = logging.sink(sinkName);

/**
 * The filter determines which logs this sink matches and will be exported
 * to the destination. For example a filter of 'severity>=INFO' will send
 * all logs that have a severity of INFO or greater to the destination.
 * See https://cloud.google.com/logging/docs/view/advanced_filters for more
 * filter information.
 */
const metadataInfo = {
  filter: filter,
};

// See
// https://googlecloudplatform.github.io/google-cloud-node/#/docs/logging/latest/logging/sink?method=setMetadata
const [metadata] = await sink.setMetadata(metadataInfo);
console.log(`Sink ${sinkName} updated.`, metadata);

PHP

View on GitHub Feedback

/**
 * Update a log sink.
 *
 * @param string $projectId
 * @param string sinkName
 * @param string $filterString
 */
function update_sink($projectId, $sinkName, $filterString)
{
    $logging = new LoggingClient(['projectId' => $projectId]);
    $sink = $logging->sink($sinkName);
    $sink->update(['filter' => $filterString]);
    printf("Updated a sink '%s'." . PHP_EOL, $sinkName);
}

Python

View on GitHub Feedback

def update_sink(sink_name, filter_):
    """Changes a sink's filter.

    The filter determines which logs this sink matches and will be exported
    to the destination. For example a filter of 'severity>=INFO' will send
    all logs that have a severity of INFO or greater to the destination.
    See https://cloud.google.com/logging/docs/view/advanced_filters for more
    filter information.
    """
    logging_client = logging.Client()
    sink = logging_client.sink(sink_name)

    sink.reload()

    sink.filter_ = filter_
    print('Updated sink {}'.format(sink.name))
    sink.update()

Ruby

View on GitHub Feedback

require "google/cloud/logging"

logging = Google::Cloud::Logging.new project: "my-gcp-project-id"
storage = Google::Cloud::Storage.new project: "my-gcp-project-id"
bucket  = storage.bucket "new-destination-bucket"
sink    = logging.sink "my-sink"

sink.destination = "storage.googleapis.com/#{bucket.id}"

sink.save

Delete a sink

Delete a sink from your project if you no longer want to export log entries through that sink.

C#

View on GitHub Feedback

private void DeleteSink(string sinkId)
{
    var sinkClient = ConfigServiceV2Client.Create();
    SinkName sinkName = new SinkName(s_projectId, sinkId);
    sinkClient.DeleteSink(SinkNameOneof.From(sinkName), _retryAWhile);
    Console.WriteLine($"Deleted {sinkId}.");
}

Go

View on GitHub Feedback

ctx := context.Background()
if err := client.DeleteSink(ctx, "severe-errors-to-gcs"); err != nil {
	return err
}

Java

View on GitHub Feedback

boolean deleted = logging.deleteSink(sinkName);
if (deleted) {
  // the sink was deleted
} else {
  // the sink was not found
}

Node.js

View on GitHub Feedback

// Imports the Google Cloud client library
const {Logging} = require('@google-cloud/logging');

// Creates a client
const logging = new Logging();

/**
 * TODO(developer): Uncomment the following line to run the code.
 */
// const sinkName = 'Name of sink to delete, e.g. my-sink';

const sink = logging.sink(sinkName);

// See
// https://googlecloudplatform.github.io/google-cloud-node/#/docs/logging/latest/logging/sink?method=delete
await sink.delete();
console.log(`Sink ${sinkName} deleted.`);

PHP

View on GitHub Feedback

/** Delete a log sink.
 *
 * @param string $projectId The Google project ID.
 * @param string $sinkName The name of the sink.
 */
function delete_sink($projectId, $sinkName)
{
    $logging = new LoggingClient(['projectId' => $projectId]);
    $logging->sink($sinkName)->delete();
    printf("Deleted a sink '%s'." . PHP_EOL, $sinkName);
}

Python

View on GitHub Feedback

def delete_sink(sink_name):
    """Deletes a sink."""
    logging_client = logging.Client()
    sink = logging_client.sink(sink_name)

    sink.delete()

    print('Deleted sink {}'.format(sink.name))

Ruby

View on GitHub Feedback

require "google/cloud/logging"

logging = Google::Cloud::Logging.new project: "my-gcp-project-id"

sink = logging.sink "my-sink"
sink.delete