Exporting Logs in the API

This guide explains how to use the Stackdriver Logging API v2 to export your logs out of Stackdriver Logging. To instead use the Logs Viewer to set up exports, see Logs Viewer: Exporting Logs. To use the command-line SDK to set up exports, see SDK: Exporting Logs. To find and use exported logs, see Using Exported Logs.

Introduction to sinks

You control the export of log entries by creating sinks in Stackdriver Logging. Sinks contain the following information:

  • A sink identifier that names the sink within the parent. Examples

    "my-first-sink"
    
  • A parent resource—a project, folder, billing account, or organization where in which sink is created. The sink can only export logs belonging to the parent resource. The parents have the following formats:

    "projects/[PROJECT_ID]"
    "organizations/[ORGANIZATION_ID]"
    "folders/[FOLDER_ID]"
    "billingAccounts/[BILLING_ACCOUNT_ID]"
    
  • An include children parameter. This is only relevant to sinks within organizations. See Aggregated Exports.

  • A logs filter to select which log entries to export. For more information, see Advanced Logs Filters. The following example matches all log entries with severity WARNING or higher from a specified Compute Engine VM instance:

    "resource.type = gce_instance AND \
       resource.labels.instance_id = "1428064241541024269" AND \
       severity >= WARNING"
    
  • A destination to which the log entries are exported. This can be a Cloud Storage bucket, a BigQuery dataset, or a Cloud Pub/Sub topic, all of which must belong to a GCP project. The destinations have the following format:

    "storage.googleapis.com/[BUCKET_ID]"
    "bigquery.googleapis.com/projects/[PROJECT_ID]/datasets/[DATASET_ID]"
    "pubsub.googleapis.com/projects/[PROJECT_ID]/topics/[TOPIC_ID]"
    

    You can export logs to destinations in any project. The sink creator doesn't require access to the destination. Only the sink's writer service account, explained next, requires access.

  • A writer identity that is a service account name. The destination's owner must give this service account permission to write to the destination. When exporting logs, Stackdriver Logging adopts this identity for authorization. For increased security, new sinks get their own unique service account:

    [GENERATED_ID_1]@[GENERATED_ID_2].iam.gserviceaccount.com
    

    Older sinks might use a shared service account for this writer:

    cloud-logs@system.gserviceaccount.com
    

    For more information on managing the writer identity, see Destination permissions.

For more information on sinks, see the LogSink type and the projects.sinks.create API method.

How sinks work

When a log entry arrives in a project, folder, billing account, or organization, Stackdriver Logging compares the log entry with the filters in the sinks created in that parent. Each sink whose filter matches the log entry writes a copy of the log entry to the sink's destination.

Organization and folder sinks can optionally be permitted to export log entries from their projects. See Aggregated Exports.

Troubleshooting: A sink might not export matching log entries for the following reasons:

  • The sink's writer identity does not have permission to write logs to the destination. Errors occur until the sink's writer has permission to write log entries to the destination.

For information about delays in finding exported logs at your destination, see Exported logs availability

Creating sinks

This section explains how to create and modify sinks. Sinks can only export logs from the same resource that contains the sink—a particular project, folder, billing account, or organization.

Using a unique writer identity

You can use either a unique writer identity (a generated service account) or the backward-compatible shared writer identity (a common group) for your sink. For better security, you should prefer to use a unique writer for all new sinks. With a unique writer, the destination administrator knows that granting write permission to the identity will not allow any other sinks to write to the destination.

You must use a unique writer if your sink's parent is not a project or if you are exporting to a destination in a different project than the one containing the sink. The shared writer works only when the destination and the sink are in the same project.

The following steps show you how to create or modify a sink:

  1. Create the sink's destination, if needed.

  2. Call the API method projects.sinks.create or projects.sinks.update to create or modify the sink. The following parameters are important for these instructions:

    • Set uniqueWriterIdentity to true. When updating a sink, you can change from using a shared writer to a unique writer. If the existing sink already uses a unique writer, the updated sink uses the same writer.

      If you are updating a sink that aleady uses a unique writer, do not default the uniqueWriterIdentity parameter. The default value is false, which is not permitted in this case.

    The methods return the new sink, which contains the new writer identity.

  3. Add the new sink's writer identity to the destination's permission list, giving the writer permission to write to the destination.

Using the shared writer identity

In new sinks, using the shared writer identity is discouraged.

You can use the backward-compatible, shared writer identity only if your sink is in the same project as your destination. Otherwise, the shared writer cannot get permission to write to the destination.

To create or update a sink with the shared writer, do the following:

  1. Create the sink's destination, if needed.

  2. Add one of the following writers—they are equivalent—to your destination's permission list, giving the writer permission to write to the destination:

    • the group, cloud-logs@google.com
    • the service account ,cloud-logs@system.gserviceaccount.com
  3. Call the API method projects.sinks.create or projects.sinks.update to create or modify your sink. The following parameter is relevant to these instructions:

    • Set uniqueWriterIdentity to false. If you are updating a sink, the existing sink must already be using the shared writer; you cannot change a sink from a unique writer to a shared writer.

    The new sink begins exporting log entries immediately.

Access control

There are three issues relevant to controlling access to exported logs:

  • Permitting users or applications to create and modify sinks.
  • Permitting a sink's writer identity to write exported logs to a destination.
  • Permitting users or applications to read or modify exported logs.

Managing sinks

The ability to create and view sinks is controlled by Logging IAM roles in the sink's parent:

  • To create or modify sinks, you or your application must have the IAM role Logging/Logs Cofiguration Writer or be an Owner of the sink's parent project or other parent resource.

  • To view existing sinks, you or your application must have the IAM roll Logging/Logs Viewer or be a Viewer of the sink's parent project or other parent resource.

Writing to the destination

When Stackdriver Logging is exporting logs through a sink, it can run under different writer identities:

Writer identity Type When creating the sink
cloud-logs@google.com group uniqueWriterIdentity=false
cloud-logs@system.gserviceaccount.com service account uniqueWriterIdentity=false
xxxxxxx@yyyyyy.gserviceaccount.com service account uniqueWriterIdentity=true

The first two writers are equivalent. The third identity is a unique writer for each sink, which has the advantage of letting you export logs to a destination in a project other than the sink's parent. If the sink's parent is not a project, you must use a unique writer, since all destinations are in projects.

Once you choose a kind of writer, use it in a way appropriate to each kind of destination:

  • In Cloud Storage, add the writer to your destination bucket's permission list and give the writer Owner access, or a more limited IAM role, if available.

  • In BigQuery, add the writer to your destination dataset's permission list and give the writer Can edit access.

  • In Cloud Pub/Sub, add the writer to your destination topic's permission list and give the writer PubSub/Publisher or Editor access.

Reading from the destination

It is the destination's responsibility to control access to exported log entries. For example, Stackdriver Logging prevents the deletion or modification of audit logs that are held in Stackdriver Logging, and can restrict access to private audit logs. When audit logs are exported, the exported copies are no longer under Stackdriver Logging's control.

Migrating sinks from the v1 API to the v2 API

Because the Stackdriver Logging API v1 has been deprecated, all new sinks should export log entries in the Stackdriver Logging API v2 format, which is the default for new sinks. To migrate your v1 sinks to v2, see Migration to Stackdriver Logging v2.

Code samples

The following code samples use the Stackdriver Logging API v2.

List your sinks

List the sinks in a project to keep track of where your logs are being exported.

C#

For more on installing and creating a Stackdriver Logging client, refer to Stackdriver Logging Client Libraries.

private void ListSinks()
{
    var sinkClient = ConfigServiceV2Client.Create();
    ProjectName projectName = new ProjectName(s_projectId);
    var listOfSinks = sinkClient.ListSinks(ParentNameOneof.From(projectName),
        callSettings: RetryAWhile);
    foreach (var sink in listOfSinks)
    {
        Console.WriteLine($"{sink.Name} {sink.ToString()}");
    }
}

Go

For more on installing and creating a Stackdriver Logging client, refer to Stackdriver Logging Client Libraries.

ctx := context.Background()

var sinks []string
it := client.Sinks(ctx)
for {
	sink, err := it.Next()
	if err == iterator.Done {
		break
	}
	if err != nil {
		return nil, err
	}
	sinks = append(sinks, sink.ID)
}

Java

For more on installing and creating a Stackdriver Logging client, refer to Stackdriver Logging Client Libraries.

Page<Sink> sinks = logging.listSinks(ListOption.pageSize(100));
for (Sink sink : sinks.iterateAll()) {
  // do something with the sink
}

Node.js

For more on installing and creating a Stackdriver Logging client, refer to Stackdriver Logging Client Libraries.

// Imports the Google Cloud client library
const Logging = require('@google-cloud/logging');

// Instantiates a client
const logging = Logging();

// See https://googlecloudplatform.github.io/google-cloud-node/#/docs/logging/latest/logging?method=getSinks
logging.getSinks()
  .then((results) => {
    const sinks = results[0];

    console.log('Sinks:');
    sinks.forEach((sink) => {
      console.log(sink.name);
      console.log(`  Destination: ${sink.metadata.destination}`);
      console.log(`  Filter: ${sink.metadata.filter}`);
    });
  })
  .catch((err) => {
    console.error('ERROR:', err);
  });

PHP

For more on installing and creating a Stackdriver Logging client, refer to Stackdriver Logging Client Libraries.

/**
 * List log sinks.
 *
 * @param string $projectId
 * @return ItemIterator<Google\Cloud\Logging\Sink>
 */
function list_sinks($projectId)
{
    $logging = new LoggingClient(['projectId' => $projectId]);
    return $logging->sinks();
}

Python

For more on installing and creating a Stackdriver Logging client, refer to Stackdriver Logging Client Libraries.

def list_sinks():
    """Lists all sinks."""
    logging_client = logging.Client()

    sinks = list(logging_client.list_sinks())

    if not sinks:
        print('No sinks.')

    for sink in sinks:
        print('{}: {} -> {}'.format(sink.name, sink.filter_, sink.destination))

Ruby

For more on installing and creating a Stackdriver Logging client, refer to Stackdriver Logging Client Libraries.

require "google/cloud/logging"

logging = Google::Cloud::Logging.new project: "my-gcp-project-id"

logging.sinks.each do |sink|
  puts "#{sink.name}: #{sink.filter} -> #{sink.destination}"
end

Create a sink

Create a sink in your project to establish a destination to which you can send your log entries. Before creating the sink, be sure you have granted cloud-logs@system.gserviceaccount.com permission to write logs to the destination. See [Setting Stackdriver Logging permissions for writing exported logs][log-permissions].

C#

For more on installing and creating a Stackdriver Logging client, refer to Stackdriver Logging Client Libraries.

        private void CreateSink(string sinkId, string logId)
        {
            var sinkClient = ConfigServiceV2Client.Create();
            CreateSinkRequest sinkRequest = new CreateSinkRequest();
            LogSink myLogSink = new LogSink();
            myLogSink.Name = sinkId;

            // This creates a sink using a Google Cloud Storage bucket 
            // named the same as the projectId.
            // This requires editing the bucket's permissions to add the Entity Group 
            // named 'cloud-logs@google.com' with 'Owner' access for the bucket.
            // If this is being run with a Google Cloud service account,
            // that account will need to be granted 'Owner' access to the Project.
            // In Powershell, use this command:
            // PS > Add-GcsBucketAcl <your-bucket-name> -Role OWNER -Group cloud-logs@google.com
            myLogSink.Destination = "storage.googleapis.com/" + s_projectId;
            LogName logName = new LogName(s_projectId, logId);
            myLogSink.Filter = $"logName={logName.ToString()}AND severity<=ERROR";
            ProjectName projectName = new ProjectName(s_projectId);
            sinkRequest.Sink = myLogSink;
            sinkClient.CreateSink(ParentNameOneof.From(projectName), myLogSink, RetryAWhile);
            Console.WriteLine($"Created sink: {sinkId}.");
        }

Go

For more on installing and creating a Stackdriver Logging client, refer to Stackdriver Logging Client Libraries.

ctx := context.Background()
_, err := client.CreateSink(ctx, &logadmin.Sink{
	ID:          "severe-errors-to-gcs",
	Destination: "storage.googleapis.com/logsinks-bucket",
	Filter:      "severity >= ERROR",
})

Java

For more on installing and creating a Stackdriver Logging client, refer to Stackdriver Logging Client Libraries.

SinkInfo sinkInfo = SinkInfo.of(sinkName, DatasetDestination.of(datasetName));
Sink sink = logging.create(sinkInfo);

Node.js

For more on installing and creating a Stackdriver Logging client, refer to Stackdriver Logging Client Libraries.

// Imports the Google Cloud client libraries
const Logging = require('@google-cloud/logging');
const Storage = require('@google-cloud/storage');

// Instantiates clients
const logging = Logging();
const storage = Storage();

// The name of the sink, e.g. "my-sink"
// const sinkName = "my-sink";

// The destination bucket, e.g. "my-bucket"
// const bucketName = "my-bucket";

// An option log filter, e.g. "severity=ERROR"
// const filter = "severity=ERROR";

// The destination can be a Cloud Storage bucket, a Cloud Pub/Sub topic,
// or a BigQuery dataset. In this case, it is a Cloud Storage Bucket.
// See https://cloud.google.com/logging/docs/api/tasks/exporting-logs for
// information on the destination format.
const destination = storage.bucket(bucketName);
const sink = logging.sink(sinkName);

/**
 * The filter determines which logs this sink matches and will be exported
 * to the destination. For example a filter of 'severity>=INFO' will send
 * all logs that have a severity of INFO or greater to the destination.
 * See https://cloud.google.com/logging/docs/view/advanced_filters for more
 * filter information.
 */
const config = {
  destination: destination,
  filter: filter
};

// See https://googlecloudplatform.github.io/google-cloud-node/#/docs/logging/latest/logging/sink?method=create
sink.create(config)
  .then(() => {
    console.log(`Created sink ${sinkName} to ${bucketName}`);
  })
  .catch((err) => {
    console.error('ERROR:', err);
  });

PHP

For more on installing and creating a Stackdriver Logging client, refer to Stackdriver Logging Client Libraries.

/** Create a log sink.
 *
 * @param string $projectId The Google project ID.
 * @param string $sinkName The name of the sink.
 * @param string $destination The destination of the sink.
 * @param string $filterString The filter for the sink.
 */
function create_sink($projectId, $sinkName, $destination, $filterString)
{
    $logging = new LoggingClient(['projectId' => $projectId]);
    $logging->createSink(
        $sinkName,
        $destination,
        ['filter' => $filterString]
    );
}

Python

For more on installing and creating a Stackdriver Logging client, refer to Stackdriver Logging Client Libraries.

def create_sink(sink_name, destination_bucket, filter_):
    """Creates a sink to export logs to the given Cloud Storage bucket.

    The filter determines which logs this sink matches and will be exported
    to the destination. For example a filter of 'severity>=INFO' will send
    all logs that have a severity of INFO or greater to the destination.
    See https://cloud.google.com/logging/docs/view/advanced_filters for more
    filter information.
    """
    logging_client = logging.Client()

    # The destination can be a Cloud Storage bucket, a Cloud Pub/Sub topic,
    # or a BigQuery dataset. In this case, it is a Cloud Storage Bucket.
    # See https://cloud.google.com/logging/docs/api/tasks/exporting-logs for
    # information on the destination format.
    destination = 'storage.googleapis.com/{bucket}'.format(
        bucket=destination_bucket)

    sink = logging_client.sink(
        sink_name,
        filter_,
        destination)

    if sink.exists():
        print('Sink {} already exists.'.format(sink.name))
        return

    sink.create()
    print('Created sink {}'.format(sink.name))

Ruby

For more on installing and creating a Stackdriver Logging client, refer to Stackdriver Logging Client Libraries.

require "google/cloud/logging"

logging = Google::Cloud::Logging.new project: "my-gcp-project-id"
storage = Google::Cloud::Storage.new project: "my-gcp-project-id"
bucket  = storage.create_bucket "my-logs-bucket"

# Grant owner permission to Cloud Logging service
email = "cloud-logs@google.com"
bucket.acl.add_owner "group-#{email}"

sink = logging.create_sink "my-sink", "storage.googleapis.com/#{bucket.id}"

Update a sink

Update a sink in a project to change its destination. For example, you might want to change the Cloud Storage bucket to which you are exporting logs monthly.

To update the sink, supply the sink's name and a new LogSink object containing the new destination. Before updating the sink, be sure you have given the sink's writer identity permission to write logs to the destination. See [Setting Stackdriver Logging permissions for writing exported logs][log-permissions].

C#

For more on installing and creating a Stackdriver Logging client, refer to Stackdriver Logging Client Libraries.

private void UpdateSinkLog(string sinkId, string logId)
{
    var sinkClient = ConfigServiceV2Client.Create();
    LogName logName = new LogName(s_projectId, logId);
    SinkName sinkName = new SinkName(s_projectId, sinkId);
    var sink = sinkClient.GetSink(SinkNameOneof.From(sinkName), RetryAWhile);
    sink.Filter = $"logName={logName.ToString()}AND severity<=ERROR";
    sinkClient.UpdateSink(SinkNameOneof.From(sinkName), sink, RetryAWhile);
    Console.WriteLine($"Updated {sinkId} to export logs from {logId}.");
}

Go

For more on installing and creating a Stackdriver Logging client, refer to Stackdriver Logging Client Libraries.

ctx := context.Background()
_, err := client.UpdateSink(ctx, &logadmin.Sink{
	ID:          "severe-errors-to-gcs",
	Destination: "storage.googleapis.com/logsinks-new-bucket",
	Filter:      "severity >= INFO",
})

Java

For more on installing and creating a Stackdriver Logging client, refer to Stackdriver Logging Client Libraries.

SinkInfo sinkInfo = SinkInfo.newBuilder(sinkName, DatasetDestination.of(datasetName))
    .setVersionFormat(SinkInfo.VersionFormat.V2)
    .setFilter("severity>=ERROR")
    .build();
Sink sink = logging.update(sinkInfo);

Node.js

For more on installing and creating a Stackdriver Logging client, refer to Stackdriver Logging Client Libraries.

// Imports the Google Cloud client library
const Logging = require('@google-cloud/logging');

// Instantiates a client
const logging = Logging();

// The sink to update, e.g. "my-sink"
// const sinkName = "my-sink";

// The the filter for the sink, e.g. "severity >= WARNING"
// const filter = "severity >= WARNING";

const sink = logging.sink(sinkName);

/**
 * The filter determines which logs this sink matches and will be exported
 * to the destination. For example a filter of 'severity>=INFO' will send
 * all logs that have a severity of INFO or greater to the destination.
 * See https://cloud.google.com/logging/docs/view/advanced_filters for more
 * filter information.
 */
const metadata = {
  filter: filter
};

// See https://googlecloudplatform.github.io/google-cloud-node/#/docs/logging/latest/logging/sink?method=setMetadata
sink.setMetadata(metadata)
  .then((results) => {
    console.log(`Sink ${sinkName} updated.`);
  })
  .catch((err) => {
    console.error('ERROR:', err);
  });

PHP

For more on installing and creating a Stackdriver Logging client, refer to Stackdriver Logging Client Libraries.

/**
 * Update a log sink.
 *
 * @param string $projectId
 * @param string sinkName
 * @param string $filterString
 */
function update_sink($projectId, $sinkName, $filterString)
{
    $logging = new LoggingClient(['projectId' => $projectId]);
    $sink = $logging->sink($sinkName);
    $sink->update(['filter' => $filterString]);
}

Python

For more on installing and creating a Stackdriver Logging client, refer to Stackdriver Logging Client Libraries.

def update_sink(sink_name, filter_):
    """Changes a sink's filter.

    The filter determines which logs this sink matches and will be exported
    to the destination. For example a filter of 'severity>=INFO' will send
    all logs that have a severity of INFO or greater to the destination.
    See https://cloud.google.com/logging/docs/view/advanced_filters for more
    filter information.
    """
    logging_client = logging.Client()
    sink = logging_client.sink(sink_name)

    sink.reload()

    sink.filter_ = filter_
    print('Updated sink {}'.format(sink.name))
    sink.update()
    # [END update]

Ruby

For more on installing and creating a Stackdriver Logging client, refer to Stackdriver Logging Client Libraries.

require "google/cloud/logging"

logging = Google::Cloud::Logging.new project: "my-gcp-project-id"
storage = Google::Cloud::Storage.new project: "my-gcp-project-id"
bucket  = storage.bucket "new-destination-bucket"
sink    = logging.sink "my-sink"

sink.destination = "storage.googleapis.com/#{bucket.id}"

sink.save

Delete a sink

Delete a sink from your project if you no longer want to export log entries through that sink.

C#

For more on installing and creating a Stackdriver Logging client, refer to Stackdriver Logging Client Libraries.

private void DeleteSink(string sinkId)
{
    var sinkClient = ConfigServiceV2Client.Create();
    SinkName sinkName = new SinkName(s_projectId, sinkId);
    sinkClient.DeleteSink(SinkNameOneof.From(sinkName), RetryAWhile);
    Console.WriteLine($"Deleted {sinkId}.");
}

Go

For more on installing and creating a Stackdriver Logging client, refer to Stackdriver Logging Client Libraries.

ctx := context.Background()
if err := client.DeleteSink(ctx, "severe-errors-to-gcs"); err != nil {
	return err
}

Java

For more on installing and creating a Stackdriver Logging client, refer to Stackdriver Logging Client Libraries.

boolean deleted = logging.deleteSink(sinkName);
if (deleted) {
  // the sink was deleted
} else {
  // the sink was not found
}

Node.js

For more on installing and creating a Stackdriver Logging client, refer to Stackdriver Logging Client Libraries.

// Imports the Google Cloud client library
const Logging = require('@google-cloud/logging');

// Instantiates a client
const logging = Logging();

// The sink to delete, e.g. "my-sink"
// const sinkName = "my-sink";

const sink = logging.sink(sinkName);

// See https://googlecloudplatform.github.io/google-cloud-node/#/docs/logging/latest/logging/sink?method=delete
sink.delete()
  .then((results) => {
    console.log(`Sink ${sinkName} deleted.`);
  })
  .catch((err) => {
    console.error('ERROR:', err);
  });

PHP

For more on installing and creating a Stackdriver Logging client, refer to Stackdriver Logging Client Libraries.

/** Delete a log sink.
 *
 * @param string $projectId The Google project ID.
 * @param string $sinkName The name of the sink.
 */
function delete_sink($projectId, $sinkName)
{
    $logging = new LoggingClient(['projectId' => $projectId]);
    $logging->sink($sinkName)->delete();
}

Python

For more on installing and creating a Stackdriver Logging client, refer to Stackdriver Logging Client Libraries.

def delete_sink(sink_name):
    """Deletes a sink."""
    logging_client = logging.Client()
    sink = logging_client.sink(sink_name)

    sink.delete()

    print('Deleted sink {}'.format(sink.name))

Ruby

For more on installing and creating a Stackdriver Logging client, refer to Stackdriver Logging Client Libraries.

require "google/cloud/logging"

logging = Google::Cloud::Logging.new project: "my-gcp-project-id"

sink = logging.sink "my-sink"
sink.delete

Monitor your resources on the go

Get the Google Cloud Console app to help you manage your projects.

Send feedback about...

Stackdriver Logging