Migration to the Stackdriver Logging API v2

Schedule:

  • October 20, 2016: The v1 API was deprecated.

  • May 5, 2017: The v1 API was shut down except for the method that writes v1 log entries.

    The following continue to work: Applications that write logs, Stackdriver Logging agents, existing v1 export sinks, and existing v1 logs-based metrics.

  • May 15, 2017: Stackdriver begins the migration of existing v1 sinks and logs-based metrics. All sinks and metrics that you do not migrate are scheduled for migration to v2.

  • October 1, 2017: The v1 API will be shut down. The projects.logs.entries.write method stops working. Stackdriver Logging agents that have not been updated stop working.

To learn more about the v2 API, see Basic Concepts, API Reference, and Client Libraries.

Migration to-do list

Go to the Logs Viewer page

If you use the Logs Viewer:
You are already using v2 log entries and interfaces in the Logs Viewer. You do not have to take any action.
If you export logs:
Look at the Exports page of the Logs Viewer. If you see Log exports using the v1 format with some sinks listed, then you must convert those v1 sinks to v2 or allow Stackdriver to convert them for you. In the right-side menu of each v1 sink, there are options to View as v2 and Schedule v2 conversion. For details about the changes, see Export sinks.
If you use logs-based metrics:
Look at the Logs-based Metrics page of the Logs Viewer. If any of your metrics are listed as "Version 1," then you must convert them to v2 or allow Stackdriver to convert them for you. There is a Convert to v2 option in the right-hand menu of each v1 metric. See also Log filters.
If you have installed logging agents on your VM instances:
Update your agents to the most recent version, so that they no longer use the v1 API. See Upgrading the agent.
If you write log entries using the v1 API method, projects.logs.entries.write:
You must change to the v2 method entries.write before the v1 method is turned down. Conversion involves changing from writing v1 log entries to writing v2 log entries. See Log entry changes.
If you have code that uses other v1 API methods:
Your code no longer works, because the v1 API (other than projects.logs.entries.write) has already been turned down. See API methods.
If you use the Cloud SDK (the gcloud command):
The gcloud beta logging command does not expose the differences between the v1 and v2 APIs for the most part. Any changes needed are minimal. The command continues to show v1 sinks until they have all been migrated.

V2 changes

This section focuses on changes that affect the migration of v1 features to v2. Additional features in the v2 API are not discussed.

Monitored resources

Monitored resources are a new feature in v2 log entries, replacing "logging services."

Log entries in v2 specify their origin with a resource field rather than metadata.serviceName. For example, the following table shows how a Compute Engine VM instance is represented in v1 and v2 log entries:

# v1 LogEntry                                       | v2 LogEntry
-------------------------------------------------   | --------------------------------------------
{                                                   | {
  metadata: {                                       |   resource: {
    serviceName: "compute.googleapis.com",          |     type: "gce_instance",
    zone: "us-central1-a",                          |     labels: {
    labels: {                                       |       zone: "us-central1-a"
      compute.googleapis.com/resource_type:         |       instance_id: "15543007601548826368"
                                        "instance"  |     }
      compute.googleapis.com/resource_id: \         |   }
                            "15543007601548826368"  | }
    }                                               |
  }                                                 |
}                                                   |

For a list of monitored resource types, see Monitored Resource List.

Log entries

The v2 API has a new format for log entries, which affects both readers and writers.

The v1 projects.logs.entries.write API method is supported for a short time, but all v1 log entries entering Stackdriver Logging are being converted to v2 log entries. The Logs Viewer displays log entries only in the v2 format.

The following table shows the corresponding information in v1 and v2 log entry types:

v1 LogEntry fields v2 LogEntry fields
log = "[LOG_ID]" logName = "projects/[PROJECT_ID]/logs/[LOG_ID]"
metadata.serviceName = "[SERVICE_NAME]" resource.type = "[RESOURCE_TYPE]"
metadata.labels = {[SERVICE_AND_USER_LABELS]} labels = {[USER_LABELS]}
resource.labels = {[RESOURCE_LABELS]}
metadata.timestamp timestamp
metadata.severity severity
metadata.projectId (value is now in logName)
metadata.region (value is now in resource.labels if needed)
metadata.zone (value is now in resource.labels if needed)
metadata.userId (the user ID is not present in v2 log entries)
insertId insertId
httpRequest httpRequest
operation operation
protoPayload protoPayload
textPayload textPayload
structPayload jsonPayload

Following are the v1 and v2 versions of the same log entry. The order of fields may differ from what is shown.

# v1 LogEntry                                       | v2 LogEntry
-------------------------------------------------   | --------------------------------------------
{                                                   | {
  textPayload: "Jun 23 18:17:01 my-gce-..."         |   textPayload: "Jun 23 18:17:01 my-gce-..."
  log: "syslog"                                     |   logName: "projects/my-gcp-project-id/logs/syslog"
  insertId: "q396nyg1lrcyra"                        |   insertId: "q396nyg1lrcyra"
  metadata: {                                       |   resource: {
    projectId: "my-gcp-project-id"                  |     type: "gce_instance"
    serviceName: "compute.googleapis.com"           |     labels: {
    zone: "us-central1-a"                           |       zone: "us-central1-a"
    labels: {                                       |       instance_id: "15543007601548826368"
      compute.googleapis.com/resource_type: \
                                        "instance"  |     }
      compute.googleapis.com/resource_id: \
                            "15543007601548826368"  |   },
    }                                               |   labels1: {
    timestamp: "2016-06-23T18:17:01.000Z"           |
  }                                                 |   },
}                                                   |   timestamp: "2016-06-23T18:17:01.000Z"
                                                    | }

Notes:
1 When a v1 log entry is converted to v2, information in the labels and serviceName fields of the v1 entry is used to create the v2 resource object. The v1 labels information is not needed any more in most cases. However, Stackdriver Logging's built-in converter typically copies the labels to the v2 log entry, although this example does not show it.

API methods

The following v1 API methods do not exist in the v2 API:

gcloud beta logging

Although the gcloud beta logging command command gives you access to the Stackdriver Logging API, the command is mostly unaffected by the difference between the v1 and v2 APIs. Use the online help for the command if you have questions.

Log filters

Advanced log filters reference v2-format log entry components rather than v1-format components. See Converting filters on this page.

Logs-based metrics

include an advanced logs filter, which is different in v2. See Converting filters on this page.

To migrate a v1 logs-based metric, you replace it with a new v2 logs-based metric that counts the same log entries.

Export sinks

Export sinks have changed considerably in the v2 API:

  • V1 log sinks, which export all log entries in a log, no longer exist. Migrate to v2 sinks that include a v2 filter that restricts the log entries to those from a particular log name.

  • V1 log service sinks, which export all log entries from a log service, no longer exist. Migrate to v2 sinks that include a v2 filter that restricts the log entries to those from a set of monitored resource types that belong to a service.

  • V1 project sinks correspond to v2 sinks. They contain an advanced logs filter that must be in v2 format.

  • v2 sinks export their log entries in the v2 format. Applications that process your exported log entries must change to use the new format. See Log entry changes.

For more details, see Converting sinks in the Logs Viewer and Converting sinks manually.

Migration how-tos

This section contains detailed instructions for some migration tasks.

Converting sinks the Logs Viewer

If you created your v1 sinks in the Logs Viewer, the easiest way to convert them is to schedule a conversion in the Logs Viewer:

  1. In the v2 Logs Viewer, go to the Exports page.
  2. Select a v1 sink, and click Convert V1 Sink at the top of the page.
  3. Click Schedule Conversion, choose a date, and click Schedule.

The conversion works as follows: A new v2 sink is created to export the same log entries as the v1 sink. At 00:00:00 UTC, at the beginning of the chosen date, your new v2 sink starts exporting log entries in the v2 format and your v1 sink stops exporting log entries in the v1 format. The v2 sink exports to the same destination as the v1 sink, but 00:00:00 UTC is also the time that exported logs are switched to a new table in BigQuery and a new object in Cloud Storage, so there is a separation within the destination.

Be aware of these issues in scheduling conversions:

  • If you are exporting to a Cloud Pub/Sub topic, then subscribers must be prepared to detect and handle the format change. The switchover will happen in the middle of a continuous stream of log entries.

  • Stackdriver Logging prohibits exporting logs in different formats to the same destination at the same time. If a planned conversion would result in two sinks exporting different formats to the same destination, the scheduling will fail. You will have to manually separate the v1 sink destinations, or else create v2 sinks exporting to a different destination.

If the Logs Viewer fails to schedule your conversion, you will have to convert your v1 sinks manually.

Converting sinks manually

If your v1 sinks were not created in the Logs Viewer, you must do the conversion manually, as described in this section. At a high level, the steps are:

  1. Create a v2 advanced logs filter that specifies the same set of log entries that the v1 sink exported.
  2. Decide whether the v2 sink should export to the same destination or a new destination.
  3. Decide when to start the v2 export and when stop the v1 export. If they export to different destinations, then you can run them in parallel.
  4. Create the new v2 sink with the new filter and a suitable start time.
  5. Modify the v1 sink to set an end time, if desired. If you plan to stop the v1 sink as you start the v2 sink, the v1 sink's end time must be the same as the v2 sink's start time.

All the steps except the filter conversion should be evident by looking at the v2 API sink methods. The following section addresses filters.

Converting filters

This section is for users who have a v1 export sink or a v1 logs-based metric. To migrate to v2, you must create a new v2 filter to specify the log entries to export or to count.

Log names: If your filter uses a log name, change its format:

v1 filter

log = "compute.googleapis.com/activity_log"

v2 filter

logName = "projects/[MY-PROJECT-ID]/logs/compute.googleapis.com%2Factivity_log"

Notice that v2 log names use URL-encoded log identifiers, like compute.googleapis.com%2Factivity_log.

Log services: If your filter refers to a v1 log service, change it to a resource type comparison in your v2 filter:

v1 filter

metadata.serviceName = "gkecluster.googleapis.com"

v2 filter

resource.type = ("gke_nodepool" OR "gke_cluster")

For the set of monitored resource types corresponding to a v1 service, see Mapping services to resources.

Some v1 log services map to more than one resource type. You must include all the mapped resource types in the v2 filter to be sure of matching all the log entries specified by the v1 filter:

Other log entry fields: Look for any additional comparisons in your v1 filter that should be converted to v2 format. See Log entry changes. For example, consider v1's structPayload and metadata.severity fields:

v1 filter

structPayload.cat = ("siamese" OR "shorthair") AND
metadata.severity >= ERROR

v2 filter

jsonPayload.cat = ("siamese" OR "shorthair") AND
severity >= ERROR

Monitor your resources on the go

Get the Google Cloud Console app to help you manage your projects.

Send feedback about...

Stackdriver Logging