Query and view logs in Log Analytics

This document describes how to query and view logs that are stored in log buckets that are upgraded to use Log Analytics. You can query logs in these buckets by using SQL, which lets you filter and aggregate your logs. For information about the Log Analytics capabilities of Cloud Logging, see Log Analytics overview.

When you upgrade a log bucket to use Log Analytics, you don't restrict your access to Logs Explorer. You can continue to troubleshoot issues and view individual log entries in these buckets by using the Logs Explorer.

For information about how to upgrade a log bucket to use Log Analytics, see the following documents:

Before you begin

  • Ensure that you have a log bucket that has been upgraded to use Log Analytics. The Logs Storage page lists your log buckets and their configuration information.

    In the navigation panel of the Google Cloud console, select Logging, and then select Logs Storage:

    Go to Logs Storage

  • To get the permissions that you need to load the Log Analytics page, run queries, and view logs, ask your administrator to grant you the following IAM roles on your project:

    For more information about granting roles, see Manage access.

    You might also be able to get the required permissions through custom roles or other predefined roles.

    The permissions that you need to view log entries and run queries on the Log Analytics page are the same as those that you need to view logs on the Logs Explorer page. For information about additional roles that you need to query views on user-defined buckets or to query the _AllLogs view of the _Default log bucket, see Cloud Logging roles.

  • To get the permissions that you need to view linked BigQuery datasets for log buckets, ask your administrator to grant you the Log Link Accessor (roles/logging.linkViewer) IAM role on your project.

  • To get the permissions that you need to view your data in BigQuery, ask your administrator to grant you the following IAM roles on your project:

Query a log view

When you are troubleshooting a problem, you might want to count the log entries with a field that match a pattern or compute average latency for HTTP request. You can perform these actions by running a SQL query on a log view.

To issue a SQL query to a view, do the following:

  1. In the navigation panel of the Google Cloud console, select Logging, and then select Log Analytics:

    Go to Log Analytics

  2. In the Log views list, find the view, and then select Query. The Query pane is populated with a default query, which includes the name of the table that is queried.

    You can also enter a query in the Query pane, or edit a displayed query.

    To set the time range of your query, use the time-range selector or add a WHERE clause that specifies the timestamp field. We recommend that you use the time-range selector to specify the time range.

    If you specify a timestamp in your query, then that timestamp overrides the selected time range in the time-range selector and the time-range selector is disabled. To use the time-range selector, remove timestamp expressions from the WHERE clause in your query.

  3. In the toolbar, click Run query.

    The query is executed and the result of the query is shown in the Results tab.

    You can use the toolbar options to format your query, clear the query, and open the BigQuery SQL reference documentation.

Display the schema of a view

The schema of a log view defines its structure and the data type for each field. This information is important to you because it determines how you construct your queries. For example, suppose you want to compute the average latency of HTTP requests. You need to know how to access the latency field and whether it is stored as an integer like 100 or stored as a string like "100". When the latency data is stored as a string, the query must cast the value to a numeric value before computing an average.

When the data type of a column is JSON, the schema doesn't list the fields available for that column. For example, a log entry can have a field with the name of json_payload. When a log bucket is upgraded to use Log Analytics, that field is mapped to a column with a data type of JSON. The schema doesn't indicate the child fields of the column. That is, you can't use the schema to determine if json_payload.url is a valid reference.

To identify the schema for a view, do the following:

  1. In the navigation panel of the Google Cloud console, select Logging, and then select Log Analytics:

    Go to Log Analytics

  2. In the Log views list, find the view, and then select the name of the view.

    The schema for the table is displayed. You can use the Filter field to locate specific fields. You can't modify the schema.

Save a query

All queries that you run are automatically saved for 30 days and are accessible by selecting the Recent tab on the Log Analytics page. You can search, view, run, and share the queries that are listed on the Recent tab.

If you want to keep a query available for future use, annotate it with information that is useful to you, or let teammates view and run your query, then save the query. You can search and sort your saved queries by their name, their description, and their visibility label. You can also edit and delete these queries. Queries that you save are retained until you delete them.

You can save 10,000 queries per Google Cloud project.

Console

To save a query, do the following:

  1. In the navigation panel of the Google Cloud console, select Logging, and then select Log Analytics:

    Go to Log Analytics

  2. Populate the Query pane with a query.

    You can populate the Query pane by entering a new query, by selecting a query from the Recent tab, or by selecting a query from the Saved tab.

    When the query in the Query pane is valid, the Save is enabled.

  3. Click Save and complete the Name and Description fields. The values you set for these fields are shown on the Saved tab.

  4. Optional: To let everyone with access to the Log Analytics page for the Google Cloud project view and run your saved query, enable the Share with project toggle.

    By default, this toggle is disabled and the visibility is restricted to you.

  5. Click Save query.

  6. Optional: To view, sort, and run saved queries that are visible to you, select the Saved tab.

    You can sort and filter your saved queries by their name, description, and visibility label. You can also filter by the contents of the query.

You can edit and delete queries that you created by using options on the Saved tab:

  • To edit a query, click More Options and select Edit. You can modify the values for the Name and Description fields; however, the query itself can't be modified.

  • To delete a saved query, click More Options and select Delete.

API

To save a query by using the Logging API, use the savedQueries.create method. For more information about this method, its parameters, and the response data, see the reference page for savedQueries.create.

You can execute the savedQueries.create method by using the APIs Explorer widget on the method's reference page. For Log Analytics queries, you must specify the opsAnalyticsQuery field. The following example illustrates a sample request body, which contains an instance of SavedQuery:

{
  "parent": "projects/my-project/locations/global"
  "saved_query":
  {
     "ops_analytics_query":
     {
        "sql_query_text" :
           "SELECT
           timestamp, log_name, severity, json_payload, resource, labels
           FROM
           `TABLE`
           WHERE
           timestamp > TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 1 HOUR)
           ORDER BY timestamp ASC
           LIMIT 100"
     }
     "visibility": "PRIVATE"
  }
}

Share a query

Console

When troubleshooting a problem, or when you see anomalous results, you might want to share a query and its results with a teammate. When you are viewing query results on the Log Analytics page, you can copy a URL that, when opened, displays the query you ran and its results.

To share a query and results with a teammate, do the following:

  1. In the navigation panel of the Google Cloud console, select Logging, and then select Log Analytics:

    Go to Log Analytics

  2. Populate the Query pane with a query and then click Run query.

    You can populate the Query pane by entering a new query, by selecting a query from the Recent tab, or by selecting a query from the Saved tab.

  3. Click Share link.

  4. Send the link to your teammate.

    When your teammate opens the link, the Log Analytics page is opened. This page displays the query that you ran and the results of the query.

    To open the URL successfully, your teammate's Identity and Access Management role on the Google Cloud project must include the permissions required to view Logging pages.

API

You can use the Logging API to create a shared query by using the savedQueries.Create method, and specifying a value of SHARED in the visibility field.

View and run recent or saved queries

To view or re-run a query, select the Recent tab on the Log Analytics page and find the query:

  • To run the query, click Run.
  • To view the query, use the options in the More Options menu.

To view, edit, or run a saved query, select the Saved tab on the Log Analytics page and find the query:

  • To run the query, click Run.
  • To edit, view, or delete the query, use the options in the More Options menu.

Query logs by using BigQuery

When you have a log bucket that has been upgraded to use Log Analytics, you can also view your data in BigQuery by creating a linked BigQuery dataset. With this configuration, you can use BigQuery to join your log data, which is accessible through the linked dataset, with other business data. For information about how to create a linked dataset, see Grant BigQuery view access to a log bucket.

To query a linked dataset, do the following:

  1. In the navigation panel of the Google Cloud console, select Logging, and then select Log Analytics:

    Go to Log Analytics

  2. In the Log views list, find the view, and then select Query. The Query pane is populated with a default query.

    You can also enter a query in the Query pane, or edit a displayed query.

  3. In the toolbar, click Run in BigQuery.

    The BigQuery Studio page opens. The FROM statement of the query is modified to specify the path to the view on the linked dataset by using the BigQuery Table path syntax.

    You can also edit the displayed query.

  4. Click Run query.

What's next