This document describes how to query and view logs that are stored in log buckets that are upgraded to use Log Analytics. You can query logs in these buckets by using SQL, which lets you filter and aggregate your logs. For information about the Log Analytics capabilities of Cloud Logging, see Log Analytics overview.
When you upgrade a log bucket to use Log Analytics, you don't restrict your access to Logs Explorer. You can continue to troubleshoot issues and view individual log entries in these buckets by using the Logs Explorer.
For information about how to upgrade a log bucket to use Log Analytics, see the following documents:
Before you begin
-
To get the permissions that you need to use Log Analytics and to perform activities like creating linked datasets, ask your administrator to grant you the following IAM roles on your project:
-
To query the
_Required
and_Default
log buckets: Logs Viewer (roles/logging.viewer
) -
To query custom log buckets:
Logs View Accessor (
roles/logging.viewAccessor
) -
(Optional) To create linked BigQuery datasets:
Logs Configuration Writer (
roles/logging.configWriter
) -
(Optional) To view linked BigQuery datasets:
Log Link Accessor (
roles/logging.linkViewer
) -
(Optional) To run queries in BigQuery:
BigQuery Job User (
roles/bigquery.jobUser
) -
(Optional) To view data in BigQuery:
BigQuery Data Viewer (
roles/bigquery.dataViewer
) -
(Optional) To save charts to dashboards:
Monitoring Editor (
roles/monitoring.editor
)
For more information about granting roles, see Manage access.
You might also be able to get the required permissions through custom roles or other predefined roles.
-
To query the
For the log views that you want to query, go to the Logs Storage page and verify that the log buckets that store those log views are upgraded to use Log Analytics. If necessary, upgrade the log bucket.
In the Google Cloud console, go to the Logs Storage page:
If you use the search bar to find this page, then select the result whose subheading is Logging.
Optional: If you want to query your log data by using a BigQuery dataset, then create a linked BigQuery dataset.
Query a log view
When you are troubleshooting a problem, you might want to count the log entries with a field that match a pattern or compute average latency for HTTP request. You can perform these actions by running a SQL query on a log view.
To issue a SQL query to a view, do the following:
-
In the Google Cloud console, go to the Log Analytics page:
If you use the search bar to find this page, then select the result whose subheading is Logging.
In the Log views list, find the view, and then select Query. The Query pane is populated with a default query, which includes the name of the table that is queried.
You can also enter a query in the Query pane, or edit a displayed query.
To set the time range of your query, use the time-range selector or add a
WHERE
clause that specifies thetimestamp
field. We recommend that you use the time-range selector to specify the time range.If you specify a timestamp in your query, then that timestamp overrides the selected time range in the time-range selector and the time-range selector is disabled. To use the time-range selector, remove timestamp expressions from the
WHERE
clause in your query.In the toolbar, click Run query.
The query is executed and the result of the query is shown in the Results tab.
You can use the toolbar options to format your query, clear the query, and open the BigQuery SQL reference documentation.
Display the schema of a view
The schema of a log view defines its structure and the data type
for each field. This information is important to you because it determines
how you construct your queries. For example, suppose you want to compute the
average latency of HTTP requests. You need to know how to access the latency
field and whether it is stored as an integer like 100
or stored as a
string like "100"
. When the latency data is stored as a string, the query
must cast the value to a numeric value before computing an average.
When the data type of a column is JSON, the schema doesn't list the fields
available for that column. For example, a log entry can have a
field with the name of json_payload
. When a log bucket is upgraded to use
Log Analytics, that field is mapped to a column with a data type of JSON.
The schema doesn't indicate the child fields of the column. That is, you
can't use the schema to determine if json_payload.url
is a valid reference.
To identify the schema for a view, do the following:
-
In the Google Cloud console, go to the Log Analytics page:
If you use the search bar to find this page, then select the result whose subheading is Logging.
In the Log views list, find the view, and then select the name of the view.
The schema for the table is displayed. You can use the Filter field to locate specific fields. You can't modify the schema.
Save a query
All queries that you run are automatically saved for 30 days and are accessible by selecting the Recent tab on the Log Analytics page. You can search, view, run, and share the queries that are listed on the Recent tab.
If you want to keep a query available for future use, annotate it with information that is useful to you, or let teammates view and run your query, then save the query. You can search and sort your saved queries by their name, their description, and their visibility label. You can also edit and delete these queries. Queries that you save are retained until you delete them.
You can save 10,000 queries per Google Cloud project.
Console
To save a query, do the following:
-
In the Google Cloud console, go to the Log Analytics page:
If you use the search bar to find this page, then select the result whose subheading is Logging.
Populate the Query pane with a query.
You can populate the Query pane by entering a new query, by selecting a query from the Recent tab, or by selecting a query from the Saved tab.
When the query in the Query pane is valid, the
Save is enabled.Click
Save and complete the Name and Description fields. The values you set for these fields are shown on the Saved tab.Optional: To let everyone with access to the Log Analytics page for the Google Cloud project view and run your saved query, enable the Share with project toggle.
By default, this toggle is disabled and the visibility is restricted to you.
Click Save query.
Optional: To view, sort, and run saved queries that are visible to you, select the Saved tab.
You can sort and filter your saved queries by their name, description, and visibility label. You can also filter by the contents of the query.
You can edit and delete queries that you created by using options on the Saved tab:
To edit a query, click more_vert More Options and select Edit. You can modify the values for the Name and Description fields; however, the query itself can't be modified.
To delete a saved query, click more_vert More Options and select Delete.
API
To save a query by using the Logging API, use the
savedQueries.create
method. For more information about this method, its
parameters, and the response data, see the reference page for
savedQueries.create
.
You can execute the savedQueries.create
method by using the
APIs Explorer widget on the method's reference page. For
Log Analytics queries, you must specify the opsAnalyticsQuery
field. The
following example illustrates a sample request body, which contains an
instance of SavedQuery
:
{ "parent": "projects/my-project/locations/global" "saved_query": { "ops_analytics_query": { "sql_query_text" : "SELECT timestamp, log_name, severity, json_payload, resource, labels FROM `TABLE` WHERE timestamp > TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 1 HOUR) ORDER BY timestamp ASC LIMIT 100" } "visibility": "PRIVATE" } }
Share a query
Console
When troubleshooting a problem, or when you see anomalous results, you might want to share a query and its results with a teammate. When you are viewing query results on the Log Analytics page, you can copy a URL that, when opened, displays the query you ran and its results.
To share a query and results with a teammate, do the following:
-
In the Google Cloud console, go to the Log Analytics page:
If you use the search bar to find this page, then select the result whose subheading is Logging.
Populate the Query pane with a query and then click Run query.
You can populate the Query pane by entering a new query, by selecting a query from the Recent tab, or by selecting a query from the Saved tab.
Click link Share link.
Send the link to your teammate.
When your teammate opens the link, the Log Analytics page is opened. This page displays the query that you ran and the results of the query.
To open the URL successfully, your teammate's Identity and Access Management role on the Google Cloud project must include the permissions required to view Logging pages.
API
You can use the Logging API to create a shared query by
using the savedQueries.Create
method, and specifying a value of SHARED
in
the visibility
field.
View and run recent or saved queries
To view or re-run a query, select the Recent tab on the Log Analytics page and find the query:
- To run the query, click Run.
- To view the query, use the options in the more_vert More Options menu.
To view, edit, or run a saved query, select the Saved tab on the Log Analytics page and find the query:
- To run the query, click Run.
- To edit, view, or delete the query, use the options in the more_vert More Options menu.
Query logs by using BigQuery
When you have a log bucket that has been upgraded to use Log Analytics, you can also view your data in BigQuery by creating a linked BigQuery dataset. With this configuration, you can use BigQuery to join your log data, which is accessible through the linked dataset, with other business data. For information about how to create a linked dataset, see Grant BigQuery view access to a log bucket.
To query a linked dataset, do the following:
-
In the Google Cloud console, go to the Log Analytics page:
If you use the search bar to find this page, then select the result whose subheading is Logging.
In the Log views list, find the view, and then select Query. The Query pane is populated with a default query.
You can also enter a query in the Query pane, or edit a displayed query.
In the toolbar, click Run in BigQuery.
The BigQuery Studio page opens. The
FROM
statement of the query is modified to specify the path to the view on the linked dataset by using the BigQuery Table path syntax.You can also edit the displayed query.
Click Run query.
What's next
- Sample queries
- Create a log bucket and upgrade it to use Log Analytics
- Upgrade a bucket to use Log Analytics
- Create a linked dataset