Shape the future of software operations and make your voice heard by taking the 2021 State of DevOps survey.

Using the Logs Explorer

This guide shows you how to search and view logs with the Logs Explorer.

Before you begin

To view the logs that you are sending from an Amazon Web Services (AWS) account to Logging, select the AWS connector project in the Google Cloud Console project picker and then use the Legacy Logs Viewer. The AWS connector project stores the Amazon Resource Name (ARN) for your AWS account and links your AWS account to Google Cloud services. For more information, see Viewing metrics for AWS accounts.

Getting started

To navigate to the Logs Explorer, do the following:

  1. Go to the Google Cloud navigation menu and select Logging > Logs Explorer:
    Go to the Logs Explorer
  2. Select a Google Cloud project.
  3. From the Upgrade menu, switch from Legacy Logs Viewer to Logs Explorer.

You're now in the Logs Explorer.

Logs Explorer interface

The Logs Explorer interface lets you retrieve logs, parse and analyze log data, and refine your query parameters.

User interface for the Logs Explorer

The Logs Explorer contains the following panes:

  1. Action bar
  2. Query builder
  3. Log fields
  4. Histogram
  5. Query results

Action bar

Action bar pane

The Action bar pane offers the following features:

  1. Options: Lets you go to the Legacy Logs Viewer, send feedback, and view a summary of new Logging features.
  2. Refine scope: Lets you scope your search by logs in your current Cloud project only or by one or more storage views. For more information about scoping, see Refining scope.
  3. Share link: Lets you create a shortened URL of the current query and copies it to your clipboard, making it easier to share a query.
  4. Time-range selector: Lets you restrict query results by time range. The default time range is one hour.
  5. Page layout: Lets you enable and disable the Histogram and Logs field explorer panes.
  6. Learn: Lets you view links to relevant documentation.

Refining scope

You can refine the scope of the logs displayed in the Logs Explorer through the Refine scope option. You can search only logs within the current Cloud project or search logs in one or more storage views. To refine the scope of the Logs Explorer, do the following:

  1. From the Logging menu, select Logs Explorer.

    Go to Logs Explorer

  2. Select Refine Scope.

  3. On the Refine scope dialog, select a Scope by option.

    The Refine scope dialog

    • Scope by Cloud project allows you to search logs that the current Cloud project generates.

    • Scope by storage allows you to search logs based on one or more storage views. For more information about log views, see Managing log views on your log buckets.

  4. If you select Scope by storage, select one or more buckets you want to view.

    The dialog lists storage views that meet the following conditions:

    • The user has access to the storage view.
    • The log buckets belong to the selected Cloud project, or the selected Cloud project has previously routed logs to the storage buckets.
  5. Click Apply.

Query builder

Query builder

From the Query builder pane, you can do the following:

  1. Query-builder field: Lets you build queries using the Logging query language.
  2. Query builder drop-down menus: Lets you add query expressions based on Resource, Log name, and Severity. For more information, see Query builder drop-down menus.
  3. Recent: Lets you view your recent queries. For more information, see Recent queries.
  4. Saved: Lets you view your saved queries and queries that other users of the Cloud project have shared with you. For more information, see Saved queries and Shared queries.
  5. Suggested: Lets you view suggested queries based on the resources in your Cloud project. For more information, see Suggested queries.
  6. Save: Lets you save queries that can be viewed and run from the Saved tab.
  7. Stream logs: Lets you view log entries as Logging ingests them. For more information, see Streaming logs.
  8. Run: Lets you run your queries after you have built them in the query-builder field.

Streaming logs

You can stream your logs as Logging ingests them, or you can add a query to stream only those logs that match the query.

To stream logs based on a query, add a query to the query-builder field, and then select Stream logs. As Logging ingests the logs data, only those logs that match the query are shown in the Query result pane. If a query isn't provided, Logging shows each log as it's ingested.

Add a query before selecting to stream logs

To stop streaming, select Stop streaming, or scroll down within the Query results pane.

Log fields pane

The Log fields pane offers a high-level summary of logs data and provides a more efficient way to refine a query. It shows log entries broken down by different dimensions, corresponding to fields in these entries. For each field, the Log fields pane shows values and their incidence in descending frequency order. The log-field counts correspond to the time range in the time-range selector.

Enabling the Log fields pane

To enable the Log fields pane, select Page Layout, and then select the Log fields checkbox. The Log fields pane appears.

Log fields pane selected

To disable the Log fields pane, clear the Log fields checkbox.

Log fields pane features

The Log fields pane is populated and updated based on an executed query in the Query builder.

If the query is empty, the Log fields pane displays the counts of log entries by the Resource type and Severity fields.

log fields pane.

If you've selected Scope by storage, you'll also see Project ID and the corresponding counts of log entries.

If you select a resource type from the Log fields pane, a set of relevant fields, based on the resource labels, populate the pane. This lets you investigate logs data for that specific resource type. You can remove them by clicking Clear next to the Resource type field.

Adding fields to Log fields pane

You can add certain LogEntry key-value pairs to the Logs field pane from the log entries populated in the Query results pane. To add a field to the Logs field pane, do the following:

  1. In the Query results pane, expand a log entry by clicking the expand button .

  2. Left-click on a field's value. From the menu, select Add field to Log fields pane.

    Menu with option to add field.

    The custom field appears in the Log fields pane as a list of key-value pairs.

To remove a custom field from the Log fields pane, click Remove next to the field. You can also remove a custom field from the list by left-clicking on it in the Query results pane and selecting Remove from Log fields.

Note that the following types of fields can't be added to the Log fields pane:

  • Fields related to time; for example, receiveTimestamp and protoPayload.startTime.
  • Fields with high cardinality; for example, insertId and protoPayload.latency.
  • Fields with array indices in their path; for example, protoPayload.authorizationInfo[0].resource.

Analyzing logs using the Log fields pane

To narrow down and refine a query, you can add field-value pairs from the Log fields pane to the Query builder expression. To do so, in the Log fields pane, click on a field's value. This adds the field-value pair to the expression using the AND operator. The query then runs.

You can add nested field-value pairs, as well as top-level field-value pairs, to the Log fields pane.

For example, suppose you added jsonPayload.message to the Log fields pane. If you select a particular jsonPayload.message value, it populates the Query builder pane:

log fields and query builder panes with custom field added.

The Histogram and Query results panes also change to reflect the current query.

When a query is executing, the log entries are scanned and the log-field counts change. When the query is complete, the total counts for all log fields are displayed.

Histogram

The Histogram pane lets you visualize the distribution of logs over time. The histogram regenerates when you run a query, making it easier to see trends in your logs data and troubleshoot problems.

Enabling the histogram pane

To enable the histogram pane, select Page Layout, and then select the Histogram checkbox. The Histogram pane appears.

Page layout is open and histograms is selected

To disable the histogram pane, clear the Histogram checkbox.

Histogram features

Histogram pane.

  1. Histogram bars: Each histogram bar represents a time range. Each bar contains a three-color breakdown for the log-severity levels captured in each bar's time range. The colors represent the following log severities:

    • Blue: Low severities such as Default, Debug, Info, and Notice.
    • Yellow: Medium severities such as Warning.
    • Red: High severities such as Error, Critical, Alert, and Emergency.

    Each histogram bar features a menu with options to analyze your logs.

  2. Time controls: Let you adjust the time range used for queries. For details on these options, see Analyzing logs using time controls.

  3. Histogram timeline: Shows you the time range of the logs, represented by histogram bars, that are currently displayed within the Query results pane. The timeline helps to orient you to the logs you're currently viewing within the larger time range of your query.

Analyzing logs using time controls

You can use the histogram's time controls to help you investigate and analyze your logs data.

Adjust time quickly

The histogram provides time controls that let you quickly adjust the data that you see in the Logs Explorer.

Histogram pane timeline is showing the quick time controls.

  • Time handles: Drag the timeline's handles inward to narrow the data or outward to widen the data in the histogram timeline. Click Run.

  • Slide the timeline forward and backward: Click the forward arrow to slide the timeline to a later time. Click the backward arrow to slide the timeline to an earlier time.

  • Zoom in and out: Click the zoom-out icon to broaden the data shown in the timeline. Click the zoom-in icon to narrow the data shown in the timeline.

When you use these time controls, the logs data in the Query results and Log fields panes adjusts according to the time range captured by the histogram timeline.

Note that timeline modifications are constrained to be between the current time ("now") and 30 days ago.

Scroll or zoom to time

In addition to the time controls above, the histogram provides the Scroll to time and Zoom to time features to give you more in-depth control of the histogram and the data that you see in other panes in the Logs Explorer.

Histogram pane timeline is showing the scroll to time and zoom to time controls.

Perhaps a particular histogram bar interests you based on its relative size or severity levels. You can select that histogram bar to adjust the logs data you see in the Logs Explorer.

The Scroll to time feature lets you browse your logs data without changing the values in the Histogram and Log fields panes. When you select the Scroll to time feature, the following happens:

  • The logs data that you see in the Query results pane adjusts according to the time range captured by the selected histogram bar.

    The query isn't run, but a partial reload of the data might occur to ensure you're seeing logs in the Query results pane that correspond with the selected histogram bar's time range.

  • The console URL updates to contain the timestamp of the most recent log captured by the time range of the selected histogram bar.

To select the Scroll to time feature, do the following:

  1. Hover over a bar in the Histogram timeline. A pane containing summary information about the logs data for the specified time range appears.

  2. In the pane, select Scroll to time.

    Alternatively, clicking on a histogram bar, instead of hovering over it, is equivalent to selecting Scroll to time.

The Zoom to time feature is similar to Scroll to time, but it runs a query on your logs data based on the time range captured by a selected histogram bar. When you select the Zoom to time feature, the following happens:

  • The logs data that you see in the Query results pane reloads and narrows according to the time-range restriction of the selected histogram bar.
  • The console URL updates to contain the timestamp of the most recent log captured by the time range of the selected histogram bar.
  • The histogram changes to show only logs that have a timestamp value that falls within the time range of the selected histogram bar.
  • The time-range selector updates to the time range captured by the selected histogram bar.
  • The data in the Log fields pane adjusts according to the time range captured by the selected histogram bar.

To select the Zoom to time feature, do the following:

  1. Hover over a bar in the Histogram timeline. A pane containing summary information about the logs data for the specified time range appears.

  2. In the pane, select Zoom to time.

Query results

Query results pane

The Query results pane lets you explore the log entries that match your query expressions.

  1. Query results: Lets you view the retrieved logs from your query.
  2. Log entries: Lets you view log entries in the structured JSON format.
  3. Expand and collapse query results: Lets you expand the query-results pane to view more log entries.
  4. Time zone: Lets you change the time zone that logs are displayed in.
  5. Trace data: Lets you view trace details and refine your query based on the trace. For more information, see Viewing trace data.
  6. Hide log summary: Lets you hide the log summary line from the query results.
  7. Expand or collapse nested log fields: Lets you expand or collapse nested fields.
  8. Copy to clipboard: Lets you copy the log entry in its JSON format.
  9. Copy link to a log entry: Lets you share a link to a log entry. For more information, see Copying a link to a log entry.
  10. Jump to now: Lets you perform a forced refresh to include the current time. If the time-range selector uses a custom range and an end time is set, it runs the query with a default time range of one hour. Otherwise, it refreshes with the current start date or duration, and runs the query.
  11. Actions: Lets you set up a logs-based metric, create a sink destination, or download your logs. For more information on downloading logs, see Downloading logs.
  12. Configure: Lets you add the value of a log field to the summary line at the beginning or end of the log entry. It also lets you choose to show newest logs either first or last. For more information on adding a summary field, see Adding summary fields.
  13. Pin log entry: Lets you pin a log entry to the Query results and Histogram panes. For more information, see Pinning logs.
  14. Cursor scroll: When you scroll the logs in Query results, the URL adjusts to include cursorTimestamp, which indicates the timestamp of the newest log shown in the current Query results pane.

Within the query-results pane, you can click the values of a field to choose to do the following:

Options after selecting field's value

  1. Show matching entries: Lets you query for matching log entries.
  2. Hide matching entries: Lets you query for log entries that don't match the selected expression.
  3. Add field to summary line: Lets you add the field as a summary line to log entries.

Adding summary fields

Summary fields help you notice patterns in your logs faster. For example, the following image shows the value for the summary field resource.labels.pod_name added before the logs that contain that value.

The Logs Explorer is showing logs that are preceded with green text displaying
pod names.

Add a summary field from a log entry

To add a summary field to a log entry, complete the following steps:

  1. Expand a log entry by clicking the expand button .

  2. Click a field's value and then select Add field to summary line.

    The summary field now appears before the log entries containing that field.

Add a summary field using the Configure button

To add a summary field using the Configure button, complete the following steps:

  1. Click Configure and select Manage Summary Fields.

    Manage summary fields is selected from the configure drop-down menu

  2. Add fields.

    The summary field selection has the following features:

    • Autocomplete using the logs currently displayed.
    • Field correction for legal characters within quotes.

      For example, if you type jsonPayload.id-field, it gets changed to jsonPayload."id-field".

  3. Click Truncate summary fields to shorten the display of the summary field values. Then choose how many characters to display before the field is truncated and whether the beginning or the end of the field is displayed.

  4. Click Apply.

    The summary field now appears before the log entries containing that field.

Pinning log entries

Pinning a log entry lets you highlight a log entry of interest.

To pin a log entry, hover over the log entry you want to pin, and then select the pin icon . After you pin a log entry, its background is darkened, and a pin icon is shown.

If you pin a log entry and rerun your query, the pinned log entry appears at either the top or bottom of the Query results pane, depending on how your logs data is configured. A pin icon also appears on the Histogram pane based on the pinned log entry's timestamp.

Logs Explorer shows a pinned log entry in the Query results and Histogram pane.

To unpin the log entry, select the pin icon, and then select Unpin log entry.

Viewing a pinned log entry in its resource context

To view the pinned log entry within its resource context, select the pin icon and then select Pin and show resource log entries.

Pin and show in resource context is selected.

Logging populates the Query builder with the resource.type field from the pinned log entry and runs the query. You can now view your pinned log entry in relationship with its resource type.

Viewing a pinned log entry in the Histogram pane

You can use the Histogram pane to highlight, scroll to, and further examine a pinned log entry.

Using the Histogram pane, select the pin icon and then choose from the following menu options:

  • Scroll to log entry: This option brings the log entry into the current Query results pane and lets you view the pinned log entry in the context of nearby logs.
  • Zoom to log entry: This option narrows the time range that the Histogram pane displays and lets you refine your query to isolate the logs near the pinned log.

Histogram timeframe is narrowed.

Viewing trace data

When a log entry contains both the trace and the latency-related field, both the latency and trace icon appear.

Log entry display that contains trace data.

When a log entry contains only the trace field, then only the trace icon appears.

Log entry display that contains only the trace field has trace icon.

To view the trace data related to the log entry, click the trace icon. You have the following options:

  • View trace details: Shows the parent span and child traces along with details about the trace. To view more details about the trace, navigate to Cloud Trace by clicking View in Trace. For more information about the content in the flyout pane, see Viewing trace details.
  • Show all logs for this trace: Refines and runs the query by adding the trace field set to the identifier of the trace associated with the log entry.

  • Show only traced requests: Refines and runs the query by adding the traceSampled field set to True. For more information on sampling, go to Sampling rate.

To share a link to a log, expand a log entry, and then select Copy link. The link is copied to your clipboard. You can now send the link to users who have access to the Cloud project. When a user pastes the link into a browser or selects it, Logging pins the log entry in their Query results pane.

Copy link to share log entry with others.

Downloading logs

You can download your logs in CSV or JSON format. You need one of the following Identity and Access Management roles to download logs:

  • Logging Admin (roles/logging.admin)
  • Logs View Accessor (roles/logging.viewAccessor)

To download your logs, do the following:

  1. Select Actions, and then Download Logs.

    Download logs with the Action button.

  2. In the Download logs dialog, select CSV or JSON format, and then select to download the logs either to your computer or to Drive, or to view them in a new tab.

    When you save a CSV and select Drive, you can open the file in Sheets.

Troubleshooting

This section provides instructions for troubleshooting common issues when using the Logs Explorer.

Selecting a Cloud project or organization

To select a Cloud project from anywhere in the Google Cloud Console, including from the Logs Explorer, use the Cloud project and organization selector:

A project is selected from the drop-down menu

Getting Cloud project or organization ID

To get a Cloud project or organization ID from anywhere in the Google Cloud Console, expand the list of Cloud projects from the Cloud project and organization selector and find the Cloud project ID in the ID column:

The ID for the project is shown

Can't see log entries

If you don't see any log entries, check the following:

  • Is the correct Cloud project selected? If not, select the correct Cloud project from the Cloud project and organization selector.

  • Is your Cloud project using resources that generate logs and is there activity on those resources? Even if the Cloud project is new, it should have audit logs recording the fact that it was created. Verify you're using a resource that generates logs, by going to the "Mapping services to resource types" section in the Monitored resource list page.

  • Is the time range too narrow? Verify the time range in your query is correct.

  • View your current exclusion queries to ensure that the logs you're looking for aren't accidentally excluded.

My query is correct but I still don't see log entries

  • You can't see log entries that are older than the Logging retention period. See Logs retention periods for the logs retention period in effect.

  • During periods of heavy load, there could be delays in sending logs to Logging or in receiving and displaying the logs.

  • The Logs Explorer doesn't show log entries that have timestamps in the future until the current time has "caught up" with them. This is an unusual situation, probably caused by a time skew in the application sending the logs.

Getting support

For information on getting support, see Google Cloud's operations suite support page.