Analyze your logs easier with log field analytics
Charles Baer
Product Manager, Google Cloud
We know that developers or operators troubleshooting applications and systems have a lot of data to sort through while getting to the root cause of issues. Often there are fields like error response codes that are critical for finding answers and resolving those issues. Today, we’re proud to announce log field analytics in Cloud Logging, a new way to search, filter and understand the structure of your logs so you can find answers faster and easier than ever before.
Log field analytics
Last year we launched Logs Explorer to make it faster to find and analyze your logs, with features like the Log fields pane and the histogram, as well as saved and shared queries. In Logs Explorer, the Log fields pane and histogram both provide useful information to help analyze your logs.
With the Log fields pane, each resource type, which maps to a specific Google Cloud service like BigQuery or Google Kubernetes Engine (GKE), includes a set of default fields and values found in the logs loaded in Logs Explorer. The Log fields pane includes the name of the log field, a list of values and an aggregated count of the number of logs that fall in that category. Let’s look at these key terms more closely:
- a log field – These are the specific fields in your logs. All logs in Cloud Logging use the LogEntry message format. For example, the logName field is present in all logs in Cloud Logging. When you write logs, you also include textPayload, jsonPayload or protoPayload fields such as jsonPayload.http_req_status.
- a log field value – The value of a specific log field. For example, for a log entry with the jsonPayload.http_req_status field, some example values could be a “200”, “404” or “500”.
Now you can gain insight into the full list of values for selected log fields and a count of how many logs match the value with log field analytics. You can analyze application or system logs using fields in the jsonPayload or protoPayloads of your log entries and then easily refine your query by selecting the field values to drill down into the matching logs.
A view of the Logs fields pane in Cloud Logging
Better troubleshooting by analyzing log values
Log field analytics make it easy to quickly spot unexpected values. By adding a field to the Log fields pane, you can view all values that appear in logs and then select any of the values to filter the logs by those values.
In this example ecommerce application, we’ve added the jsonPayload.http_req_path field, and now it is possible to look at the request paths over time. In the screenshot below, it’s easy to see that there are several unexpected values that would indicate a problem such as “/products/error” and “products/incorrectproduct”. Next to those values are the aggregated number of matching log entries. These values can help you narrow your troubleshooting or analysis.
Filter using field values
The field value selection in the Log fields pane can be used to refine your query so you can see just the logs that contain the selected value. In our example above using the jsonPayload.http_req_path field, it’s possible to select a specific value, “/cart” in this case, and view the logs broken down by severity.
Better understand your audit logs
Using log field analytics, you can easily find values in audit logs for Google Cloud services. For example, you may want to identify the accounts that are making data access requests for a particular GKE cluster. If you add the protoPayload.authenticationInfo.principalEmail field as a custom field to the Log fields pane, you get both a list of the accounts making the requests and the number of log entries for each of the account values.
The Log fields pane displaying the number of log entries for each principalEmail value
Get started today
Log field analytics, Log fields, and the Histogram are features that we’ve recently added to Logs Explorer and they’re ready for you to get started with today. But we’re not stopping there! Please join us in our discussion forum for more information about what is coming next and to provide feedback on your experiences using Cloud Logging.
If you would like to learn more about Cloud Logging, you can also visit our qwiklab quest for a guided walk through of the capabilities.
