Introducing Cloud Logging - Log Analytics, powered by BigQuery
Charles Baer
Product Manager, Google Cloud
Logging is a critical part of the software development lifecycle allowing developers to debug their apps, DevOps/SRE teams to troubleshoot issues, and security admins to analyze access. Cloud Logging provides a powerful pipeline to reliably ingest logs at scale and quickly find your logs. Today, we’re pleased to announce Log Analytics, a new set of features in Cloud Logging available in Preview, powered by BigQuery that allows you to gain even more insights and value from your logs. We are also doing a live webinar on Nov 15, 2022. Register here to learn how you can get started immediately.
Introducing Log Analytics
Log Analytics brings entirely new capabilities to search, aggregate, or transform logs at query time directly into Cloud Logging with a new user experience that's optimized for analyzing logs data through the power of BigQuery. BigQuery is a cost-effective, serverless, multi cloud data warehouse to power your data-driven innovation.
With Log Analytics, you can now harness SQL (see figure 1) and the capabilities of BigQuery to analyze their logs. Now, Cloud Logging offers the functionality you had in the past and analytical capabilities through Log Analytics
Secure, compliant and scalable logging ingestion pipeline through Logs Router
Managed logging-as-a-service solution with a specialized user interface for log analysis.
Support for centralized logging across Google Cloud, other clouds, and on prem
Automated insights and suggestions such as Error Reporting
Log based metrics and alerts for real-time aggregation, visualization and alerting of logs
Flexible pay as you go pricing
NEW - Powerful BigQuery engine and SQL option for ad hoc log processing
NEW - Automatic read only access to all log analytics logs in BigQuery
NEW - Rich visualization of log data (Figure-2, in private preview)
Why is Log Analytics powerful?
Log Analytics leverages the power of BigQuery to enable Cloud Logging users to perform Analytics on Log data.
Centralized logging – By collecting and centrally storing the log data in a dedicated Log Bucket, it allows multiple stakeholders to manipulate their data from the same datasource. You don’t need to make duplicate copies of the data.
Reduced cost and complexity – Log Analytics allows reuse of data across the organization, effectively saving cost and reducing complexities.
Ad hoc log analysis – It allows for ad-hoc query-time log analysis without requiring complex pre-processing.
Scalable platform – Log Analytics can scale for observability using the serverless BQ platform and perform aggregation at petabyte scale efficiently
Log Analytics is designed for multiple users in an organization and aims to break down silos. Here are the top categories we hear from our users:
Developers & DevOps use it for Infrastructure and Application troubleshooting
Security Teams use it for Audit Log Analysis.
Networking professionals use it to perform network log analysis
Business Operations Teams can potentially manipulate the data, create KPIs and in the future create dashboards.
Pricing
Log Analytics is included in the standard Cloud Logging pricing. Queries submitted through the Log Analytics user interface do not incur any additional cost. Enabling analysis in BigQuery is optional and, if enabled, queries submitted against the BigQuery linked data set including Data Studio, Looker and via BigQuery API, incur the standard BigQuery query cost.
Get started
We are hosting a live webinar on Log Analytics on Nov 15, 2022. To attend the event, register here. Visit the Log Analytics page in the Cloud Console and upgrade an existing Log Bucket or create a new Log Bucket. Check out our sample queries to get you started. Charting in Log Analytics is available now as a Private Preview (sign-up here). In the next blog post, we will talk about how and when users should leverage Log Analytics, how you will get started with Log Analytics and dive into a few common use cases. You can join the discussion in our Cloud Operations page on the Google Cloud Community site.
