Bucket list: Better log storage and management for Cloud Logging
Mary Koes
Product Manager, Google Cloud
As more organizations move to the cloud, the volume of machine generated data has grown exponentially and is increasingly important for many teams. Software engineers and SREs rely on logs to develop new applications and troubleshoot existing apps to meet reliability targets. Security operators depend on logs to find and address threats and meet compliance needs. And well structured logs provide invaluable insight that can fuel business growth. But first logs must be collected, stored and analyzed with the right tools, and many organizations have found they can be expensive to store and difficult to manage at scale.
Our goal for Google Cloud Logging has always been to make logging simpler, faster, and more useful for our customers. That means making it easy to search and analyze logs as well as providing a secure, compliant, and scalable log storage solution. Today we’re announcing a number of improvements to log storage and management, building on several recent improvements for exploring and analyzing logs. Here’s a selection of what’s new:
Logs buckets (beta)
Logs views (alpha)
Regionalized log storage (alpha)
Customizable retention (generally available)
Cloud Logging Router (generally available - new functionality in beta)
Exploring and analyzing logs (generally available)
Cloud Logging has been deeply integrated in Google Cloud Platform from the beginning. We automatically collect logs from dozens of Google Cloud services including audit logs, which play a key role in security and compliance. These logs are available right in context from places like Compute Engine, Cloud Functions, App Engine and more to improve development velocity and troubleshooting. Our challenge was to build a logging storage solution that was flexible enough to meet many different organizational needs while preserving the in-context experience and enterprise-class security around logs.
We do this by introducing “logs buckets” as a first-class logs storage solution in Cloud Logging. Using logs buckets, you can centralize or subdivide your logs based on your needs. From the name, logs buckets may sound like Cloud Storage buckets, but logs buckets are built on the same logging tech stack we’ve been using to deliver your logs in real time with advanced indexing and optimizations for timestamps so that you can keep benefiting from our logs analytics features.
In order to support logs buckets, we’ve also augmented the Cloud Logging router to give you more control over where your logs go. Previously, there were different models to manage which logs went to Cloud Logging vs. other destinations including BigQuery, Cloud Storage and Pub/Sub. Now, you can manage all destinations consistently using log sinks, and all log sinks can also support exclusions, making it easier to configure the logs you want to the right destination. You can also now route logs from one project to another or even use aggregated log sinks from across folders or organization level for security and ease of maintenance.
Here are some examples of solutions our alpha customers have built using logs buckets:
Log centralization - Centralize all logs from across an organization to a single Cloud Logging project. This solution was so popular among security teams that we’ve put together a dedicated user guide for centralizing audit logs, but you can centralize any or all logs in your org. This allows you to identify patterns and comparisons across projects.
Splitting up logs from a single project for GKE multi-tenancy - Send logs from one shared project to other projects owned by individual development teams. One of our alpha customers’ favorite things about logs buckets is that we do magic behind the scenes to look up where your logs are stored. That way, you can, for example, still view those logs for your Kubernetes cluster in the GKE console in project A, even if they’re stored centrally in project B. Get started with this user guide.
Compliance-related retention - Logs buckets also allow you to take advantage of advanced management capabilities such as setting custom retention limits or locking a logs bucket so that the retention cannot be modified. We’ve recently launched custom retention to GA and are excited to announce that you can use custom retention through the end of March 2021 for no additional cost. This gives you a chance to try out log management for your long-term compliance and analytics needs for logs without a commitment.
Regionalized log storage - You can now keep your logs data in a specific region for compliance purposes. When you create a logs bucket, you can set the region in which you want to store your logs data. Setting the location to
globalmeans that it is not specified where the logs are physically stored. The logs bucket beta only supports thegloballocation, but more regions are available in the regionalized logs storage alpha. Sign up for the alpha or to be notified when more regions are publicly available.
Another piece of feedback we hear is that you’d like to be able to configure who has access to logs based on the source project, resource type or log name. We’ve also introduced log views so that you can specify which logs a user should have access to, all using standard IAM controls. Logs views can help you build a system using the principle of least privilege, limiting sensitive logs to only users who need this information. While we’ve created logs views automatically for you to preserve limited access to sensitive logs, you’ll soon be able to create your own logs views based on the source project, resource type or log name. If you’d like to try it out in alpha, sign up here.
Getting started
Having the right logs, and being able to access them easily, is essential for development and operations teams alike. We hope these new Cloud Logging features make it easier for you to find and examine the logs you need. To learn more about managing logs in Google Cloud, check out these resources:
