Hybrid and multi-cloud monitoring and logging patterns

Last reviewed 2023-03-29 UTC

This document discusses monitoring and logging architectures for hybrid and multi-cloud deployments, and provides best practices for implementing them by using Google Cloud. With this document, you can identify which patterns and products are best suited for your environments.

Every enterprise has a unique portfolio of application workloads that place requirements and constraints on the architecture of a hybrid or multi-cloud setup. Although you must design and tailor your architecture to meet these constraints and requirements, you can rely on some common patterns.

The patterns covered in this document fall into two categories:

  • In a single pane of glass architecture, all monitoring and logging is centralized, with the aim of providing a single point of access and control.
  • In a separate application and operations architecture, sensitive application data is segregated from less sensitive operations data, with the aim of meeting compliance requirements for sensitive data.

Choosing your architecture pattern

You can use the decision tree in the following diagram to identify the best architecture for your use case.

Decision tree for selecting a monitoring and logging architecture.

Details of each architecture are discussed further in this document, but at a high level, your choices are as follows:

  • Export from Monitoring to legacy solution.
  • Export directly to legacy solution.
  • Use Monitoring with Prometheus and Fluentd or Fluent Bit.
  • Use Monitoring with observIQ BindPlane.

Single pane of glass architecture

A common goal for a hybrid system is to integrate monitoring and logging information from various sources across multiple applications and environments into a single display. This type of display is called a single pane of glass.

The following diagram illustrates this pattern where monitoring and logging data from all applications, both on-premises and in the cloud, is centralized into a single repository hosted in the cloud.

High-level architecture for monitoring and logging.

This architecture has the following advantages:

  • You have a single, consistent view for all monitoring and logging.
  • You have a single place to manage data storage and retention.
  • You get centralized access control and auditing. However, you still need to ensure the security of data in transit to the central repository.

Monitoring as a single pane of glass

Cloud Monitoring is a Google-managed monitoring and management solution for services, containers, applications, and infrastructure. For a single pane of glass and a robust storage solution for metrics, logs, traces, and events, use Google Cloud Observability. The suite also provides a complete suite of observability tooling, such as dashboards, reporting, and alerting.

All Google Cloud products and services support integration with Monitoring. In addition, there are several integrated tools that you can use to extend Monitoring to hybrid and on-premises resources.

The following best practices apply to all architectures using Monitoring as a single pane of glass:

Hybrid monitoring and logging with Monitoring and BindPlane by observIQ

With BindPlane from Google's partner observIQ, you can import monitoring and logging data from both on-premises VMs and other cloud providers, such as Amazon Web Services (AWS), Microsoft Azure, Alibaba Cloud, and IBM Cloud into Google Cloud. The following diagram shows how Monitoring and BindPlane can provide a single pane of glass for a hybrid cloud.

High-level architecture for monitoring and logging with BindPlane and
Monitoring.

This architecture has the following advantages:

For more details about implementing this pattern, see Logging and monitoring on-premises resources with BindPlane.

Hybrid Google Kubernetes Engine monitoring with Prometheus and Monitoring

With Google Cloud Managed Service for Prometheus, a popular open source monitoring solution fully managed by Google Cloud, you can monitor applications running on multiple Kubernetes clusters with Monitoring. This architecture is useful when running Kubernetes workloads distributed across Google Kubernetes Engine (GKE) on Google Cloud and GKE on VMware in your on-premises data center, because it provides a unified interface across both. The following diagram shows how to use Prometheus and the Monitoring collectors for data collection.

High-level architecture for GKE monitoring with Prometheus and
Monitoring.

This architecture has the following advantages:

  • Consistent Kubernetes metrics across cloud and on-premises environments.
  • It lets you globally monitor and alert on your workloads by using Prometheus, without having to manually manage and operate Prometheus at scale.
  • There are no additional licensing costs for using Prometheus. Prometheus metrics are imported into Monitoring. The imports are chargeable and priced by the number of samples ingested.

This architecture has the following disadvantages:

  • Prometheus supports monitoring only, so logging has to be configured separately. The following section discusses a common option for logging using either Fluentd or Fluent Bit.

We recommend the following best practice:

  • By default, Prometheus collects all exposed metrics, each of which becomes a chargeable metric. To avoid unexpected costs, consider implementing Monitoring cost controls.

Hybrid GKE logging with Fluentd or Fluent Bit and Cloud Logging

With Fluentd or Fluent Bit, a popular open source logging agent and Cloud Logging, you can ingest logs from applications running on multiple GKE clusters to Cloud Logging. This architecture is useful when running Kubernetes workloads distributed across GKE on Google Cloud and GKE on VMware in your on-premises data center, because it provides a unified interface across both. The following diagram illustrates the flow of logs.

High-level architecture for GKE monitoring with Fluentd or Fluent Bit,
Monitoring, and Logging.

This architecture has the following advantages:

  • You can have consistent Kubernetes logging across cloud and on-premises environments.
  • You can customize Logging to filter out sensitive information.
  • There are no additional licensing costs for using Fluentd or Fluent Bit. Logs that are imported into Logging by using Fluentd or Fluent Bit are chargeable.

This architecture has the following disadvantages:

  • Fluentd and Fluent Bit support logging only, so monitoring has to be configured separately. The previous section discusses a common option for monitoring with Prometheus.

For more details about implementing this pattern, see Customizing Logging logs for Google Kubernetes Engine with Fluentd or Customizing Logging logs for Google Kubernetes Engine with Fluent Bit.

Partner services as single panes of glass

If you are already using a third-party monitoring or logging service such as Datadog or Splunk, you might not want to move to Logging. If so, you can export data from Google Cloud to many common monitoring and logging services. You can choose to use an integrated monitoring and logging service, or select separate monitoring and logging services that best fit your needs.

Export from Logging to partner services

In this pattern, you authorize the partner's monitoring service, such as Datadog, to connect to the Cloud Monitoring API. This authorization lets the service ingest all the metrics available to Logging, so Datadog can function as a single pane of glass for monitoring.

For logging data, Logging provides exports (log sinks) to Pub/Sub. These exports provide a performant and resilient method for partner logging services such as Elastic and Splunk to ingest large volumes of logs from Logging in real time, so these partner services can serve a single pane of glass for logs.

The combined architecture for logging and monitoring is shown in the following diagram.

High-level architecture for exporting monitoring and logging data to partner services.

This architecture has the following advantages:

  • You can continue to use familiar existing tools.
  • Google Cloud Support continues to have access to Logging logs for troubleshooting.

This architecture has the following disadvantages:

  • Partner solutions are typically externally hosted, which means they might not be available or collect data if network connections are disrupted. Sometimes, you can mitigate this risk by self-hosting, but at the cost of having to maintain the infrastructure for the solution yourself.
  • Externally hosted dashboards aren't directly available to Google Cloud Support. This lack of availability can slow down troubleshooting and mitigation.
  • Commercial partner solutions might entail more licensing fees.

Some detailed example integrations include the following:

Analyze metrics from Prometheus and Logging with Grafana

Grafana is a popular open source monitoring tool commonly paired with Prometheus for metrics collection. In this architecture, you use Prometheus as the on-premises collection layer and use Grafana as a single pane of glass for both Google Cloud and on-premises resources. The following diagram shows a sample architecture that analyzes metrics from Google Cloud and on-premises.

High-level architecture for monitoring with Grafana as a single pane of
glass.

This architecture has the following advantages:

  • It's suitable for hybrid environments with both VMs and containers.
  • If your organization is already using Prometheus and Grafana, your users can continue to use them.

This architecture has the following disadvantages:

For more information, see Better troubleshooting with a Cloud Logging plugin for Grafana.

Export logs using Fluentd

An earlier pattern covered using Fluentd or Fluent Bit as a log collector for Logging. The same basic architecture can also be used for other logging or data analytics systems that support Fluentd or Fluent Bit, including BigQuery, Elastic, and Splunk. The following diagram illustrates this pattern.

High-level architecture of exporting logs directly from Fluentd or Fluent Bit.

This architecture has the following advantages:

  • It's suitable for hybrid environments with both VMs and containers.
  • Fluentd can read from many data sources, including system logs.
  • Fluentd offers output plugins for many popular third-party logging and data analytics systems.
  • Fluent Bit can also read from many inputs, including system logs.
  • Fluent Bit offers outputs for many popular third-party logging and data analytics systems.

This architecture has the following disadvantages:

  • Fluentd and Fluent Bit support logs only, so monitoring has to be configured separately. The previous section discusses common options for monitoring with Prometheus and Grafana.
  • Fluentd and Fluent Bit are third-party tools and not official Google products. Google doesn't offer support for them.
  • Exported logs are not available to Google Cloud Support for troubleshooting. In particular, Google does not offer support for GKE on VMware clusters without Logging enabled.

Separate application and operations data

Single pane of glass architectures require streaming application monitoring and logging data to the cloud. However, you might have regulatory or compliance requirements that either require keeping customer data on-premises or place strict constraints on what data can be stored in the public cloud.

A useful pattern for these hybrid environments is to separate sensitive application data from lower-risk operations data, as illustrated in the following diagram.

High-level architecture for separating application and operations data.

Separate application and system data with GKE Enterprise

GKE Enterprise on VMware, a part of the GKE Enterprise suite, includes Grafana for monitoring on-premises clusters. In addition, you can opt to install a partner solution such as Elastic Stack or Splunk for logging. Using these solutions, you can ingest and view sensitive application data entirely on-premises, while still exporting system data to Logging on Google Cloud. The following diagram illustrates this architecture.

Separating application and system data with GKE Enterprise.

This architecture has the following advantages:

  • Sensitive application data is kept entirely on-premises.
  • On-premises monitoring and logging have no cloud dependencies and remain available even if the network connection is interrupted.
  • All GKE system data, both on-premises and Google Cloud, is centralized in Logging and is also accessible to Google Cloud Support as needed.

For an example implementation using Elastic Stack as the partner solution, see Monitoring GKE Enterprise with the Elastic Stack.

What's next