Hybrid and multi-cloud monitoring and logging patterns

This document discusses monitoring and logging architectures for hybrid and multi-cloud deployments, and provides best practices for implementing them by using Google Cloud. With this document, you can identify which patterns and products are best suited for your environments.

Every enterprise has a unique portfolio of application workloads that place requirements and constraints on the architecture of a hybrid or multi-cloud setup. Although you must design and tailor your architecture to meet these constraints and requirements, you can rely on some common patterns.

The patterns covered in this document fall into two categories:

  • In a single pane of glass architecture, all monitoring and logging is centralized, with the aim of providing a single point of access and control.
  • In a separate application and operations architecture, sensitive application data is segregated from less sensitive operations data, with the aim of meeting compliance requirements for sensitive data.

Choosing your architecture pattern

You can use the decision tree in the following diagram to identify the best architecture for your use case.

Decision tree for selecting a monitoring and logging architecture.

Details of each architecture are discussed further in this document, but at a high level, your choices are as follows:

  • Export from Monitoring to legacy solution.
  • Export directly to legacy solution.
  • Use Monitoring with Prometheus and Fluentd.
  • Use Monitoring with Blue Medora BindPlane

Single pane of glass architecture

A common goal for a hybrid system is to integrate monitoring and logging information from various sources across multiple applications and environments into a single display. This type of display is called a single pane of glass.

The following diagram illustrates this pattern where monitoring and logging data from all applications, both on-premises and in the cloud, is centralized into a single repository hosted in the cloud.

High-level architecture for monitoring and logging.

This architecture has the following advantages:

  • You have a single, consistent view for all monitoring and logging.
  • You have a single place to manage data storage and retention.
  • You get centralized access control and auditing. However, you still need to ensure the security of data in transit to the central repository.

Monitoring as a single pane of glass

Cloud Monitoring is a Google-managed monitoring and management solution for services, containers, applications, and infrastructure. The Google Cloud operations suite provides a single pane of glass and a robust storage solution for metrics, logs, traces, and events. The suite also provides a complete suite of observability tooling, such as dashboards, reporting, and alerting.

All Google Cloud products and services support integration with Monitoring. In addition, there are several integrated tools that you can use to extend Monitoring to hybrid and on-premises resources.

The following best practices apply to all architectures using Monitoring as a single pane of glass:

Hybrid monitoring and logging with Monitoring and BlueMedora BindPlane

With Google's partner Blue Medora BindPlane, you can import monitoring and logging data from both on-premises VMs and other cloud providers, such as Amazon Web Services (AWS), Microsoft Azure, Alibaba Cloud, and IBM Cloud into Monitoring. The following diagram shows how Monitoring and BindPlane can provide a single pane of glass for a hybrid cloud.

High-level architecture for monitoring and logging with BindPlane and
Monitoring.

This architecture has the following advantages:

For more details about implementing this pattern, see Logging and monitoring on-premises resources with Blue Medora.

Hybrid Google Kubernetes Engine monitoring with Prometheus and Monitoring

With Prometheus, a popular open source monitoring solution, and the Monitoring Prometheus sidecar, you can monitor applications running on multiple Kubernetes clusters with Monitoring. This architecture is useful when running Kubernetes workloads distributed across Google Kubernetes Engine (GKE) on Google Cloud and GKE on-prem in your on-premises data center, because it provides a unified interface across both. The following diagram shows how to use Prometheus and the Monitoring sidecar for data collection.

High-level architecture for GKE monitoring with Prometheus and
Monitoring.

This architecture has the following advantages:

  • Consistent Kubernetes metrics across cloud and on-premises environments.
  • There are no additional licensing costs for using Prometheus. Prometheus metrics are imported into Monitoring as custom metrics, which are charged at standard rates.

This architecture has the following disadvantages:

  • The Monitoring Prometheus sidecar is supported only in GKE environments.
  • Prometheus supports monitoring only, so logging has to be configured separately. The following section discusses a common option for logging, Fluentd.

We recommend the following best practice:

  • By default, Prometheus collects all exposed metrics, each of which becomes a chargeable custom metric. To avoid unexpected costs, implement Monitoring cost controls and consider adding filters in the Monitoring Prometheus sidecar to limit ingestion.

For more details about implementing this pattern, see Monitoring apps running on multiple GKE clusters using Prometheus and Monitoring.

Hybrid GKE logging with Fluentd and Logging

With Fluentd, a popular open source logging agent and Cloud Logging, you can ingest logs from applications running on multiple GKE clusters to Cloud Logging. This architecture is useful when running Kubernetes workloads distributed across GKE on Google Cloud and GKE on-prem in your on-premises data center, because it provides a unified interface across both. The following diagram illustrates the flow of logs.

High-level architecture for GKE monitoring with Fluentd,
Monitoring, and Logging.

This architecture has the following advantages:

  • You can have consistent Kubernetes logging across cloud and on-premises environments.
  • You can customize Logging to filter out sensitive information.
  • There are no additional licensing costs for using Fluentd. Fluentd logs imported into Logging are charged at standard rates.

This architecture has the following disadvantages:

For more details about implementing this pattern, see Customizing Logging logs for Google Kubernetes Engine with Fluentd.

Partner services as single panes of glass

If you are already using a third-party monitoring or logging service such as Datadog or Splunk, you might not want to move to Logging. If so, you can export data from Google Cloud to many common monitoring and logging services. You can choose to use an integrated monitoring and logging service, or select separate monitoring and logging services that best fit your needs.

Export from Logging to partner services

In this pattern, you authorize the partner's monitoring service, such as Datadog, to connect to the Cloud Monitoring API. This authorization lets the service ingest all the metrics available to Logging, so Datadog can function as a single pane of glass for monitoring.

For logging data, Logging provides exports (log sinks) to Pub/Sub. These exports provide a performant and resilient method for partner logging services such as Elastic and Splunk to ingest large volumes of logs from Logging in real time, so these partner services can serve a single pane of glass for logs.

The combined architecture for logging and monitoring is shown in the following diagram.

High-level architecture for exporting monitoring and logging data to partner services.

This architecture has the following advantages:

  • You can continue to use familiar existing tools.
  • Google Cloud Support continues to have access to Logging logs for troubleshooting.

This architecture has the following disadvantages:

  • Partner solutions are typically externally hosted, which means they might not be available or collect data if network connections are disrupted. Sometimes, you can mitigate this risk by self-hosting, but at the cost of having to maintain the infrastructure for the solution yourself.
  • Externally hosted dashboards aren't directly available to Google Cloud Support. This lack of availability can slow down troubleshooting and mitigation.
  • Commercial partner solutions might entail more licensing fees.

Some detailed example integrations include the following:

Export metrics from Prometheus and Logging to Grafana

Grafana is a popular open source monitoring tool commonly paired with Prometheus for metrics collection. In this architecture, you use Prometheus as the on-premises collection layer and use Grafana as a single pane of glass for both Google Cloud and on-premises resources. The following diagram shows a sample architecture exporting metrics from Google Cloud and on-premises.

High-level architecture for monitoring with Grafana as a single pane of
glass.

This architecture has the following advantages:

  • It's suitable for hybrid environments with both VMs and containers.
  • If your organization is already using Prometheus and Grafana, your users can continue to use them.

This architecture has the following disadvantages:

  • Prometheus and Grafana support monitoring only, so logging has to be configured separately, for example, using Fluentd.
  • Prometheus is open source and extensible, but supports only a limited range of enterprise software integrations.
  • Prometheus and Grafana are third-party tools and not official Google products. Google doesn't offer support for Prometheus or Grafana.

For more information, see Introducing Logging as a data source for Grafana. For a detailed tutorial of deploying Prometheus and Grafana with GKE and Logging, see Using Prometheus and Grafana for IoT monitoring.

Export logs using Fluentd

An earlier pattern covered using Fluentd as a log collector for Logging. The same basic architecture can also be used for other logging or data analytics systems that support Fluentd, including BigQuery, Elastic, and Splunk. The following diagram illustrates this pattern.

High-level architecture of exporting logs directly from Fluentd.

This architecture has the following advantages:

  • It's suitable for hybrid environments with both VMs and containers.
  • Fluentd can read from many data sources, including system logs.
  • Fluentd offers output plugins for many popular third-party logging and data analytics systems.

This architecture has the following disadvantages:

  • Fluentd supports logs only, so monitoring has to be configured separately. The previous section discusses common options for monitoring with Prometheus and Grafana.
  • Fluentd is a third-party tool and not an official Google product. Google doesn't offer support for Fluentd.
  • Exported logs are not available to Google Cloud Support for troubleshooting. In particular, Google does not offer support for GKE on-prem clusters without Logging enabled.

For an example of integrating Fluentd with BigQuery, see Analyzing logs in real time using Fluentd and BigQuery.

Separate application and operations data

Single pane of glass architectures require streaming application monitoring and logging data to the cloud. However, you might have regulatory or compliance requirements that either require keeping customer data on-premises or place strict constraints on what data can be stored in the public cloud.

A useful pattern for these hybrid environments is to separate sensitive application data from lower-risk operations data, as illustrated in the following diagram.

High-level architecture for separating application and operations data.

Separate application and system data with Anthos

Anthos on VMware, a part of the Anthos suite, includes Grafana for monitoring on-premises clusters. In addition, you can opt to install a partner solution such as Elastic Stack or Splunk for logging. Using these solutions, you can ingest and view sensitive application data entirely on-premises, while still exporting system data to Logging on Google Cloud. The following diagram illustrates this architecture.

Separating application and system data with Anthos.

This architecture has the following advantages:

  • Sensitive application data is kept entirely on-premises.
  • On-premises monitoring and logging have no cloud dependencies and remain available even if the network connection is interrupted.
  • All GKE system data, both on-premises and Google Cloud, is centralized in Logging and is also accessible to Google Cloud Support as needed.

For an example implementation using Elastic Stack as the partner solution, see Monitoring Anthos with the Elastic Stack.

What's next