This document is one part of a two-part series on extending Cloud Logging and Cloud Monitoring to include on-premises infrastructure and apps.
- Log on-premises resources with BindPlane: Read about how Cloud Logging supports logging from on-premises resources.
- Monitor on-premises resources with BindPlane (this document): Read about how Cloud Monitoring supports monitoring of on-premises resources.
You might consider using Logging and Monitoring for logging and monitoring of your on-premises resources for the following reasons:
- You need a temporary solution as you move infrastructure to Google Cloud and you want to monitor your on-premises resources until they're decommissioned.
- You might have a diverse computing environment with multiple clouds and on-premises resources.
In either case, with the Logging and Monitoring APIs and BindPlane, you can gain visibility into your on-premises resources. This document is intended for DevOps practitioners, managers, and executives who are interested in a monitoring strategy for resources in Google Cloud and their remaining on-premises infrastructure and apps.
Ingesting metrics with Monitoring
You can get metrics into Monitoring in the following two ways:
- Use BindPlane from observIQ to ingest metrics from your on-premises or other cloud sources.
- Use OpenCensus to write to the Cloud Monitoring API.
Using BindPlane to ingest metrics
The following diagram shows the architecture of how BindPlane collects metrics and then how these metrics are ingested into Monitoring.
observIQ offers several versions of BindPlane: BindPlane for Google, self-hosted, SaaS, and Enterprise. For more information about these versions, see the BindPlane solutions page.
Advantages:
- Requires configuration, not instrumentation of your apps, which reduces time to implement.
- Included in the cost of using Monitoring.
- Supported configuration by Monitoring product and support.
- Can extend to metrics not provided by the default configuration.
Disadvantages:
- Requires the use of the observIQ BindPlane agent to relay metrics to Monitoring, which can add complexity to the overall system.
This option is the recommended method because it requires the lowest amount of effort. This solution requires configuration rather than development.
Using OpenCensus to write to the Monitoring API
The following diagram shows the architecture of how OpenCensus collects metrics and how these metrics are ingested into Monitoring.
Using the Monitoring API directly means that you need to add instrumentation code to your apps to send metrics directly to the API. You can do this directly by using the Monitoring API to write metrics or by instrumenting your app with the Monitoring exporter for OpenCensus. OpenCensus is an open source project that defines a standard data structure for traces and metrics. Using OpenCensus has the advantage of supporting multiple backends, including Monitoring. Using OpenCensus also implements all the low-level technical details of using the Monitoring API.
Advantages:
- Provides flexibility because the instrumentation required is easily implemented with the use of the OpenCensus Exporter
Disadvantages:
- Requires a separate solution for infrastructure metrics by writing a custom agent.
- Requires app instrumentation, which might mean higher cost to implement.
- Requires open source libraries.
This option isn't the recommended method because it requires the highest amount of effort and doesn't cover infrastructure metrics.
Using BindPlane
This document covers using BindPlane from observIQ to ingest metrics into Monitoring. The BindPlane service works by defining a series of sources, ingesting those metrics, and then sending the metrics to Monitoring as the destination. BindPlane supports agents that run on select versions of Windows, Linux, and Kubernetes.
Sources, agents, destinations, and processors
BindPlane has the following features:
- Sources: Items that generate metrics such as Google Kubernetes Engine (GKE), Amazon Elastic Container Service for Kubernetes (Amazon EKS), or Microsoft Azure Container Service.
- Agents: Lightweight processes that remotely monitor your environment and forward metric data to BindPlane.
- Destinations: Services that BindPlane forwards the metrics. In this case, the destination is the process on BindPlane that uses the Monitoring API to write metrics to Monitoring.
- Processors: Configurations that can transform your data before it arrives at your destination. This includes adding attributes, filtering, and converting logs to metrics.
For more detailed information about sources, agents, destinations, and processors, see the BindPlane QuickStart Guide.
Example use case
As an example, ExampleOrganization has resources deployed to Google Cloud, Microsoft Azure, and on-premises resources deployed by using vSphere. In Google Cloud, there is a GKE cluster and a demo app deployed, which runs the company's website. In the Microsoft Azure environment, Azure Kubernetes Service (AKS) is running a set of microservices, providing a REST API endpoint to external developers. In the vSphere environment, MySQL, Oracle, and Microsoft SQL Server support several corporate apps.
With resources in each environment, ExampleOrganization wants to monitor each component regardless of where the component is deployed. Sending the metrics from each environment to Logging and Monitoring by using BindPlane brings all the metrics into a single location for monitoring and alerting purposes.
Send metrics from BindPlane to Monitoring
After BindPlane is set up and begins sending metrics, those metrics are sent to your Monitoring Workspace. You can then use Monitoring to view, configure, alert, and build dashboards from the time series like you can for any metrics or time series in Monitoring. For more information, see Metrics, time series, and resources.
Use metrics in Monitoring
In the previous example, BindPlane was configured to send metrics from Google Cloud, Microsoft Azure, and on-premises sources. The following three metrics appear in Monitoring:
- GKE cluster metrics
- AKS cluster metrics
- vSphere on-premises database metrics
GKE cluster metrics
If you have GKE clusters set up, the GKE cluster metrics appear in the Kubernetes Clusters page or Kubernetes Workloads page. You can see multiple views of the Kubernetes components running in Monitoring. The metrics, logs, and configuration are available for each pod.
For details, see View observability metrics.
AKS cluster metrics
In the same Monitoring environment, metrics for AKS are collected. The metrics appear in Monitoring and can be used for any purposes in Monitoring including dashboards, alerting, and the Metrics Explorer.
The Metrics Explorer page provides a way to find, filter,
and build charts from metrics. Note that the metrics sent in by BindPlane have
the workload.googleapis.com/THIRD_PARTY_APP_NAME
prefix for the metric name.
The Metrics Explorer can produce a chart for the metric. For more information about charts, see Create charts with Metrics Explorer.
Like all metrics in Monitoring, you can use these metrics to build dashboards that display multiple charts. The dashboard can represent metrics produced by AKS, collected by BindPlane, and stored in Monitoring. For more information about dashboards, see View and customize Google Cloud dashboards.
vSphere on-premises cluster metrics
The last part of this example includes database metrics from vSphere. The metrics from vSphere appear in Monitoring and can be used in the same way as any other metric in Monitoring. The Oracle metrics from vSphere appear in the list of metrics on the Metrics Explorer page.
Like all metrics in Monitoring, metrics can be used to build alerts. The alert can represent metrics produced by Oracle running in vSphere, collected by BindPlane, stored in Monitoring. For more information about alerts, see Alerting overview.
Conclusion
Monitoring provides dashboards, alerting, and incident response for you to get insights into your platforms. Together, Monitoring and BindPlane provide you the ability to gain visibility into your on-premises resources.
What's next
- Cloud Logging and Cloud Monitoring
- BindPlane QuickStart Guide
- For more reference architectures, diagrams, and best practices, explore the Cloud Architecture Center.