- Logging (this solution): Read about how Logging supports logging from on-premises resources.
- Monitoring: Read about how Monitoring supports monitoring on-premises resources.
You might consider using Stackdriver for logging and monitoring of your on-premises resources for the following reasons:
- You want a temporary solution while you move infrastructure to Google Cloud Platform (GCP) and you want to log and monitor your on-premises resources until they're decommissioned.
- You might have a diverse computing environment with multiple clouds and on-premises resources.
In either case, with the Stackdriver API and BindPlane you can gain visibility into your on-premises resources. This solution is intended for DevOps practitioners, managers, and executives who are interested in a logging strategy for resources in GCP and their remaining on-premises infrastructure and apps.
Ingesting logs with Logging
You can get logs into Logging by using the API in two supported ways:
- Use the BindPlane tool from Blue Medora to ingest logs from your on-premises or other cloud sources.
- Use the Stackdriver Logging API directly from your app or by using a custom agent.
Using BindPlane to ingest Stackdriver logs
The following diagram shows the architecture of how BindPlane ingests logs and then how those logs are ingested into Logging.
BindPlane provides an integrated service to ingest on-premises logs into Logging. BindPlane uses the Logging APIs by means of a Fluentd agent to send logs to Logging, which is similar to the operation of the Logging agent. For more information, read about the architecture of BindPlane. This option requires the lowest amount of effort to deploy because it requires configuration to set up rather than development.
- Requires configuration, not development.
- Included in the cost of using Logging.
- Is a supported configuration by Logging product and support.
- Can extend to logs not provided by the default configuration.
- Requires the use of a third-party tool.
- Might need to provide the Fluentd plugin configuration if log plugin isn't provided by default. The provided list of logs is available in Sources.
Using the Logging API directly
The following diagram shows the architecture of how logs are collected by instrumentation and ingested into Logging.
Using the APIs directly means that you need to instrument your applications to send logs directly to the API or develop a custom agent to send logs to the API. This is the option that requires the highest level of effort because it requires development effort.
- Provides flexibility because you can implement the instrumentation with client logging libraries.
- Requires separate solution for infrastructure logs, such as a custom agent.
- Requires code instrumentation, which might mean higher cost to implement.
- Requires the use of batching and other scalable ingestion techniques for proper ingestion performance.
- Support is provided for the Logging API only, not custom-developed code.
This solution covers using the BindPlane tool from Blue Medora to ingest logs into Logging. Because it's included in the cost of Stackdriver, BindPlane doesn't require development and provides a product-supported solution.
Agents, sources, and destinations
BindPlane has the following features:
- Agents: Software used to collect logs, which is usually installed on the machine producing the logs. Agents also send the logs to the Logging API.
- Sources: Configuration that specifies which logs to ingest, such as
syslogs, Apache, and Kubernetes, and how to parse the logs. For more
information, see the
logs supported by BindPlane.
Any unsupported logs can be ingested using a generic
Tail input. For custom logs, you can specify a
Custom Input, which takes a custom Fluentd configuration block as input.
- Destinations: The third-party services, such as Stackdriver, to which BindPlane forwards the logs. In the case of Stackdriver, the destination is the Logging API to write metrics to Logging.
For more detailed information about agents, sources, and destinations, see getting started with BindPlane.
Example use case
As an example, ExampleOrganization has resources deployed to GCP, Microsoft Azure, and on-premises resources deployed by using vSphere. With resources in each environment, ExampleOrganization wants to collect logs for each component regardless of where the component is deployed. Sending the logs from each environment to Stackdriver by using BindPlane agents brings all the logs into a single location for centralized reporting, monitoring, and operational purposes.
In this example, Stackdriver is set up as the destination for an array of agents including those on GCP, Microsoft Azure, and vSphere. After the logs are generated, the logs are ingested into Logging by BindPlane.
Send logs from on-premises to Logging
After you set up BindPlane and start sending logs, those logs are sent to
Logging. To view, process, and export logs, go to the
Google Cloud Platform Console.
The logs are listed as
generic_taskresource types. For
more information about the labels included in each resource type, see the
Logging resource list.
Logging supports non-Stackdriver logs through the use of two resource types:
- Generic node: Identifies a machine or other computational resource for which no other resource type is applicable. The label values must uniquely identify the node.
- Generic task: Identifies an app process for which no other resource is applicable, such as a process scheduled by a custom orchestration system. The label values must uniquely identify the task.
View logs in Logging
In the GCP Console, Generic Node appears as a resource type in the list on the Logging page.
The following logs were captured as the
generic_node resource type and appear
The following log entry uses a
format, which provides a richer format for searching the logs because the log
payload is stored as a
jsonPayload. The structured logging format makes logs
more accessible because you can use the fields in the payload as a part of
the search. The BindPlane logging agent provides a mapping from the original log
entry to the structured log in Logging.
With logs available in Logging, you can take advantage of the full use of the Logging features. Logs appear in the GCP Console. You can export logs with Logging exports and use them to create metrics and alerts in Monitoring by using logs-based metrics.