This document shows how to run the OpenTelemetry Collector in a GKE cluster to collect OTLP logs, metrics, and traces from instrumented applications and export that data to Google Cloud.
Before you begin
Running the OpenTelemetry Collector on GKE requires the following resources:
A Google Cloud project with the Cloud Monitoring API, Cloud Trace API, and Cloud Logging API enabled.
If you don't have a Google Cloud project, then do the following:
In the Google Cloud console, go to New Project:
In the Project Name field, enter a name for your project and then click Create.
Go to Billing:
Select the project you just created if it isn't already selected at the top of the page.
You are prompted to choose an existing payments profile or to create a new one.
The Monitoring API, Trace API, and Logging API are enabled by default for new projects.
If you already have a Google Cloud project, then ensure that the Monitoring API, Trace API, and Logging API is enabled:
Go to APIs & services:
Select your project.
Click add Enable APIs and services.
Search for each API by name.
In the search results, click the named API. The Monitoring API appears as "Stackdriver Monitoring API".
If "API enabled" is not displayed, then click the Enable button.
A Kubernetes cluster. If you don't have a Kubernetes cluster, then follow the instructions in the Quickstart for GKE.
The following command-line tools:
gcloud
kubectl
The
gcloud
andkubectl
tools are part of the Google Cloud CLI. For information about installing them, see Managing Google Cloud CLI components. To see the gcloud CLI components you have installed, run the following command:gcloud components list
Deploy the Collector
The Collector pipeline can be deployed directly from GitHub with the following commands after replacing PROJECT_ID with the ID of your Google Cloud project:
export GCLOUD_PROJECT=PROJECT_ID kubectl kustomize https://github.com/GoogleCloudPlatform/otlp-k8s-ingest.git/k8s/base | envsubst | kubectl apply -f -
Observe and debug the Collector
The OpenTelemetry Collector provides self-observability metrics out of the box to help you monitor its performance and ensure continued uptime of the OTLP ingestion pipeline.
To monitor the Collector, install the sample dashboard for the Collector. This dashboard offers at-a-glance insights into several metrics from the Collector, including uptime, memory usage, and API calls to Google Cloud Observability.
To install the dashboard, do the following:
-
In the Google Cloud console, go to the Dashboards page:
If you use the search bar to find this page, then select the result whose subheading is Monitoring.
- Select the Sample Library tab.
- Select the OpenTelemetry Collector category.
- Select the "OpenTelemtry Collector" dashboard.
- Click Import.
For more information about the installation process, see Install sample dashboards.
Configure the Collector
The self-managed OTLP ingest pipeline includes a default OpenTelemetry Collector configuration that is designed to deliver high volumes of OTLP metrics, logs, and traces with consistent GKE and Kubernetes metadata attached. It's also designed to prevent common ingestion issues.
However, you might have unique needs that require customization of the default config. This section describes the defaults shipped with the pipeline and how you can customize those defaults to fit your needs.
The default Collector configuration is located on GitHub as
config/collector.yaml
:
Exporters
The default exporters include the
googlecloud
exporter (for logs and traces) and the
googlemanagedprometheus
exporter (for metrics).
The googlecloud
exporter is configured with a default log name. The
googlemanagedprometheus
exporter does not require any default configuration;
see Get started with the OpenTelemetry
Collector
in the Google Cloud Managed Service for Prometheus documentation for more information on
configuring this exporter.
Processors
The default configuration includes the following processors:
batch
: Configured to batch telemetry requests at the Google Cloud maximum number of entries per request, or at the Google Cloud minimum interval of every 5 seconds (whichever comes first).k8sattributes
: Automatically maps Kubernetes resource attributes to telemetry labels.memory_limiter
: Caps Collector memory usage at a reasonable level to prevent out-of-memory crashes by dropping data points beyond this level.resourcedetection
: Automatically detects Google Cloud resource labels such as cluster name and project ID.transform
: Renames metric labels which would collide with Google Cloud monitored resource fields.
Receivers
The default configuration only includes the otlp
receiver.
See Choose an instrumentation
approach for detailed
instructions on instrumenting your applications to push OTLP traces and metrics
to the Collector's OTLP endpoint.
Next steps: collect and view telemetry
This section describes deploying a sample application and pointing that application to the Collector's OTLP endpoint, and viewing the telemetry in Google Cloud. The sample application is a small generator that exports traces, logs, and metrics to the Collector.
If you already have an application instrumented with an OpenTelemetry SDK, then you can point your application to the Collector's endpoint instead.
To deploy the sample application, run the following command:
kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/otlp-k8s-ingest/main/sample/app.yaml
To point existing applications that use the OpenTelemetry SDK at the Collector's
endpoint, set the OTEL_EXPORTER_OTLP_ENDPOINT
environment variable to
http://opentelemetry-collector.opentelemetry.svc.cluster.local:4317
.
After a few minutes, telemetry generated by the application begins flowing through the Collector to the Google Cloud console for each signal.