Send Config Sync metrics to a custom monitoring system

The page describes how to send metrics from Config Sync to a custom OpenTelemetry exporter.

Caution: This solution is provided for convenience and is not maintained by Config Sync. The custom configurations override the default settings and persist between upgrades. Config Sync tries to maintain the consistency of metrics.

Config Sync uses OpenCensus to create and record metrics and OpenTelemetry to export its metrics. This page explains how to configure those metrics to a custom exporter. For other ways to export metrics, see Monitor Config Sync with Cloud Monitoring or Monitor Config Sync with Prometheus.

Configure a custom OpenTelemetry exporter

If you want to send your metrics to a monitoring system other than Prometheus or Cloud Monitoring, you can modify the OpenTelemetry configuration. For a list of supported monitoring systems, see OpenTelemetry Collector Exporters and OpenTelemetry Collector-Contrib Exporters.

OpenTelemetry monitoring resources are managed in a separate config-management-monitoring namespace. To configure a custom OpenTelemetry exporter for use with Config Sync, you need to create a ConfigMap with the name otel-collector-custom in that config-management-monitoring namespace. The ConfigMap should contain a otel-collector-config.yaml key and the value should be the file contents of the custom OpenTelemetry Collector configuration. For more information on the configuration options, see the OpenTelemetry Collector configuration documentation.

The following ConfigMap is an example of a ConfigMap with a custom logging exporter:

# otel-collector-custom-cm.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: otel-collector-custom
  namespace: config-management-monitoring
  labels:
    app: opentelemetry
    component: otel-collector
data:
  otel-collector-config.yaml: |
    receivers:
      opencensus:
    exporters:
      logging:
        logLevel: debug
    processors:
      batch:
    extensions:
      health_check:
    service:
      extensions: [health_check]
      pipelines:
        metrics:
          receivers: [opencensus]
          processors: [batch]
          exporters: [logging]

All custom configurations must define an opencensus receiver and metrics pipeline. The rest of the fields are optional and configurable, but we recommend that you include a batch processor and health check extension like in the example.

After you have created the ConfigMap, use kubectl to create the resource:

kubectl apply -f otel-collector-custom-cm.yaml

The OpenTelemetry Collector Deployment picks up this ConfigMap and automatically restarts to apply the custom configuration.

Limitations

When you upgrade Config Sync to a new version, if you created a otel-collector-custom ConfigMap for a previous version, your custom settings might not be compatible with the new version of Config Sync. For example, metrics names, labels, and attributes can change between Config Sync versions.

When changes are made to Config Sync metrics, they are announced in the release notes. These changes might require you to update your custom otel-collector configuration.

Because of this limitation, we recommend that you only use custom metrics configurations for temporary workarounds. If you find you need to use a custom configuration, file a support ticket to request the workaround as a feature or bug fix.