Mulai menggunakan OpenTelemetry Collector

Dokumen ini menjelaskan cara menyiapkan OpenTelemetry Collector untuk menyalin metrik Prometheus standar dan melaporkan metrik tersebut ke Google Cloud Managed Service for Prometheus. OpenTelemetry Collector adalah agen yang dapat Anda deploy sendiri dan konfigurasikan untuk diekspor ke Managed Service for Prometheus. Penyiapannya mirip dengan menjalankan Google Cloud Managed Service for Prometheus dengan pengumpulan yang di-deploy sendiri.

Anda dapat memilih OpenTelemetry Collector daripada pengumpulan yang di-deploy sendiri karena alasan berikut:

• OpenTelemetry Collector memungkinkan Anda merutekan data telemetri ke beberapa backend dengan mengonfigurasi berbagai eksportir di pipeline Anda.
• Pengumpul juga mendukung sinyal dari metrik, log, dan trace, sehingga dengan menggunakannya, Anda dapat menangani ketiga jenis sinyal dalam satu agen.
• Format data OpenTelemetry yang tidak bergantung pada vendor (OpenTelemetry Protocol, atau OTLP) mendukung ekosistem library dan komponen Pengumpul yang dapat dicolokkan yang kuat. Hal ini memungkinkan berbagai opsi penyesuaian untuk menerima, memproses, dan mengekspor data Anda.

Kompromi untuk manfaat ini adalah menjalankan OpenTelemetry Collector memerlukan pendekatan deployment dan pemeliharaan mandiri. Pendekatan yang Anda pilih akan bergantung pada kebutuhan spesifik Anda, tetapi dalam dokumen ini kami menawarkan panduan yang direkomendasikan untuk mengonfigurasi OpenTelemetry Collector menggunakan Managed Service for Prometheus sebagai backend.

Catatan: Dukungan teknis Google Cloud memberikan bantuan terbatas untuk pengumpulan dengan OpenTelemetry Collector.

Sebelum memulai

Bagian ini menjelaskan konfigurasi yang diperlukan untuk tugas yang dijelaskan dalam dokumen ini.

Menyiapkan project dan alat

Untuk menggunakan Google Cloud Managed Service for Prometheus, Anda memerlukan resource berikut:

• Project Google Cloud dengan Cloud Monitoring API diaktifkan.

  • Jika Anda tidak memiliki project Google Cloud, lakukan hal berikut:

    1. Di konsol Google Cloud, buka New Project:

       Membuat Project Baru

    2. Di kolom Project Name, masukkan nama untuk project Anda, lalu klik Create.

    3. Buka Penagihan:

       Buka Penagihan

    4. Pilih project yang baru saja Anda buat jika belum dipilih di bagian atas halaman.

    5. Anda akan diminta untuk memilih profil pembayaran yang sudah ada atau membuat profil baru.

    Monitoring API diaktifkan secara default untuk project baru.

  • Jika Anda sudah memiliki project Google Cloud, pastikan Monitoring API diaktifkan:

    1. Buka API & layanan:

       Buka API & layanan

    2. Pilih project Anda.

    3. Klik Aktifkan API dan Layanan.

    4. Telusuri "Pemantauan".

    5. Di hasil penelusuran, klik "Cloud Monitoring API".

    6. Jika "API enabled" tidak ditampilkan, klik tombol Enable.

• Cluster Kubernetes. Jika Anda tidak memiliki cluster Kubernetes, ikuti petunjuk di Memulai cepat untuk GKE.

Anda juga memerlukan alat command line berikut:

• gcloud
• kubectl

Alat gcloud dan kubectl adalah bagian dari Google Cloud CLI. Untuk mengetahui informasi tentang cara menginstalnya, lihat Mengelola komponen Google Cloud CLI. Untuk melihat komponen gcloud CLI yang telah Anda instal, jalankan perintah berikut:

gcloud components list

Mengonfigurasi lingkungan Anda

Agar tidak perlu memasukkan project ID atau nama cluster berulang kali, lakukan konfigurasi berikut:

• Konfigurasikan alat command line sebagai berikut:

  • Konfigurasikan gcloud CLI untuk merujuk ke ID project Google Cloud Anda:

    gcloud config set project PROJECT_ID
    

  • Konfigurasikan kubectl CLI untuk menggunakan cluster Anda:

    kubectl config set-cluster CLUSTER_NAME
    

  Untuk informasi selengkapnya tentang alat ini, lihat referensi berikut:

  • Ringkasan gcloud CLI
  • Perintah kubectl

Menyiapkan namespace

Buat namespace Kubernetes NAMESPACE_NAME untuk resource yang Anda buat sebagai bagian dari aplikasi contoh:

kubectl create ns NAMESPACE_NAME

Memverifikasi kredensial akun layanan

Anda dapat melewati bagian ini jika cluster Kubernetes Anda telah mengaktifkan Workload Identity Federation for GKE.

Saat berjalan di GKE, Managed Service for Prometheus akan otomatis mengambil kredensial dari lingkungan berdasarkan akun layanan default Compute Engine. Akun layanan default memiliki izin yang diperlukan, monitoring.metricWriter dan monitoring.viewer, secara default. Jika Anda tidak menggunakan Workload Identity Federation untuk GKE, dan sebelumnya telah menghapus salah satu peran tersebut dari akun layanan node default, Anda harus menambahkan kembali izin yang hilang tersebut sebelum melanjutkan.

Jika Anda tidak menjalankannya di GKE, lihat Memberikan kredensial secara eksplisit.

Mengonfigurasi akun layanan untuk Workload Identity Federation for GKE

Anda dapat melewati bagian ini jika cluster Kubernetes Anda tidak mengaktifkan Workload Identity Federation untuk GKE.

Managed Service for Prometheus mengambil data metrik menggunakan Cloud Monitoring API. Jika cluster Anda menggunakan Workload Identity Federation untuk GKE, Anda harus memberikan izin akun layanan Kubernetes ke Monitoring API. Bagian ini menjelaskan hal berikut:

• Membuat akun layanan Google Cloud khusus, gmp-test-sa.
• Mengikat akun layanan Google Cloud ke akun layanan Kubernetes default di namespace pengujian, NAMESPACE_NAME.
• Memberikan izin yang diperlukan ke akun layanan Google Cloud.

Membuat dan mengikat akun layanan

Langkah ini muncul di beberapa tempat dalam dokumentasi Managed Service for Prometheus. Jika Anda telah melakukan langkah ini sebagai bagian dari tugas sebelumnya, Anda tidak perlu mengulanginya. Langsung ke Memberikan otorisasi ke akun layanan.

Urutan perintah berikut membuat akun layanan gmp-test-sa dan mengikatnya ke akun layanan Kubernetes default di namespace NAMESPACE_NAME:

gcloud config set project PROJECT_ID \
&&
gcloud iam service-accounts create gmp-test-sa \
&&
gcloud iam service-accounts add-iam-policy-binding \
  --role roles/iam.workloadIdentityUser \
  --member "serviceAccount:PROJECT_ID.svc.id.goog[NAMESPACE_NAME/default]" \
  gmp-test-sa@PROJECT_ID. \
&&
kubectl annotate serviceaccount \
  --namespace NAMESPACE_NAME \
  default \
  iam.gke.io/gcp-service-account=gmp-test-sa@PROJECT_ID.

Jika Anda menggunakan namespace atau akun layanan GKE yang berbeda, sesuaikan perintah dengan tepat.

Memberikan otorisasi ke akun layanan

Grup izin terkait dikumpulkan ke dalam peran, dan Anda memberikan peran tersebut kepada akun utama, dalam contoh ini, akun layanan Google Cloud. Untuk mengetahui informasi selengkapnya tentang peran Monitoring, lihat Kontrol akses.

Perintah berikut memberikan peran Monitoring API yang diperlukan akun layanan Google Cloud, gmp-test-sa, untuk menulis data metrik.

Jika Anda telah memberikan peran tertentu kepada akun layanan Google Cloud sebagai bagian dari tugas sebelumnya, Anda tidak perlu melakukannya lagi.

gcloud projects add-iam-policy-binding PROJECT_ID\
  --member=serviceAccount:gmp-test-sa@PROJECT_ID. \
  --role=roles/monitoring.metricWriter

Men-debug konfigurasi Workload Identity Federation for GKE

Jika Anda mengalami masalah saat membuat Workload Identity Federation for GKE berfungsi, lihat dokumentasi untuk memverifikasi penyiapan Workload Identity Federation for GKE dan panduan pemecahan masalah Workload Identity Federation for GKE.

Karena kesalahan ketik dan penyalinan sebagian adalah sumber error paling umum saat mengonfigurasi Workload Identity Federation untuk GKE, sebaiknya gunakan variabel yang dapat diedit dan ikon salin-tempel yang dapat diklik yang disematkan dalam contoh kode dalam petunjuk ini.

Workload Identity Federation for GKE di lingkungan produksi

Contoh yang dijelaskan dalam dokumen ini mengikat akun layanan Google Cloud ke akun layanan Kubernetes default dan memberi akun layanan Google Cloud semua izin yang diperlukan untuk menggunakan Monitoring API.

Dalam lingkungan produksi, Anda mungkin ingin menggunakan pendekatan yang lebih terperinci, dengan akun layanan untuk setiap komponen, masing-masing dengan izin minimal. Untuk mengetahui informasi selengkapnya tentang cara mengonfigurasi akun layanan untuk pengelolaan identitas beban kerja, lihat Menggunakan Workload Identity Federation untuk GKE.

Menyiapkan OpenTelemetry Collector

Bagian ini memandu Anda menyiapkan dan menggunakan kolektor OpenTelemetry untuk menyalin metrik dari aplikasi contoh dan mengirim data ke Google Cloud Managed Service for Prometheus. Untuk informasi konfigurasi mendetail, lihat bagian berikut:

• Mengambil metrik Prometheus
• Menambahkan prosesor
• Mengonfigurasi eksportir googlemanagedprometheus

OpenTelemetry Collector analog dengan biner agen Managed Service for Prometheus. Komunitas OpenTelemetry secara rutin memublikasikan rilis termasuk kode sumber, biner, dan image container.

Anda dapat men-deploy artefak ini di VM atau cluster Kubernetes menggunakan default praktik terbaik, atau Anda dapat menggunakan builder kolektor untuk mem-build kolektor Anda sendiri yang hanya terdiri dari komponen yang Anda perlukan. Untuk mem-build kolektor yang akan digunakan dengan Managed Service for Prometheus, Anda memerlukan komponen berikut:

• Eksportir Managed Service for Prometheus, yang menulis metrik Anda ke Managed Service for Prometheus.
• Penerima untuk meng-scrap metrik Anda. Dokumen ini mengasumsikan bahwa Anda menggunakan penerima Prometheus OpenTelemetry, tetapi pengekspor Managed Service for Prometheus kompatibel dengan penerima metrik OpenTelemetry apa pun.
• Pemroses untuk mengelompokkan dan menandai metrik Anda guna menyertakan ID resource penting, bergantung pada lingkungan Anda.

Komponen ini diaktifkan menggunakan file konfigurasi yang diteruskan ke Pengumpul dengan flag --config.

Bagian berikut membahas cara mengonfigurasi setiap komponen ini secara lebih mendetail. Dokumen ini menjelaskan cara menjalankan kolektor di GKE dan di tempat lain.

Mengonfigurasi dan men-deploy Pengumpul

Baik Anda menjalankan koleksi di Google Cloud atau di lingkungan lain, Anda tetap dapat mengonfigurasi OpenTelemetry Collector untuk diekspor ke Managed Service for Prometheus. Perbedaan terbesarnya adalah cara Anda mengonfigurasi Pengumpul. Di lingkungan non-Google Cloud, mungkin ada format data metrik tambahan yang diperlukan agar kompatibel dengan Managed Service for Prometheus. Namun, di Google Cloud, sebagian besar format ini dapat otomatis dideteksi oleh Collector.

Menjalankan OpenTelemetry Collector di GKE

Anda dapat menyalin konfigurasi berikut ke dalam file bernama config.yaml untuk menyiapkan OpenTelemetry Collector di GKE:

receivers:
  prometheus:
    config:
      scrape_configs:
      - job_name: 'SCRAPE_JOB_NAME'
        kubernetes_sd_configs:
        - role: pod
        relabel_configs:
        - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_name]
          action: keep
          regex: prom-example
        - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
          action: replace
          target_label: __metrics_path__
          regex: (.+)
        - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
          action: replace
          regex: (.+):(?:\d+);(\d+)
          replacement: $$1:$$2
          target_label: __address__
        - action: labelmap
          regex: __meta_kubernetes_pod_label_(.+)

processors:
  resourcedetection:
    detectors: [gcp]
    timeout: 10s

  transform:
    # "location", "cluster", "namespace", "job", "instance", and "project_id" are reserved, and
    # metrics containing these labels will be rejected.  Prefix them with exported_ to prevent this.
    metric_statements:
    - context: datapoint
      statements:
      - set(attributes["exported_location"], attributes["location"])
      - delete_key(attributes, "location")
      - set(attributes["exported_cluster"], attributes["cluster"])
      - delete_key(attributes, "cluster")
      - set(attributes["exported_namespace"], attributes["namespace"])
      - delete_key(attributes, "namespace")
      - set(attributes["exported_job"], attributes["job"])
      - delete_key(attributes, "job")
      - set(attributes["exported_instance"], attributes["instance"])
      - delete_key(attributes, "instance")
      - set(attributes["exported_project_id"], attributes["project_id"])
      - delete_key(attributes, "project_id")

  batch:
    # batch metrics before sending to reduce API usage
    send_batch_max_size: 200
    send_batch_size: 200
    timeout: 5s

  memory_limiter:
    # drop metrics if memory usage gets too high
    check_interval: 1s
    limit_percentage: 65
    spike_limit_percentage: 20

# Note that the googlemanagedprometheus exporter block is intentionally blank
exporters:
  googlemanagedprometheus:

service:
  pipelines:
    metrics:
      receivers: [prometheus]
      processors: [batch, memory_limiter, resourcedetection, transform]
      exporters: [googlemanagedprometheus]

Konfigurasi sebelumnya menggunakan penerima Prometheus dan eksportir Managed Service for Prometheus untuk menyalin endpoint metrik di Pod Kubernetes dan mengekspor metrik tersebut ke Managed Service for Prometheus. Pemroses pipeline memformat dan mengelompokkan data.

Untuk mengetahui detail selengkapnya tentang fungsi setiap bagian konfigurasi ini, beserta konfigurasi untuk berbagai platform, lihat bagian terperinci di bawah tentang metrik scraping dan menambahkan pemroses.

Saat menggunakan konfigurasi Prometheus yang ada dengan penerima prometheus OpenTelemetry Collector, ganti karakter $ dengan $$ to avoid triggering environment variable substitution. For more information, see Scrape Prometheus metrics.

You can modify this config based on your environment, provider, and the metrics you want to scrape, but the example config is a recommended starting point for running on GKE.

Run the OpenTelemetry Collector outside Google Cloud

Running the OpenTelemetry Collector outside Google Cloud, such as on-premises or on other cloud providers, is similar to running the Collector on GKE. However, the metrics you scrape are less likely to automatically include data that best formats it for Managed Service for Prometheus. Therefore, you must take extra care to configure the collector to format the metrics so they are compatible with Managed Service for Prometheus.

You can the following config into a file called config.yaml to set up the OpenTelemetry Collector for deployment on a non-GKE Kubernetes cluster:

receivers:
  prometheus:
    config:
      scrape_configs:
      - job_name: 'SCRAPE_JOB_NAME'
        kubernetes_sd_configs:
        - role: pod
        relabel_configs:
        - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_name]
          action: keep
          regex: prom-example
        - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
          action: replace
          target_label: __metrics_path__
          regex: (.+)
        - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
          action: replace
          regex: (.+):(?:\d+);(\d+)
          replacement: $$1:$$2
          target_label: __address__
        - action: labelmap
          regex: __meta_kubernetes_pod_label_(.+)

processors:
  resource:
    attributes:
    - key: "cluster"
      value: "CLUSTER_NAME"
      action: upsert
    - key: "namespace"
      value: "NAMESPACE_NAME"
      action: upsert
    - key: "location"
      value: "REGION"
      action: upsert

  transform:
    # "location", "cluster", "namespace", "job", "instance", and "project_id" are reserved, and
    # metrics containing these labels will be rejected.  Prefix them with exported_ to prevent this.
    metric_statements:
    - context: datapoint
      statements:
      - set(attributes["exported_location"], attributes["location"])
      - delete_key(attributes, "location")
      - set(attributes["exported_cluster"], attributes["cluster"])
      - delete_key(attributes, "cluster")
      - set(attributes["exported_namespace"], attributes["namespace"])
      - delete_key(attributes, "namespace")
      - set(attributes["exported_job"], attributes["job"])
      - delete_key(attributes, "job")
      - set(attributes["exported_instance"], attributes["instance"])
      - delete_key(attributes, "instance")
      - set(attributes["exported_project_id"], attributes["project_id"])
      - delete_key(attributes, "project_id")

  batch:
    # batch metrics before sending to reduce API usage
    send_batch_max_size: 200
    send_batch_size: 200
    timeout: 5s

  memory_limiter:
    # drop metrics if memory usage gets too high
    check_interval: 1s
    limit_percentage: 65
    spike_limit_percentage: 20

exporters:
  googlemanagedprometheus:
    project: "PROJECT_ID"

service:
  pipelines:
    metrics:
      receivers: [prometheus]
      processors: [batch, memory_limiter, resource, transform]
      exporters: [googlemanagedprometheus]

This config does the following:

• Sets up a Kubernetes service discovery scrape config for Prometheus. For more information, see scraping Prometheus metrics.
• Manually sets cluster, namespace, and location resource attributes. For more information about resource attributes, including resource detection for Amazon EKS and Azure AKS, see Detect resource attributes.
• Sets the project option in the googlemanagedprometheus exporter. For more information about the exporter, see Configure the googlemanagedprometheus exporter.

When using an existing Prometheus configuration with the OpenTelemetry Collector's prometheus receiver, replace any $ characters with $$ untuk menghindari pemicuan substitusi variabel lingkungan. Untuk informasi selengkapnya, lihat Mengambil metrik Prometheus.

Untuk informasi tentang praktik terbaik dalam mengonfigurasi Pemroses di cloud lain, lihat Amazon EKS atau Azure AKS.

Men-deploy aplikasi contoh

Aplikasi contoh memunculkan metrik penghitung example_requests_total dan metrik histogram example_random_numbers (di antara yang lainnya) di port metrics-nya. Manifes untuk contoh ini menentukan tiga replika.

Untuk men-deploy aplikasi contoh, jalankan perintah berikut:

kubectl -n NAMESPACE_NAME apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/prometheus-engine/v0.13.0/examples/example-app.yaml

Membuat konfigurasi kolektor sebagai ConfigMap

Setelah membuat konfigurasi dan menempatkannya dalam file bernama config.yaml, gunakan file tersebut untuk membuat ConfigMap Kubernetes berdasarkan file config.yaml Anda. Saat di-deploy, kolektor akan memasang ConfigMap dan memuat file.

Untuk membuat ConfigMap bernama otel-config dengan konfigurasi Anda, gunakan perintah berikut:

kubectl -n NAMESPACE_NAME create configmap otel-config --from-file config.yaml

Men-deploy kolektor

Buat file bernama collector-deployment.yaml dengan konten berikut:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: NAMESPACE_NAME:prometheus-test
rules:
- apiGroups: [""]
  resources:
  - pods
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: NAMESPACE_NAME:prometheus-test
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: NAMESPACE_NAME:prometheus-test
subjects:
- kind: ServiceAccount
  namespace: NAMESPACE_NAME
  name: default
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: otel-collector
spec:
  replicas: 1
  selector:
    matchLabels:
      app: otel-collector
  template:
    metadata:
      labels:
        app: otel-collector
    spec:
      containers:
      - name: otel-collector
        image: otel/opentelemetry-collector-contrib:0.106.0
        args:
        - --config
        - /etc/otel/config.yaml
        - --feature-gates=exporter.googlemanagedprometheus.intToDouble
        volumeMounts:
        - mountPath: /etc/otel/
          name: otel-config
      volumes:
      - name: otel-config
        configMap:
          name: otel-config

Buat deployment Collector di cluster Kubernetes Anda dengan menjalankan perintah berikut:

kubectl -n NAMESPACE_NAME create -f collector-deployment.yaml

Setelah dimulai, pod akan mengambil aplikasi contoh dan melaporkan metrik ke Managed Service for Prometheus.

Untuk informasi tentang cara membuat kueri data, lihat Membuat kueri menggunakan Cloud Monitoring atau Membuat kueri menggunakan Grafana.

Memberikan kredensial secara eksplisit

Saat berjalan di GKE, OpenTelemetry Collector akan otomatis mengambil kredensial dari lingkungan berdasarkan akun layanan node. Di cluster Kubernetes non-GKE, kredensial harus diberikan secara eksplisit ke OpenTelemetry Collector menggunakan flag atau variabel lingkungan GOOGLE_APPLICATION_CREDENTIALS.

1. Tetapkan konteks ke project target Anda:

   gcloud config set project PROJECT_ID
   

2. Buat akun layanan:

   gcloud iam service-accounts create gmp-test-sa
   

   Langkah ini akan membuat akun layanan yang mungkin sudah Anda buat di petunjuk Workload Identity Federation untuk GKE.

3. Berikan izin yang diperlukan ke akun layanan:

   gcloud projects add-iam-policy-binding PROJECT_ID\
     --member=serviceAccount:gmp-test-sa@PROJECT_ID. \
     --role=roles/monitoring.metricWriter
   

4. Buat dan download kunci untuk akun layanan:

   gcloud iam service-accounts keys create gmp-test-sa-key.json \
     --iam-account=gmp-test-sa@PROJECT_ID.
   

5. Tambahkan file kunci sebagai secret ke cluster non-GKE Anda:

   kubectl -n NAMESPACE_NAME create secret generic gmp-test-sa \
     --from-file=key.json=gmp-test-sa-key.json
   

6. Buka referensi Deployment OpenTelemetry untuk mengedit:

   kubectl -n NAMESPACE_NAME edit deployment otel-collector
   

1. Tambahkan teks yang ditampilkan dalam cetak tebal ke resource:

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     namespace: NAMESPACE_NAME
     name: otel-collector
   spec:
     template
       spec:
         containers:
         - name: otel-collector
           env:
           - name: "GOOGLE_APPLICATION_CREDENTIALS"
             value: "/gmp/key.json"
   ...
           volumeMounts:
           - name: gmp-sa
             mountPath: /gmp
             readOnly: true
   ...
         volumes:
         - name: gmp-sa
           secret:
             secretName: gmp-test-sa
   ...
   

2. Simpan file dan tutup editor. Setelah perubahan diterapkan, pod akan dibuat ulang dan mulai mengautentikasi ke backend metrik dengan akun layanan yang diberikan.

Mengambil metrik Prometheus

Bagian ini dan bagian berikutnya memberikan informasi penyesuaian tambahan untuk menggunakan OpenTelemetry Collector. Informasi ini mungkin berguna dalam situasi tertentu, tetapi tidak ada yang diperlukan untuk menjalankan contoh yang dijelaskan di Menyiapkan OpenTelemetry Collector.

Jika aplikasi Anda sudah mengekspos endpoint Prometheus, OpenTelemetry Collector dapat meng-scrape endpoint tersebut menggunakan format konfigurasi scrape yang sama yang akan Anda gunakan dengan konfigurasi Prometheus standar. Untuk melakukannya, aktifkan penerima Prometheus di konfigurasi kolektor Anda.

Konfigurasi penerima Prometheus sederhana untuk pod Kubernetes mungkin terlihat seperti berikut:

receivers:
  prometheus:
    config:
      scrape_configs:
      - job_name: 'kubernetes-pods'
        kubernetes_sd_configs:
        - role: pod
        relabel_configs:
        - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
          action: keep
          regex: true
        - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
          action: replace
          target_label: __metrics_path__
          regex: (.+)
        - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
          action: replace
          regex: (.+):(?:\d+);(\d+)
          replacement: $$1:$$2
          target_label: __address__
        - action: labelmap
          regex: __meta_kubernetes_pod_label_(.+)

service:
  pipelines:
    metrics:
      receivers: [prometheus]

Ini adalah konfigurasi scrape berbasis penemuan layanan sederhana yang dapat Anda ubah sesuai kebutuhan untuk meng-scrape aplikasi.

Saat menggunakan konfigurasi Prometheus yang ada dengan penerima prometheus OpenTelemetry Collector, ganti karakter $ dengan $$ to avoid triggering environment variable substitution. This is especially important to do for the replacement value within your relabel_configs section. For example, if you have the following relabel_config section:

- source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
  action: replace
  regex: (.+):(?:\d+);(\d+)
  replacement: $1:$2
  target_label: __address__

Then rewrite it to be:

- source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
  action: replace
  regex: (.+):(?:\d+);(\d+)
  replacement: $$1:$$2
  target_label: __address__


For more information, see the OpenTelemetry documentation.


Next, we strongly recommend that you use processors to format your metrics. In
many cases, processors must be used to properly format your metrics.

Add processors

OpenTelemetry
processors 
modify telemetry data before it is exported. You can use the processors below
to ensure that your metrics are written in a format compatible with
Managed Service for Prometheus.

Detect resource attributes

The Managed Service for Prometheus exporter for OpenTelemetry uses the
prometheus_target monitored
resource
to uniquely identify time series data points. The exporter parses the required
monitored-resource fields from resource attributes on the metric data points.
The fields and the attributes from which the values are scraped are:


project_id: auto-detected by Application Default
Credentials,
gcp.project.id, or project in exporter config (see configuring the
exporter)
location: location, cloud.availability_zone, cloud.region
cluster: cluster, k8s.cluster_name
namespace: namespace, k8s.namespace_name
job: service.name + service.namespace
instance: service.instance.id


Failure to set these labels to unique values can result in "duplicate
timeseries" errors when exporting to Managed Service for Prometheus.
Note: The terms labels and attributes, when referring to metric data points,
  represent essentially the same concept in Prometheus and OpenTelemetry,
  respectively. In this context, a Prometheus metric with the label foo will
  be converted into an OpenTelemetry data point with an attribute foo. The
  specific labels/attributes listed above are converted into resource
  attributes, which are another OpenTelemetry concept for identifying data
  points specific to the source of the data. These resource attributes are then
  mapped to the monitored resource fields listed.
The Prometheus receiver automatically sets the service.name attribute
based on the job_name in the scrape config, and service.instance.id
attribute based on the scrape target's instance. The receiver also sets
k8s.namespace.name when using role: pod in the scrape config.

We recommend populating the other attributes automatically using the resource
detection
processor.
However, depending on your environment, some attributes might not be automatically
detectable. In this case, you can use other processors to either manually
insert these values or parse them from metric labels. The following sections
illustration configurations for doing this processing on various platforms

GKE

When running OpenTelemetry on GKE, you only need to enable the
resource-detection processor to fill out the resource labels. Be sure that your
metrics don't already contain any of the reserved resource labels. If this is
unavoidable, see Avoid resource attribute collisions by renaming
attributes.

processors:
  resourcedetection:
    detectors: [gcp]
    timeout: 10s


This section can be copied directly into your config file, replacing the
processors section if it already exists.

Amazon EKS

The EKS resource detector does not automatically fill in the cluster or
namespace attributes. You can provide these values manually by using
the resource
processor,
as shown in the following example:

processors:
  resourcedetection:
    detectors: [eks]
    timeout: 10s

  resource:
    attributes:
    - key: "cluster"
      value: "my-eks-cluster"
      action: upsert
    - key: "namespace"
      value: "my-app"
      action: upsert


You can also convert these values from metric labels using the groupbyattrs
processor (see move metric labels to resource labels below).

Azure AKS

The AKS resource detector does not automatically fill in the cluster or
namespace attributes. You can provide these values manually by using the
resource
processor,
as shown in the following example:

processors:
  resourcedetection:
    detectors: [aks]
    timeout: 10s

  resource:
    attributes:
    - key: "cluster"
      value: "my-eks-cluster"
      action: upsert
    - key: "namespace"
      value: "my-app"
      action: upsert


You can also convert these values from metric labels by using the groupbyattrs
processor; see Move metric labels to resource labels.

On-premises and non-cloud environments

With on-premises or non-cloud environments, you probably can't
detect any of the necessary resource attributes automatically. In this case, you
can emit these labels in your metrics and move them to resource attributes (see
Move metric labels to resource labels), or manually set all
of the resource attributes as shown in the following example:

processors:
  resource:
    attributes:
    - key: "cluster"
      value: "my-on-prem-cluster"
      action: upsert
    - key: "namespace"
      value: "my-app"
      action: upsert
    - key: "location"
      value: "us-east-1"
      action: upsert


Create your collector config as a ConfigMap describes how
to use the config. That section assumes you have put your config in a file
called config.yaml.

The project_id resource attribute can still be automatically set when running
the Collector with Application Default
Credentials.
If your Collector does not have access to Application Default Credentials, see
Setting project_id.

Alternatively, you can manually set the resource attributes you need in an
environment variable, OTEL_RESOURCE_ATTRIBUTES, with a comma-separated list of
key/value pairs, for example:

export OTEL_RESOURCE_ATTRIBUTES="cluster=my-cluster,namespace=my-app,location=us-east-1"


Then use the env resource detector
processor 
to set the resource attributes:

processors:
  resourcedetection:
    detectors: [env]


Avoid resource attribute collisions by renaming attributes

If your metrics already contain labels that collide with the required
resource attributes (such as location, cluster, or namespace), rename them
to avoid the collision. The Prometheus convention is to add the prefix exported_
to the label name. To add this prefix, use the transform
processor.

The following processors config renames any potential collisions and
resolves any conflicting keys from the metric:

processors:
  transform:
    # "location", "cluster", "namespace", "job", "instance", and "project_id" are reserved, and
    # metrics containing these labels will be rejected.  Prefix them with exported_ to prevent this.
    metric_statements:
    - context: datapoint
      statements:
      - set(attributes["exported_location"], attributes["location"])
      - delete_key(attributes, "location")
      - set(attributes["exported_cluster"], attributes["cluster"])
      - delete_key(attributes, "cluster")
      - set(attributes["exported_namespace"], attributes["namespace"])
      - delete_key(attributes, "namespace")
      - set(attributes["exported_job"], attributes["job"])
      - delete_key(attributes, "job")
      - set(attributes["exported_instance"], attributes["instance"])
      - delete_key(attributes, "instance")
      - set(attributes["exported_project_id"], attributes["project_id"])
      - delete_key(attributes, "project_id")


Move metric labels to resource labels

In some cases, your metrics might be intentionally reporting labels such as
namespace because your exporter is monitoring multiple namespaces. For
example, when running the
kube-state-metrics 
exporter.

In this scenario, these labels can be moved to resource attributes using the
groupbyattrs
processor:

processors:
  groupbyattrs:
    keys:
    - namespace
    - cluster
    - location


In the above example, given a metric with the labels namespace, cluster,
and/or location, those labels will be converted to the matching resource
attributes.

Limit API requests and memory usage

Two other processors, the batch
processor 
and memory limiter
processor 
allow you to limit the resource consumption of your collector.

Batch processing

Batching requests lets you define how many data points to send in a single
request. Note that Cloud Monitoring has a
limit of 200 time series per
request. Enable the batch processor by using the following settings:

processors:
  batch:
    # batch metrics before sending to reduce API usage
    send_batch_max_size: 200
    send_batch_size: 200
    timeout: 5s


Memory limiting

We recommend enabling the memory-limiter processor to prevent your collector
from crashing at times of high throughput. Enable the processing by using
the following settings:

processors:
  memory_limiter:
    # drop metrics if memory usage gets too high
    check_interval: 1s
    limit_percentage: 65
    spike_limit_percentage: 20


Configure the googlemanagedprometheus exporter

By default, using the googlemanagedprometheus exporter on GKE
requires no additional configuration. For many use cases you only need to enable
it with an empty block in the exporters section:

exporters:
  googlemanagedprometheus:


However, the exporter does provide some optional configuration settings. The
following sections describe the other configuration settings.

Setting project_id

To associate your time series with a Google Cloud project, the
prometheus_target monitored resource must have project_id set.

When running OpenTelemetry on Google Cloud, the
Managed Service for Prometheus exporter defaults to setting this value
based on the Application Default
Credentials
it finds. If no credentials are available, or you want to override the default
project, you have two options:


Set project in the exporter config
Add a gcp.project.id resource attribute to your metrics.


We strongly recommend using the default (unset) value for project_id rather
than explicitly setting it, when possible.
Note: When changing the project_id, the Collector's Service Account must have
  the roles/monitoring.metricWriter IAM role for the destination
  project.
Set project in the exporter config



The following config excerpt sends metrics to
Managed Service for Prometheus in the Google Cloud project MY_PROJECT:

receivers:
  prometheus:
    config:
    ...

processors:
  resourcedetection:
    detectors: [gcp]
    timeout: 10s

exporters:
  googlemanagedprometheus:
    project: MY_PROJECT

service:
  pipelines:
    metrics:
      receivers: [prometheus]
      processors: [resourcedetection]
      exporters: [googlemanagedprometheus]


The only change from previous examples is the new line project: MY_PROJECT.
This setting is useful if you know that every metric coming through this
Collector should be sent to MY_PROJECT.

Set gcp.project.id resource attribute

You can set project association on a per-metric basis by adding a
gcp.project.id resource attribute to your metrics. Set the value of the
attribute to the name of the project the metric should be associated with.

For example, if your metric already has a label project, this label can be
moved to a resource attribute and renamed to gcp.project.id by using
processors in the Collector config, as shown in the following example:

receivers:
  prometheus:
    config:
    ...

processors:
  resourcedetection:
    detectors: [gcp]
    timeout: 10s

  groupbyattrs:
    keys:
    - project

  resource:
    attributes:
    - key: "gcp.project.id"
      from_attribute: "project"
      action: upsert

exporters:
  googlemanagedprometheus:

service:
  pipelines:
    metrics:
      receivers: [prometheus]
      processors: [resourcedetection, groupbyattrs, resource]
      exporters: [googlemanagedprometheus]


Setting client options

The googlemanagedprometheus exporter uses gRPC clients for
Managed Service for Prometheus. Therefore, optional settings
are available for configuring the gRPC client:


compression: Enables gzip compression for gRPC requests, which is useful for
minimizing data transfer fees when sending data from other clouds to
Managed Service for Prometheus (valid values: gzip).
user_agent: Overrides the user-agent string sent on requests to
Cloud Monitoring; only applies to metrics.
Defaults to the build and version number of your OpenTelemetry Collector,
for example, opentelemetry-collector-contrib 0.106.0.
endpoint: Sets the endpoint to which metric data is going to be sent.
use_insecure: If true, uses gRPC as the communication transport. Has an
effect only when the endpoint value is not "".
grpc_pool_size: Sets the size of the connection pool in the gRPC client.
prefix: Configures the prefix of metrics sent to
Managed Service for Prometheus. Defaults to
prometheus.googleapis.com.
Don't change this prefix; doing so causes metrics to not be
queryable with PromQL in the Cloud Monitoring UI.


In most cases, you don't need to change these values from their
defaults. However, you can change them to accommodate special
circumstances.

All of these settings are set under a metric block in the
googlemanagedprometheus exporter section, as shown in the following example:

receivers:
  prometheus:
    config:
    ...

processors:
  resourcedetection:
    detectors: [gcp]
    timeout: 10s

exporters:
  googlemanagedprometheus:
    metric:
      compression: gzip
      user_agent: opentelemetry-collector-contrib 0.106.0
      endpoint: ""
      use_insecure: false
      grpc_pool_size: 1
      prefix: prometheus.googleapis.com

service:
  pipelines:
    metrics:
      receivers: [prometheus]
      processors: [resourcedetection]
      exporters: [googlemanagedprometheus]


What's next




Use PromQL in Cloud Monitoring to query Prometheus metrics.
Use Grafana to query Prometheus metrics.
Set up the OpenTelemetry Collector as a sidecar agent in Cloud Run.















































































































































The Cloud Monitoring Metrics Management page provides information
that can help you control the amount you spend on billable metrics
without affecting observability. The Metrics Management page reports the
following information:

  
Ingestion volumes for both byte- and sample-based billing, across metric
    domains and for individual metrics.
  
Data about labels and cardinality of metrics.
  
Number of reads for each metric.
  
Use of metrics in alerting policies and custom dashboards.
  
Rate of metric-write errors.

You can also use the Metrics Management to 
exclude unneeded metrics,
eliminating the cost of ingesting them.






For more information about the Metrics Management page, see
View and manage metric usage.