Get started with self-deployed collection

This document describes how to set up Google Cloud Managed Service for Prometheus with self-deployed collection. An example application is deployed to a Kubernetes cluster and is monitored by a Prometheus server that stores collected metrics in Monarch.

This document shows you how to do the following:

  • Set up your environment and command-line tools.
  • Configure a service account for Workload Identity-enabled clusters.
  • Run the drop-in Prometheus binary on Kubernetes.
  • Control which metrics are ingested into Managed Service for Prometheus.
  • Integrate Managed Service for Prometheus with prometheus-operator setups.
  • Manually compile and run the Prometheus binary.

With self-deployed data collection, you manage your Prometheus installation as you always have. The only difference is that you run the Managed Service for Prometheus drop-in replacement binary, gke.gcr.io/prometheus-engine/prometheus:v2.41.0-gmp.9-gke.0, instead of the upstream Prometheus binary.

The drop-in binary provides additional configuration options with the --export.* flags. For more information, see the output of the --help option. This document points out the most important options.

Managed Service for Prometheus does not support exporting metrics from a federation server or from a server used as a remote-write receiver. You can replicate all federation server functionality, including reducing ingestion volume by aggregating data before sending to Monarch, by using filters and local aggregations.

Streaming data to Managed Service for Prometheus consumes additional resources. If you are self-deploying collectors, then we recommend increasing CPU and memory limits by 5x and adjusting them based on actual usage.

For more information about managed and self-deployed data collection, see Data collection with Managed Service for Prometheus.

Before you begin

This section describes the configuration needed for the tasks described in this document.

Set up projects and tools

To use Google Cloud Managed Service for Prometheus, you need the following resources:

  • A Google Cloud project with the Cloud Monitoring API enabled.

    • If you don't have a Google Cloud project, then do the following:

      1. In the Google Cloud console, go to New Project:

        Create a New Project

      2. In the Project Name field, enter a name for your project and then click Create.

      3. Go to Billing:

        Go to Billing

      4. Select the project you just created if it isn't already selected at the top of the page.

      5. You are prompted to choose an existing payments profile or to create a new one.

      The Monitoring API is enabled by default for new projects.

    • If you already have a Google Cloud project, then ensure that the Monitoring API is enabled:

      1. Go to APIs & services:

        Go to APIs & services

      2. Select your project.

      3. Click Enable APIs and Services.

      4. Search for "Monitoring".

      5. In the search results, click through to "Cloud Monitoring API".

      6. If "API enabled" is not displayed, then click the Enable button.

  • A Kubernetes cluster. If you do not have a Kubernetes cluster, then follow the instructions in the Quickstart for GKE.

You also need the following command-line tools:

  • gcloud
  • kubectl

The gcloud and kubectl tools are part of the Google Cloud CLI. For information about installing them, see Managing Google Cloud CLI components. To see the gcloud CLI components you have installed, run the following command:

gcloud components list

Configure your environment

To avoid repeatedly entering your project ID or cluster name, perform the following configuration:

  • Configure the command-line tools as follows:

    • Configure the gcloud CLI to refer to the ID of your Google Cloud project:

      gcloud config set project PROJECT_ID
      
    • Configure the kubectl CLI to use your cluster:

      kubectl config set-cluster CLUSTER_NAME
      

    For more information about these tools, see the following:

Set up a namespace

Create the NAMESPACE_NAME Kubernetes namespace for resources you create as part of the example application:

kubectl create ns NAMESPACE_NAME

Verify service account credentials

You can skip this section if your Kubernetes cluster has Workload Identity enabled.

When running on GKE, Managed Service for Prometheus automatically retrieves credentials from the environment based on the Compute Engine default service account. The default service account has the necessary permissions, monitoring.metricWriter and monitoring.viewer, by default. If you don't use Workload Identity, and you have previously removed either of those roles from the default node service account, you will have to re-add those missing permissions before continuing.

If you are not running on GKE, see Provide credentials explicitly.

Configure a service account for Workload Identity

You can skip this section if your Kubernetes cluster does not have Workload Identity enabled.

Managed Service for Prometheus captures metric data by using the Cloud Monitoring API. If your cluster is using Workload Identity, you must grant your Kubernetes service account permission to the Monitoring API. This section describes the following:

Create and bind the service account

This step appears in several places in the Managed Service for Prometheus documentation. If you have already performed this step as part of a prior task, then you don't need to repeat it. Skip ahead to Authorize the service account.

The following command sequence creates the gmp-test-sa service account and binds it to the default Kubernetes service account in the NAMESPACE_NAME namespace:

gcloud config set project PROJECT_ID \
&&
gcloud iam service-accounts create gmp-test-sa \
&&
gcloud iam service-accounts add-iam-policy-binding \
  --role roles/iam.workloadIdentityUser \
  --member "serviceAccount:PROJECT_ID.svc.id.goog[NAMESPACE_NAME/default]" \
  gmp-test-sa@PROJECT_ID.iam.gserviceaccount.com \
&&
kubectl annotate serviceaccount \
  --namespace NAMESPACE_NAME \
  default \
  iam.gke.io/gcp-service-account=gmp-test-sa@PROJECT_ID.iam.gserviceaccount.com

If you are using a different GKE namespace or service account, adjust the commands appropriately.

Authorize the service account

Groups of related permissions are collected into roles, and you grant the roles to a principal, in this example, the Google Cloud service account. For more information about Monitoring roles, see Access control.

The following command grants the Google Cloud service account, gmp-test-sa, the Monitoring API roles it needs to write metric data.

If you have already granted the Google Cloud service account a specific role as part of prior task, then you don't need to do it again.

gcloud projects add-iam-policy-binding PROJECT_ID\
  --member=serviceAccount:gmp-test-sa@PROJECT_ID.iam.gserviceaccount.com \
  --role=roles/monitoring.metricWriter

Debug your Workload Identity configuration

If you are having trouble getting Workload Identity to work, see the documentation for verifying your Workload Identity setup and the Workload Identity troubleshooting guide.

As typos and partial copy-pastes are the most common sources of errors when configuring Workload Identity, we strongly recommend using the editable variables and clickable copy-paste icons embedded in the code samples in these instructions.

Workload Identity in production environments

The example described in this document binds the Google Cloud service account to the default Kubernetes service account and gives the Google Cloud service account all necessary permissions to use the Monitoring API.

In a production environment, you might want to use a finer-grained approach, with a service account for each component, each with minimal permissions. For more information on configuring service accounts for workload-identity management, see Using Workload Identity.

Set up self-deployed collection

This section describes how to set up and run an example application that uses self-deployed collection.

Deploy the example application

The example application emits the example_requests_total counter metric and the example_random_numbers histogram metric (among others) on its metrics port. The manifest for the application defines three replicas.

To deploy the example application, run the following command:

kubectl -n NAMESPACE_NAME apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/prometheus-engine/v0.8.2/examples/example-app.yaml

Run the replacement Prometheus binary

To ingest the metric data emitted by the example application, you deploy Google's forked version of the Prometheus server, which is configured to scrape the workload's metrics as well as its own metrics endpoint.

  1. To deploy the forked server, run the following command:

    kubectl -n NAMESPACE_NAME apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/prometheus-engine/v0.8.2/examples/prometheus.yaml
    

    This deployed Prometheus server is a thin fork of the upstream Prometheus binary. It behaves like a standard Prometheus server, but it also ingests data into Managed Service for Prometheus.

    The manifest above provides a basic working example that sends data to the global data store, Monarch. It does not persistently store a local copy of data. For information on how this predefined configuration works and how to extend it, see the open source Prometheus configuration documentation.

    The pre-built image only works on Linux nodes. To scrape targets running on Windows nodes, either deploy the server on a Linux node and configure it to scrape endpoints on the Windows nodes or build the binary for Windows yourself.

  2. Verify that the pods for the Prometheus server and the example application deployed successfully:

    kubectl -n NAMESPACE_NAME get pod
    

    If the deployment was successful, then you see output similar to the following:

    NAME                            READY   STATUS    RESTARTS   AGE
    prom-example-84c6f547f5-fglbr   1/1     Running   0          5m
    prom-example-84c6f547f5-jnjp4   1/1     Running   0          5m
    prom-example-84c6f547f5-sqdww   1/1     Running   0          5m
    prometheus-test-0               2/2     Running   1          3m
    

If you are running on GKE, then you can do the following:

If you are running outside of GKE, then you need to create a service account and authorize it to write your metric data, as described in the following section.

Provide credentials explicitly

When running on GKE, the collecting Prometheus server automatically retrieves credentials from the environment based on the node's service account or the Workload Identity setup. In non-GKE Kubernetes clusters, credentials must be explicitly provided to the collecting Prometheus server by using flags or the GOOGLE_APPLICATION_CREDENTIALS environment variable.

  1. Set the context to your target project:

    gcloud config set project PROJECT_ID
    
  2. Create a service account:

    gcloud iam service-accounts create gmp-test-sa
    

    This step creates the service account that you might have already created in the Workload Identity instructions.

  3. Grant the required permissions to the service account:

    gcloud projects add-iam-policy-binding PROJECT_ID\
      --member=serviceAccount:gmp-test-sa@PROJECT_ID.iam.gserviceaccount.com \
      --role=roles/monitoring.metricWriter
    

  4. Create and download a key for the service account:

    gcloud iam service-accounts keys create gmp-test-sa-key.json \
      --iam-account=gmp-test-sa@PROJECT_ID.iam.gserviceaccount.com
    
  5. Add the key file as a secret to your non-GKE cluster:

    kubectl -n NAMESPACE_NAME create secret generic gmp-test-sa \
      --from-file=key.json=gmp-test-sa-key.json
    

  6. Open the Prometheus StatefulSet resource for editing:

    kubectl -n NAMESPACE_NAME edit statefulset prometheus-test
    
    1. Add the text shown in bold to the resource:

      apiVersion: apps/v1
      kind: StatefulSet
      metadata:
        namespace: NAMESPACE_NAME
        name: example
      spec:
        template
          containers:
          - name: prometheus
            args:
            - --export.credentials-file=/gmp/key.json
      ...
            volumeMounts:
            - name: gmp-sa
              mountPath: /gmp
              readOnly: true
      ...
          volumes:
          - name: gmp-sa
            secret:
              secretName: gmp-test-sa
      ...
      

    2. Save the file and close the editor. After the change is applied, the pods are re-created and start authenticating to the metric backend with the given service account.

    Alternatively, instead of using the flags set in this example, you can set the key-file path by using the GOOGLE_APPLICATION_CREDENTIALS environment variable.

    Additional topics for self-deployed collection

    This section describes how to do the following:

    • Filter the data you export to the managed service.
    • Convert your existing deployment configurations.
    • Run the Prometheus binary in high-availability mode.
    • Build and run the replacement Prometheus binary.
    • Run Managed Service for Prometheus outside of Google Cloud.

    Filter exported metrics

    If you collect a lot of data, you might want to prevent some time series from being sent to Managed Service for Prometheus to keep costs down.

    You can use regular metric-relabeling configs in your Prometheus scraping configuration. With relabeling configs, you can drop metrics based on label matches at ingestion time.

    Sometimes, you might want to ingest data locally but not export it to Managed Service for Prometheus. To filter exported metrics, you can use the --export.match flag.

    The flag specifies one or more PromQL series selectors, and the flag can be used multiple times. A time series is exported to Managed Service for Prometheus if it satisfies all of the selectors in at least one of the flags; that is, when determining eligibility, conditions within a single flag are ANDed while conditions in separate flags are ORed. The following example uses two instances of the flag:

    ./prometheus \
      --export.match='{job="prometheus"}' \
      --export.match='{__name__=~"job:.+"}' \
      ...
    

    This change causes only metrics for the "prometheus" job as well as metrics produced by recording rules that aggregate to the job level (when following naming best practices) to be exported. Samples for all other series are filtered out. By default, no selectors are specified and all time series are exported.

    The --export.match flag has the same semantics as the match[] parameter for Prometheus federation. You can therefore migrate federation setups to Managed Service for Prometheus by using the selectors from your federation server directly as flags on the Prometheus servers scraped by your federation Prometheus server. Exporting metrics from a federation server to the managed service is not supported.

    To include histogram-type metrics in a filter, you must specify the _count, _sum, and _bucket metrics. You can also do this with a wildcard matcher, for example the selector {__name__=~"histogram_metric_.+"}.

    If you are using the prometheus-operator library, then set any --export.match flags using the EXTRA_ARGS environment variable of the container. For more information, see Use with prometheus-operator.

    You can combine filter flags with locally run recording rules to "roll up" data before sending to Monarch, reducing your cardinality and cost. For more information, see Cost controls and attribution.

    The Cloud Monitoring Metrics Management page provides information that can help you control the amount you spend on chargeable metrics without affecting observability. The Metrics Management page reports the following information:

    • Ingestion volumes for both byte- and sample-based billing, across metric domains and for individual metrics.
    • Data about labels and cardinality of metrics.
    • Use of metrics in alerting policies and custom dashboards.
    • Rate of metric-write errors.
    For more information about the Metrics Management page, see View and manage metric usage.

    Use with prometheus-operator

    The Managed Service for Prometheus Prometheus binary can also be used with an existing GKE Prometheus deployment managed by prometheus-operator.

    To use the managed service's binary, replace the image specification in the Prometheus resource:

      apiVersion: monitoring.coreos.com/v1
      kind: Prometheus
      metadata:
        name: NAMESPACE_NAME
        namespace: gmp-system
      spec:
        image: gke.gcr.io/prometheus-engine/prometheus:v2.41.0-gmp.9-gke.0
        ...
        replicas: 1
        serviceAccountName: default
        version: v2.35.0
        ...
    

    If you are in a Workload Identity cluster and the namespace or service account in your resource differs, repeat the Workload Identity instructions for the additional namespace and Kubernetes service account pair.

    When running on a non-GKE Kubernetes cluster, you need to manually provide credentials. To provide credentials, do the following:

    1. Add an appropriate service account key file as a secret, as described in Provide credentials explicitly.

    2. Modify the Prometheus resource to add the text shown in bold type:

        apiVersion: monitoring.coreos.com/v1
        kind: Prometheus
        metadata:
          namespace: gmp-test
          name: example
        spec:
          ...
          secrets:
          - gmp-test-sa
          containers:
          - name: prometheus
            env:
            - name: GOOGLE_APPLICATION_CREDENTIALS
              value: /gmp/key.json
            volumeMounts:
            - name: secret-gmp-test-sa
              mountPath: /gmp
              readOnly: true
      

    You can set the EXTRA_ARGS environment variable of the container to add additional flags, such as the metric filtering flags. This is done through an environment variable because the args section of the container specification is managed by Prometheus Operator.

    Use with kube-prometheus

    You can configure deployments created using the popular kube-prometheus library to use Managed Service for Prometheus.

    Kube-prometheus has some tight internal dependencies on its default namespaces and service accounts, so we recommend only changing the minimum number of fields necessary to send data to Managed Service for Prometheus.

    Within manifests/prometheus-prometheus.yaml, replace the image specification and turn off high-availability collection by reducing replicas to 1:

        apiVersion: monitoring.coreos.com/v1
        kind: Prometheus
        ...
        spec:
          image: gke.gcr.io/prometheus-engine/prometheus:v2.41.0-gmp.9-gke.0
          ...
          replicas: 1
          version: v2.35.0
          ...
      

    If you are running on GKE and have not modified the default service account on the node, applying the modified manifests should immediately start sending data to Managed Service for Prometheus. Otherwise, you might have to configure and apply a service account. When running on GKE and using workload identity, you might have to create and authorize the prometheus-k8s service account within the monitoring namespace. When running on a non-GKE Kubernetes cluster, follow the instructions in the prometheus-operator section.

    Note that kube-prometheus collects a lot of metrics by default, most of which are often not necessary in a managed Kubernetes environment like GKE. To save on ingestion costs, you can customize kube-prometheus so that it scrapes only metrics you care about and filter exported metrics aggressively.

    For more suggestions, see Cost controls and attribution.

    High-availability deployment

    The replacement Prometheus binary comes with built-in support for highly available collection by using leader election. Replicated Prometheus servers in high-availability mode both collect metrics and evaluate rules as usual, but only one of them sends data to Google Cloud Managed Service for Prometheus.

    Replicas of the same Prometheus server must always have identical configurations, including the same external_labels. This requirement differs from other systems, which rely on a special external label, such as __replica__, to make replicas explicitly different.

    The Kubernetes API server is a supported leader-election backend and can be enabled by setting the following flags:

    ./prometheus
      ...
      --export.ha.backend=kube \
      --export.ha.kube.namespace=LEASE_NAMESPACE \
      --export.ha.kube.name=LEASE_NAME
    

    The LEASE_NAMESPACE and LEASE_NAME values identify the Lease resource through which leader election takes place. All Prometheus servers pointing at the same resource belong to the same replica set. The Kubernetes service account of the Prometheus deployment needs permission to read and write the respective Lease resource. When running the Prometheus server outside of a Kubernetes cluster, you can provide an explicit config by using the --export.ha.kube.config flag.

    After you do this, you can increase the replicas value to 2 or greater.

    Binary deployments

    If you want to run in a non-containerized environment, you can build the replacement Prometheus binary directly.

    Building the source

    If you have an existing process for compiling Prometheus yourself, you can transparently substitute our GitHub repository into your process. Managed Service for Prometheus has its own version-tag extension to distinguish its releases from upstream releases.

    To build the plain binary, the Go toolchain and recent versions of NPM/Yarn must be installed on the machine. For more information, see the upstream build instructions.

    1. Clone the repository:

      git clone https://github.com/GoogleCloudPlatform/prometheus &&
      cd prometheus
      
    2. Check out the desired version tag:

      git checkout v2.41.0-gmp.9
      
    3. To create a Managed Service for Prometheus tarball, run the following commands:

      make build && make tarball
      

    The resulting tarball and binaries are fully compatible with their upstream variants in terms of directory structure and functionality.

    Limits on creating and updating metrics and labels

    Managed Service for Prometheus enforces a per-minute rate limit on creating new metrics and on adding new metric labels to existing metrics. This rate limit is usually only hit when first integrating with Managed Service for Prometheus, for example when you migrate an existing, mature Prometheus deployment to use self-deployed collection. This is not a rate limit on ingesting data points. This rate limit only applies when creating never-before-seen metrics or when adding new labels to existing metrics.

    This quota is fixed, but any issues should automatically resolve as new metrics and metric labels get created up to the per-minute limit.

    For more information, see Troubleshooting.

    Run self-deployed collection outside of Google Cloud

    In Compute Engine environments, GKE environments, or on a machine where you ran gcloud login with a sufficiently authorized account, you can run self-deployed collection without further configuration. Outside of Google Cloud, you need to explicitly provide credentials, a project_id to contain your metrics, and a location (Google Cloud region) to store your metrics in. You should also set the cluster and namespace labels, even if running in a non-Kubernetes environment.

    You can provide a service account key by using the --export.credentials-file flag or the GOOGLE_APPLICATION_CREDENTIALS environment variable as described in Provide credentials explicitly.

    We recommend choosing project_id based on your planned tenancy model for reads. Pick a project to store metrics in based on how you plan to organize reads later with metrics scopes. If you don't care, you can put everything into one project.

    For location, we recommend choosing the nearest Google Cloud region to your deployment. The further the chosen Google Cloud region is from your deployment, the more write latency you'll have and the more you'll be affected by potential networking issues. You might want to consult this list of regions across multiple clouds. If you don't care, you can put everything into one Google Cloud region. You can't use global as your location.

    If running in a Kubernetes environment, set the cluster and namespace values to the local cluster and namespace. If running outside Kubernetes, set them to values that make sense hierarchically. For example, in a VM-based environment running on AWS, set the cluster value to __aws__ and the namespace value to the instance ID. You can dynamically fill in the instance ID by using a relabeling rule that calls the local metadata server.

    For a minimal working example, you can run a local, self-monitoring Prometheus binary with the following command:

    ./prometheus \
      --config.file=documentation/examples/prometheus.yaml \
      --export.label.project-id=PROJECT_ID \
      --export.label.location=REGION \
      --export.label.cluster=CLUSTER_NAME \
    

    This example assumes you have set the REGION variable to a value like us-central1, for example.

    However, we recommend that you set the export target labels for the managed service in the global.external_labels section of your Prometheus config. For example, in Kubernetes environments you might use the following config:

    global:
      external_labels:
        project_id: PROJECT_ID
        location: REGION
        cluster: CLUSTER_NAME
        namespace: local-testing
    
    scrape_configs:
      ...
    

    Running Managed Service for Prometheus outside of Google Cloud incurs data transfer fees. There are fees to transfer data into Google Cloud, and you might incur fees to transfer data out of another cloud. You can minimize this cost by enabling compression with the --export.compression=gzip flag.

    What's next