Label metrics

Before you begin

To get the permissions you need to label metrics or view metric labels, ask your Project IAM Admin to grant you one of the following roles in your project namespace:

  • Monitoring Target Editor: edits or modifies MonitoringTarget custom resources. Request the Monitoring Target Editor (monitoringtarget-editor) role.
  • Monitoring Target Viewer: views MonitoringTarget custom resources. Request the Monitoring Target Viewer (monitoringtarget-viewer) role.

Labels let you differentiate the characteristics of a metric. Apart from collecting metrics, the MonitoringTarget custom resource lets you label metrics for optional features such as the following:

For more information on metrics relabeling, see https://grafana.com/blog/2022/03/21/how-relabeling-in-prometheus-works/

Send metrics to another project

Container authors can write in their code that the system must send specific metrics to another project for system monitoring, even if the project corresponds to a different GDC persona. To send metrics to a specific project, add the _gdch_project label to a metric in your code. Set its value to the name of the project you want to own that particular observability metric. For example, "_gdch_project": "another-project-name".

Alternatively, use the MonitoringTarget custom resource to send all the metric data it collects to another project for data observability. To do so, set the _gdch_project label as a target label in the metricsRelabelings field and establish a replacement value.

The following code sample shows how to replace the _gdch_project label value with another-project-name in the MonitoringTarget custom resource:

apiVersion: monitoring.gdc.goog/v1alpha1
kind: MonitoringTarget
metadata:
  # Choose the same namespace as the workload pods
  namespace: PROJECT_NAMESPACE
  name: string
spec:
  ...
  podMetricsEndpoints:
    ...
    metricsRelabelings:
      ...
      - action: replace
        targetLabel: _gdch_project
        replacement: another-project-name

Replace PROJECT_NAMESPACE with the namespace of your project.

Add new labels to metrics

Labels are useful for data observability and system monitoring because they let you identify different characteristics of a metric. You can add new labels to metrics that the scrapped containers don't initially expose. In the MonitoringTarget custom resource, set the new label as a target and establish a replacement to serve as the value of the label. The system adds the new key-value pair to all metrics the custom resource collects for your data observability purposes.

The following code sample shows how to add the my_new_label label and set its value to my_label_value in the MonitoringTarget custom resource:

apiVersion: monitoring.gdc.goog/v1alpha1
kind: MonitoringTarget
metadata:
  # Choose the same namespace as the workload pods
  namespace: PROJECT_NAMESPACE
  name: string
spec:
  ...
  podMetricsEndpoints:
    ...
    metricsRelabelings:
      ...
      - action: replace
        targetLabel: my_new_label
        replacement: my_label_value

Replace PROJECT_NAMESPACE with the namespace of your project.

Rename a metric

You can entirely rename a metric a container produces from the MonitoringTarget CR. Set the original name of the metric and its replacement in the metricsRelabelings field. The custom resource takes the original metric name as a regular expression and replaces it with the new name you added.

The following code sample shows how to replace the cpu_usage metric name with cpu_usage_new_name in the MonitoringTarget CR:

apiVersion: monitoring.gdc.goog/v1alpha1
kind: MonitoringTarget
metadata:
  # Choose the same namespace as the workload pods
  namespace: PROJECT_NAMESPACE
  name: string
spec:
  ...
  podMetricsEndpoints:
    ...
    metricsRelabelings:
      ...
      - action: replace
        regex: cpu_usage
        replacement: cpu_usage_new_name
        sourceLabels: [__name__]
        targetLabel: __name__

Replace PROJECT_NAMESPACE with the namespace of your project.