Label PA metrics

Labels let you differentiate the characteristics of a metric. Apart from collecting metrics, the MonitoringTarget custom resource lets you label metrics for optional observability features such as the following:

• Send metrics to another project
• Add new labels to metrics
• Rename a metric

Important: Every unique combination of key-value label pairs represents a new time series, which can very much increase the amount of data stored. Don't use labels to store dimensions with high cardinality (many different label values), such as user IDs, email addresses, or other unbounded sets of values.

For more information on metrics relabeling, see https://grafana.com/blog/2022/03/21/how-relabeling-in-prometheus-works/

Before you begin

To get the permissions you need to label metrics or view metric labels, ask your Organization IAM Admin to grant you one of the following roles in the platform-obs namespace:

• Monitoring Target PA Creator: creates MonitoringTarget custom resources. Request the Monitoring Target PA Creator (monitoringtarget-pa-creator) cluster role.
• Monitoring Target PA Editor: edits or modifies MonitoringTarget custom resources. Request the Monitoring Target PA Editor (monitoringtarget-pa-editor) cluster role.
• Monitoring Target PA Viewer: views MonitoringTarget custom resources. Request the Monitoring Target PA Viewer (monitoringtarget-pa-viewer) cluster role.

Send metrics to another project

Container authors can write in their code that the system must send specific metrics to another project for system monitoring, even if the project corresponds to a different GDC persona. To send metrics to a specific project, add the _gdch_project label to a metric in your code. Set its value to the name of the project you want to own that particular observability metric. For example, "_gdch_project": "another-project-name".

Alternatively, use the MonitoringTarget custom resource to send all the metric data it collects to another project for data observability. To do so, set the _gdch_project label as a target label in the metricsRelabelings field and establish a replacement value.

The following code sample shows how to replace the _gdch_project label value with another-project-name in the MonitoringTarget custom resource:

apiVersion: monitoring.gdc.goog/v1
kind: MonitoringTarget
metadata:
  # Choose the same namespace as the workload pods
  namespace: PROJECT_NAMESPACE
  name: string
spec:
  ...
  podMetricsEndpoints:
    ...
    metricsRelabelings:
      ...
      - action: replace
        targetLabel: _gdch_project
        replacement: another-project-name

Replace PROJECT_NAMESPACE with the namespace of your project.

Add new labels to metrics

Labels are useful for data observability and system monitoring because they let you identify different characteristics of a metric. You can add new labels to metrics that the scrapped containers don't initially expose. In the MonitoringTarget custom resource, set the new label as a target and establish a replacement to serve as the value of the label. The system adds the new key-value pair to all metrics the custom resource collects for your data observability purposes.

The following code sample shows how to add the my_new_label label and set its value to my_label_value in the MonitoringTarget custom resource:

apiVersion: monitoring.gdc.goog/v1
kind: MonitoringTarget
metadata:
  # Choose the same namespace as the workload pods
  namespace: PROJECT_NAMESPACE
  name: string
spec:
  ...
  podMetricsEndpoints:
    ...
    metricsRelabelings:
      ...
      - action: replace
        targetLabel: my_new_label
        replacement: my_label_value

Replace PROJECT_NAMESPACE with the namespace of your project.

Rename a metric

You can entirely rename a metric a container produces from the MonitoringTarget custom resource. Set the original name of the metric and its replacement in the metricsRelabelings field. The custom resource takes the original metric name as a regular expression and replaces it with the new name you added.

The following code sample shows how to replace the cpu_usage metric name with cpu_usage_new_name in the MonitoringTarget custom resource:

apiVersion: monitoring.gdc.goog/v1
kind: MonitoringTarget
metadata:
  # Choose the same namespace as the workload pods
  namespace: PROJECT_NAMESPACE
  name: string
spec:
  ...
  podMetricsEndpoints:
    ...
    metricsRelabelings:
      ...
      - action: replace
        regex: cpu_usage
        replacement: cpu_usage_new_name
        sourceLabels: [__name__]
        targetLabel: __name__

Replace PROJECT_NAMESPACE with the namespace of your project.