Calculating metrics means generating new metrics based on pre-existing logs or metrics for data observability. You calculate metrics in Google Distributed Cloud (GDC) air-gapped by configuring recording rules in a MonitoringRule
custom resource. Recording rules precompute expressions that are frequently needed or computationally expensive. These rules save their result as a new set of time series. Querying the precomputed result is faster than executing the original expression whenever needed. This precomputation is especially useful for dashboards and complex alerts, which must query the same expression repeatedly every time they refresh or are evaluated.
You configure recording rules in the MonitoringRule
custom resource to calculate metrics based on other system monitoring metrics. The MonitoringRule
custom resource contains recording rules that describe the conditions to calculate new metrics based on pre-existing metrics.
Before you begin
To get the permissions you need to calculate metrics from metrics, ask your Project IAM Admin to grant you one of the following roles in your project namespace:
- Monitoring Rule Editor: edits or modifies
MonitoringRule
custom resources. Request the Monitoring Rule Editor (monitoringrule-editor
) role. - Monitoring Rule Viewer: views
MonitoringRule
custom resources. Request the Monitoring Rule Viewer (monitoringrule-viewer
) role.
Create recording rules
Define a record name and a valid expression that evaluates the rule. The expression must resolve to a numeric value to record it as a new metric. Deploy the custom resource in your project namespace on the org admin cluster to create the recording rules for the Observability platform.
For more information about recording rules for Observability, see https://grafana.com/docs/loki/latest/rules/
Work through the following steps to create recording rules and calculate metrics in your project namespace:
- Open or create the YAML file for the custom resource using the
MonitoringRule
custom resource template for metric-based recording rules. - In the
namespace
field of the custom resource, enter your project namespace. - In the
name
field, enter the name for the rule configuration. - In the
interval
field, enter the number of seconds for the duration of the rule evaluation interval. - Optional: In the
limit
field, enter the maximum number of alerts. Enter0
for unlimited alerts. In the
recordRules
field, enter the following information to calculate metrics:- In the
record
field, enter the record name. This value defines the time series in which to write the recording rule and it must be a valid metric name. In the
expr
field, enter a PromQL (Prometheus Query Language) expression for the metric rule.This expression must resolve to a numeric value to be recorded as a new metric.
Optional: In the
labels
field, define the labels that you want to add or overwrite as key-value pairs.
- In the
Save the YAML file of the custom resource.
Deploy the custom resource in your project namespace on the admin cluster to create the recording rules.
The MonitoringRule
recording rules
A MonitoringRule
custom resource contains recording rules that describe the conditions to calculate new metrics based on pre-existing system monitoring metrics for observability.
The following YAML file shows a template for the MonitoringRule
custom resource:
# Configures either an alert or a target record for precomputation
apiVersion: monitoring.gdc.goog/v1
kind: MonitoringRule
metadata:
# Choose a namespace that matches the project namespace
# The alert or record is produced in the same namespace
namespace: PROJECT_NAMESPACE
name: alerting-config
spec:
# Rule evaluation interval
interval: 60s
# Configure the limit for the number of alerts
# A value of '0' means no limit
# Optional
# Default value: '0'
limit: 0
# Configure recording rules to generate new metrics based on pre-existing metrics.
# Recording rules generate metrics based on metrics.
# Recording rules precompute expressions that are frequently needed or computationally expensive.
# These rules save their result as a new set of time series.
recordRules:
# Define which timeseries to write to
# The value must be a valid metric name
- record: MyMetricsName
# Define the PromQL expression to evaluate for this rule
expr: rate({service_name="bob-service"} [1m])
# Define labels to add or overwrite
# Map of key-value pairs
# Optional
labels:
verb: read
...
Replace PROJECT_NAMESPACE with the namespace of your project.