Calculate metrics from metrics data

Calculating metrics means generating new metrics based on pre-existing logs or metrics for data observability. You calculate metrics in GDC by configuring recording rules in a MonitoringRule custom resource. Recording rules precompute expressions that are frequently needed or computationally expensive. These rules save their result as a new set of time series. Querying the precomputed result is faster than executing the original expression whenever needed. This precomputation is especially useful for dashboards and complex alerts, which must query the same expression repeatedly every time they refresh or are evaluated.

You configure recording rules in the MonitoringRule custom resource to calculate metrics based on other system monitoring metrics. The MonitoringRule custom resource contains recording rules that describe the conditions to calculate new metrics based on pre-existing metrics.

Before you begin

To get the permissions you need to calculate metrics from metrics, ask your Project IAM Admin to grant you one of the following roles:

  • Monitoring Rule Editor: edits or modifies MonitoringRule custom resources. Request the Monitoring Rule Editor (monitoringrule-editor) role.
  • Monitoring Rule Viewer: views MonitoringRule custom resources. Request the Monitoring Rule Viewer (monitoringrule-viewer) role.

Create recording rules

Define a record name and a valid expression that evaluates the rule. The expression must resolve to a numeric value to record it as a new metric. Deploy the custom resource in your project namespace on the admin cluster to create the recording rules.

For more information about recording rules, see https://grafana.com/docs/loki/latest/rules/

Work through the following steps to create record rules and calculate metrics in your project's namespace:

  1. Open or create the YAML file for the CR using the MonitoringRule CR template for monitoring record rules.
  2. In the namespace field of the CR, enter your project's namespace.
  3. In the name field, enter the name for the rule configuration.
  4. Optional: You can choose the log source in the source field. Valid options are operational and audit.
  5. In the interval field, enter the number of seconds for the duration of the rule evaluation interval.
  6. Optional: In the limit field, enter the maximum number of alerts. Enter 0 for unlimited alerts.

  7. In the recordRules field, enter the following information to calculate metrics:

    • In the record field, enter the record name. This value defines the time series in which to write the record rule and it must be a valid metric name.

    • In the expr field, enter a PromQL expression for the metric rule.

      This expression must resolve to a numeric value to be recorded as a new metric.

    • Optional: In the labels field, define the labels that you want to add or overwrite as key-value pairs.

  8. Save the YAML file of the CR.

  9. Deploy the CR in your project's namespace of the admin cluster to create the record rules.

The MonitoringRule record rules

A MonitoringRule custom resource contains recording rules that describe the conditions to calculate new metrics based on pre-existing system monitoring metrics.

The following YAML file shows a template for the recordRules field of the MonitoringRule CR.

# Configures either an alert or a target record for precomputation
apiVersion: monitoring.gdc.goog/v1alpha1
kind: MonitoringRule
metadata:
  # Choose namespace that matches the project's namespace
  # Note: The alert or record will be produced in the same namespace
  namespace: PROJECT_NAMESPACE
  name: alerting-config
spec:
  # Rule evaluation interval
  interval: <duration>

  # Configure limit for number of alerts (0: no limit)
  # Optional, Default: 0 (no limit)
  limit: <int>

  # Configure record rules to generate new metrics based on pre-existing metrics.
  # Record rules precompute expressions that are frequently needed or computationally expensive.
  # These rules save their result as a new set of time series.
  recordRules:
    # Define which timeseries to write to (must be a valid metric name)
  - record: <string>

    # Define PromQL expression to evaluate for this rule
    expr: <string>

    # Define labels to add or overwrite
    # Optional, Map of {key, value} pairs
    labels:
      <labelname>: <labelvalue>
...

Replace PROJECT_NAMESPACE with the namespace of your project.

The LoggingRule record rules

A LoggingRule CR contains record rules that describe the conditions to calculate new metrics based on pre-existing logs.

The following YAML file shows a template for the recordRules field of the LoggingRule CR.

# Configures either an alert or a target record for precomputation
apiVersion: logging.gdc.goog/v1alpha1
kind: LoggingRule
metadata:
  # Choose namespace that matches the project's namespace
  # Note: The alert or record will be produced in the same namespace
  namespace: g-fleetns-a
  name: alerting-config
spec:
  # Choose which log source to base alerts on (Operational/Audit/Security Logs)
  # Optional, Default: Operational
  source: <string>

  # Rule evaluation interval
  interval: <duration>

  # Configure limit for number of alerts (0: no limit)
  # Optional, Default: 0 (no limit)
  limit: <int>

  # Configure record rules to generate new metrics based on pre-existing logs.
  # Record rules generate metrics based on logs.
  # Use record rules for complex alerts, which query the same expression repeatedly every time they are evaluated.
  recordRules:
    # Define which timeseries to write to (must be a valid metric name)
  - record: <string>

    # Define LogQL expression to evaluate for this rule
    # https://grafana.com/docs/loki/latest/rules/
    expr: <string>

    # Define labels to add or overwrite
    # Optional, Map of {key, value} pairs
    labels:
      <labelname>: <labelvalue>
...