Calculate PA metrics based on log data

Calculating metrics means generating new metrics based on pre-existing logs or metrics for data observability. You calculate metrics in Google Distributed Cloud (GDC) air-gapped by configuring recording rules in a LoggingRule custom resource. Recording rules precompute expressions that are frequently needed or computationally expensive. These rules save their result as a new set of time series. Querying the precomputed result is faster than executing the original expression whenever needed. This precomputation is especially useful for dashboards and complex alerts, which must query the same expression repeatedly every time they refresh or are evaluated.

You configure recording rules in the LoggingRule custom resource to calculate metrics based on system logging data. The LoggingRule custom resource contains recording rules that describe the conditions to calculate new metrics based on pre-existing logs.

Before you begin

To get the permissions you need to calculate metrics from logs, ask your Organization IAM Admin to grant you one of the following roles in the platform-obs namespace:

  • Logging Rule PA Creator: creates LoggingRule custom resources. Request the Logging Rule PA Creator (loggingrule-pa-creator) cluster role.
  • Logging Rule PA Editor: edits or modifies LoggingRule custom resources. Request the Logging Rule PA Editor (loggingrule-pa-editor) cluster role.
  • Logging Rule PA Viewer: views LoggingRule custom resources. Request the Logging Rule PA Viewer (loggingrule-pa-viewer) cluster role.

Create recording rules

Define a record name and a valid expression that evaluates the rule. The expression must resolve to a numeric value to record it as a new metric. Deploy the custom resource in your project namespace on the org admin cluster to create the recording rules for the Logging platform.

For more information about recording rules for Logging, see https://grafana.com/docs/loki/latest/rules/

Work through the following steps to create recording rules and calculate metrics in your project namespace:

  1. Open or create the YAML file for the custom resource using the LoggingRule custom resource template for log-based recording rules.
  2. In the namespace field of the custom resource, enter your project namespace.
  3. In the name field, enter the name for the rule configuration.
  4. Choose the log source in the source field. Valid options are operational and audit.
  5. In the interval field, enter the number of seconds for the duration of the rule evaluation interval.
  6. Optional: In the limit field, enter the maximum number of alerts. Enter 0 for unlimited alerts.

  7. In the recordRules field, enter the following information to calculate metrics:

    • In the record field, enter the record name. This value defines the time series in which to write the recording rule and it must be a valid metric name.

    • In the expr field, enter a LogQL (Log Query Language) expression for the log rule.

      This expression must resolve to a numeric value to be recorded as a new metric.

    • Optional: In the labels field, define the labels that you want to add or overwrite as key-value pairs.

  8. Save the YAML file of the custom resource.

  9. Deploy the custom resource in your project namespace on the admin cluster to create the recording rules.

The LoggingRule recording rules

A LoggingRule custom resource contains recording rules that describe the conditions to calculate new metrics based on pre-existing system logging data for observability.

The following YAML file shows a template for the LoggingRule custom resource:

# Configures either an alert or a target record for precomputation
apiVersion: logging.gdc.goog/v1
kind: LoggingRule
metadata:
  # Choose a namespace that matches the project namespace
  # The alert or record is produced in the same namespace
  namespace: PROJECT_NAMESPACE
  name: alerting-config
spec:
  # Choose which log source to base alerts on (Operational or audit logs)
  # Optional
  # Valid options: 'operational' and 'audit'
  # Default value: 'operational'
  source: operational

  # Rule evaluation interval
  interval: 60s

  # Configure the limit for the number of alerts
  # A value of '0' means no limit
  # Optional
  # Default value: '0'
  limit: 0

  # Configure recording rules to generate new metrics based on pre-existing logs.
  # Recording rules generate metrics based on logs.
  # Use recording rules for complex alerts,
  # which query the same expression repeatedly every time they are evaluated.
  recordRules:
    # Define which timeseries to write to
    # The value must be a valid metric name
  - record: MyMetricsName

    # Define the LogQL expression to evaluate for this rule
    # For more information, see https://grafana.com/docs/loki/latest/rules/
    expr: rate({service_name="bob-service"} [1m])

    # Define labels to add or overwrite
    # Map of key-value pairs
    # Optional
    labels:
      verb: read
...

Replace PROJECT_NAMESPACE with the namespace of your project.