Query and view metrics

After collecting metrics from workloads deployed in your GDC project, you can query and view metrics on dashboards from the project's system monitoring instance or query metrics from the GDC air-gapped appliance HTTP API.

Query and view metrics on dashboards

The Grafana instance of your project, which is also called the system monitoring instance, includes data to monitor project-level components for data observability, such as CPU utilization, storage consumption, network monitoring, server monitoring, logs, alerts, and metrics.

To view metrics, use dashboards on the Explore page of the system monitoring instance. To view specific metrics, make queries from the Explore page.

Before you begin

Before querying and viewing metrics on dashboards, you must obtain access to the monitoring instance. For more information, see Get access to dashboards.

Build queries for your metrics

Monitor and view metrics for data observability purposes on your applications through the system monitoring instance of your project. The integrated user interface (UI) of the instance contains default dashboards from standard components and customized dashboards you create for your specific use case needs. Assuming your environment is up and running, you immediately see a few metrics dashboards in the Available Dashboards page.

Query metrics from the UI to visually retrieve metrics from your project and get an integrated view to monitor your applications. To filter results, search for metrics by labels using query language expressions.

System monitoring instance endpoint

The following URL is the endpoint of the monitoring instance of the platform-obs project:

https://GDC_URL/PROJECT_NAMESPACE/grafana

Replace the following:

• GDC_URL: the URL of your organization in GDC
• PROJECT_NAMESPACE: the namespace of your project

Query metrics

Complete the following steps to query metrics from your project:

Complete the following steps to retrieve metrics:

1. In the GDC console, select a project.
2. On the navigation menu, click Operations > Monitoring.
3. Click View all in Grafana.

A new page opens the URL of the monitoring instance of your project.

1. On the UI of the monitoring instance, click explore Explore from the navigation menu to open the Explore page.
2. From the drop-down menu at the Explore bar, select prometheus to retrieve metrics.

3. Enter a query to search for metrics using PromQL (Prometheus Query Language) expressions. You can do this step in either of the following ways:

   • Select a metric and a label for your query from the Metric and Label filters drop-down menus. Click add Add to add more labels in your query. Then, click Run query.
   • Enter your query directly in the Metrics text field and press Shift+Enter to run the query.

   The page displays the metrics matching your query.

Figure 1. Menu option to query metrics from the Grafana UI.

In figure 1, the prometheus option displays the interface that lets you build queries from Grafana to retrieve metrics.

For value examples of labels that you can use to query metrics, see Sample queries and labels.

Sample queries and labels

You can query collected metrics using the metric name and key-value labels. For this reason, a PromQL query has the following syntax:

metric_name{label_one="value", label_two="value"}

Labels let you differentiate the characteristics of a metric. In this way, container authors make their workloads generate metrics and add tags to filter those metrics for their needs. For example, you can have an api_http_requests_total metric to count the number of HTTP requests received. Then, you can add a request_method label on this metric, which takes a POST, GET, or PUT value. Consequently, you create three metric streams for each request type you might receive. In this case, to find the number of HTTP GET requests, you run the following query:

api_http_requests_total{request_method="GET"}

Note: See https://prometheus.io/docs/practices/naming/ for more information about metrics and labels.

The following are some of the default labels that the MonitoringTarget custom resource adds and that you can use to query metrics:

• _gdch_service: The short name of the service
• cluster: The name of the cluster
• container_name: The name of the container within a pod
• namespace_name: Your project namespace
• pod_name: The pod name prefix

The following table describes the labels that the scraping service adds automatically:

Default labels of the scraping service

Metric label	Description
job	The internal name of the scrape job used to collect the metric. Jobs created by the MonitoringTarget custom resource have a name with the following pattern: PROJECT_NAME-obs-system/MONITORINGTARGET_NAME.MONITORINGTARGET_NAMESPACE/I/J PROJECT_NAME-obs-system/MONITORINGTARGET_NAME.MONITORINGTARGET_NAMESPACE/I/J I and J are unique numbers determined internally to avoid name conflicts.
instance	The $IP:$PORT of the scrapped service. If a workload resource has multiple replicas, use this field to differentiate them.

The following code samples show the use of labels and values to query different metrics for data observability:

• View all the metric streams of the processed operations in your project:

  processed_ops_total
  

• View the processed operations collected in a cluster:

  processed_ops_total{cluster="CLUSTER_NAME"}
  

• View the CPU usage collected on a cluster:

  cpu_usage{cluster="CLUSTER_NAME"}
  

Use the metrics relabeling tool to add labels not exposed initially by the scrapped containers and rename produced metrics. You must configure the MonitoringTarget CR to add labels on the metrics it collects. Specify those labels in the metricsRelabelings field of the CR. For more information, see Label metrics.

Query metrics from the Cortex API

Cortex is the long-term storage for Prometheus metrics in GDC. The Observability platform exposes a Cortex HTTP API endpoint for querying and reading metrics, alerts, and other Prometheus time series data from your project.

Query metrics directly from the HTTP API to read and export metrics and other time series data to external tools, set up automated tasks, adapt responses, and build integrations according to your system monitoring use case. For example, insert the output into another command, export details to text file formats, or configure a Linux cron job. You can call the API from the command-line interface (CLI) or a web browser and use query language expressions as endpoint parameters to obtain the result in JSON format.

This section explains how to call the Cortex API endpoint from the CLI using the Prometheus API specification to query metrics for data observability..

Before you begin

You must obtain authorization to query metrics from the CLI. To get the permissions you need to access the Cortex API endpoint, ask your Project IAM Admin to grant you the Project Cortex Prometheus Viewer (project-cortex-prometheus-viewer) role.

Create the following role bindings using the kubectl commands to Cortex prometheus for each persona:

1. Infrastructure Operator (IO) Root-Admin - Project Cortex Prometheus Viewer:

   kubectl --kubeconfig $HOME/root-admin-kubeconfig create rolebinding 
   io-cortex-prometheus-viewer-binding -n infra-obs 
   --user=[e-mail] 
   --role=project-cortex-prometheus-viewer
   

2. Platform-Admin (PA) Root-Admin - Project Cortex Prometheus Viewer:

   kubectl --kubeconfig $HOME/root-admin-kubeconfig create rolebinding 
   pa-cortex-prometheus-viewer-binding -n platform-obs 
   --user=[e-mail] --role=project-cortex-prometheus-viewer
   

3. Application Operator (AO) Root-Admin - Project Cortex Prometheus Viewer :

   kubectl --kubeconfig $HOME/root-admin-kubeconfig create rolebinding 
   project-cortex-prometheus-viewer-binding -n PROJECT_NAME 
   --user=USER_NAME --role=project-cortex-prometheus-viewer
   

   Replace the following:

   • PROJECT_NAME: The name of your project.
   • USER_NAME: The account name of the user that requires the role binding.

After the role binding is created, you can access Prometheus API with your login username.

HTTP API endpoint

The following URL is the HTTP API endpoint for accessing metrics in the platform-obs project:

https://GDC_URL/PROJECT_NAME/cortex/prometheus/

Replace the following:

• GDC_URL: The URL of your organization in GDC.
• PROJECT_NAME: The name of your project.

Note: You can directly open the URL in a web browser to query metrics. The instructions in the following section use the curl command to call the API endpoint, but the same applies to a web browser.

Call the API endpoint

Work through the following steps to reach the Cortex API endpoint from the CLI and query metrics:

1. Ensure you meet the prerequisites.
2. Open the CLI.

3. Use the curl tool to call the Cortex endpoint and extend the URL using the standard https://prometheus.io/docs/prometheus/latest/querying/api/ to query metrics. For example:

   curl https://console.org-1.zone1.google.gdch.test/alice/cortex/prometheus/api/v1/query?query=my_metric{cluster="org-1-system"}&time=2015-07-01T20:10:51.781Z
   

You obtain the output in the CLI following the command. The API response format is JSON.

Note: Add the api/v1/query endpoint to query metrics from the URL. You can include optional parameters for your API call after a question mark (?) at the end of the URL. Separate these parameters with ampersand (&) symbols as in the example of the query from the last step.