Query and view metrics

After collecting metrics from your deployed workloads in Google Distributed Cloud (GDC) air-gapped appliance, you can start analyzing them. To analyze metrics, you can visualize and filter them on informative Grafana dashboards or access them directly from Cortex using the curl tool for flexible scripting and automation.

You can access your metrics in one of the following two methods:

Grafana dashboards: Explore trends and identify anomalies with intuitive visualizations of key metrics like CPU utilization, storage consumption, and network activity. Grafana provides a user-friendly interface for filtering and analyzing your workload data in dashboards.
Cortex endpoint: For more advanced use cases, query your project's Cortex instance directly using the curl tool on a command line. Cortex stores your project's Prometheus metrics and provides an HTTP endpoint for programmatic access. This access lets you export data, automate tasks, and build custom integrations.

Before you begin

To get the permissions that you need to query and visualize metrics on Grafana dashboards, ask your Organization IAM Admin or Project IAM Admin to grant you one of the predefined Organization Grafana Viewer or Project Grafana Viewer roles. Depending on the level of access and permissions you need, you might obtain Grafana roles in an organization or a project.

Alternatively, to get the permissions that you need to query metrics from the Cortex endpoint, ask your Project IAM Admin to grant you the Project Cortex Prometheus Viewer role in your project namespace.

For more information about these roles, see Prepare IAM permissions.

Obtain and filter your metrics

Select one of the following methods to build queries, visualize trends, and filter metrics from your project workloads:

Grafana dashboards

This section describes how to access your metrics using Grafana dashboards.

Identify your Grafana endpoint

The following URL is the endpoint of the Grafana instance of your project:

  https://GDC_URL/PROJECT_NAMESPACE/grafana

Replace the following:

GDC_URL: the URL of your organization in GDC.
PROJECT_NAMESPACE: your project namespace.

For example, the Grafana endpoint for the platform-obs project in the org-1 organization is https://org-1/platform-obs/grafana.

View metrics in the Grafana user interface

Retrieve metrics in the Grafana user interface:

In the GDC console, select your project.
In the navigation menu, select Operations > Monitoring.
Click View all in Grafana.

A new page opens your Grafana endpoint and displays the user interface.
On the user interface,click explore Explore from the navigation menu to open the Explore page.
From the menu at the Explore bar, select a data source to retrieve metrics, according to your universe type:
- Single-zone universes: Select prometheus to display metrics from the single zone of your universe.
Enter a query to search for metrics using PromQL (Prometheus Query Language) expressions. You can do this step in either of the following ways:
- Select a metric and a label for your query from the Metric and Label filters menus. Click add Add to add more labels to your query. Then, click Run query.
- Enter your query directly in the Metrics text field and press Shift+Enter to run the query.
The page displays the metrics matching your query.

Figure 1. Menu option to query metrics from the Grafana user interface.

In figure 1, the prometheus option displays the interface that lets you build queries from Grafana to retrieve metrics.

For value examples of labels that you can use to query metrics, see Sample queries and labels.

Cortex endpoint

This section describes how to access your metrics using Cortex.

Identify your Cortex endpoint

The following URL is the endpoint of the Cortex instance of your project:

  https://GDC_URL/PROJECT_NAMESPACE/cortex/prometheus/

Replace the following:

GDC_URL: the URL of your organization in GDC.
PROJECT_NAMESPACE: your project namespace.

For example, the Cortex endpoint for the platform-obs project in the org-1 organization is https://org-1/platform-obs/cortex/prometheus/.

Note: You can directly open the URL in a web browser to query metrics. The instructions in the following sections use the curl command to call the endpoint, but the same applies to a web browser.

Authenticate the `curl` request

Download and install the gdcloud CLI.

Set the gdcloud core/organization_console_url property:

gdcloud config set core/organization_console_url
https://GDC_URL

Sign in with the configured identity provider:
```
gdcloud auth login
```
Use your username and password to authenticate and sign in.

When the login is successful, you can use the authorization header in your cURL request through the gdcloud auth print-identity-token command. For more information, see gdcloud auth.

Call the Cortex endpoint

Complete the following steps to reach the Cortex endpoint using the curl tool:

Authenticate the curl request.
Use curl to call the Cortex endpoint and extend the URL using the standard Prometheus HTTP API (https://prometheus.io/docs/prometheus/latest/querying/api/) to query metrics.

The following is an example of a curl request:
```
curl https://GDC_URL/PROJECT_NAME/cortex/prometheus/api/v1/query?query=my_metric{cluster="my-cluster"}&time=2015-07-01T20:10:51.781Z \
-H "Authorization: Bearer $(gdcloud auth print-identity-token \
--audiences=https://GDC_URL)"
```
You obtain the output following the command. The API response is in JSON format.

Note: Add the api/v1/query endpoint to query metrics from the URL. You can include optional parameters for your API call after a question mark (?) at the end of the URL. Separate these parameters with ampersand (&) symbols as in the example of the curl request fromnthe last step.

Sample queries and labels

You can query metrics using the metric name and key-value pairs for labels. A PromQL query has the following syntax:

metric_name{label_one="value", label_two="value"}

Labels let you differentiate the characteristics of a metric. In this way, container authors make their workloads generate metrics and add tags to filter those metrics.

For example, you can have an api_http_requests_total metric to count the number of HTTP requests received. Then, you can add a request_method label on this metric, which takes a POST, GET, or PUT value. Consequently, you create three metric streams for each request type you might receive. In this case, to find the number of HTTP GET requests, you run the following query:

api_http_requests_total{request_method="GET"}

See https://prometheus.io/docs/practices/naming/ for more information about metrics and labels.

The following are some of the default labels that the MonitoringTarget custom resource adds. You can use these default labels to query metrics:

_gdch_service: the short name of the service.
cluster: the name of the cluster.
container_name: the name of the container within a pod.
namespace_name: your project namespace.
pod_name: the pod name prefix.

The following table describes the labels that Prometheus adds automatically:

Default labels
Metric label	Description
`job`	The internal name of the scrape job used to collect the metric. Jobs created by the `MonitoringTarget` custom resource have a name with the following pattern: `obs-system/OBS_SHADOW_PROJECT_NAME/MONITORINGTARGET_NAME.MONITORINGTARGET_NAMESPACE/I/J` `I` and `J` are unique numbers determined internally to avoid name conflicts.
`instance`	The `$IP:$PORT` of the scrapped service. If a workload resource has multiple replicas, use this field to differentiate them.

The following code samples show the use of key-value pairs for labels to query different metrics:

View all the metric streams of the processed operations in your project:
```
processed_ops_total
```
View the processed operations collected in a Kubernetes cluster:
```
processed_ops_total{cluster="CLUSTER_NAME"}
```
View the CPU usage collected in a Kubernetes cluster:
```
cpu_usage{cluster="CLUSTER_NAME"}
```

Use the metrics relabeling tool to add labels not exposed initially by the scrapped containers and rename produced metrics. You must configure the MonitoringTarget custom resource to add labels on the metrics it collects. Specify those labels in the metricsRelabelings field of the custom resource. For more information, see Label metrics.

Query metrics from the Cortex API

Cortex is the long-term storage for Prometheus metrics in GDC. The Observability platform exposes a Cortex HTTP API endpoint for querying and reading metrics, alerts, and other Prometheus time series data from your project.

Query metrics directly from the HTTP API to read and export metrics and other time series data to external tools, set up automated tasks, adapt responses, and build integrations according to your system monitoring use case. For example, insert the output into another command, export details to text file formats, or configure a Linux cron job. You can call the API from the command-line interface (CLI) or a web browser and use query language expressions as endpoint parameters to obtain the result in JSON format.

This section explains how to call the Cortex API endpoint from the CLI using the Prometheus API specification to query metrics for data observability..

Before you begin

You must obtain authorization to query metrics from the CLI. To get the permissions you need to access the Cortex API endpoint, ask your Project IAM Admin to grant you the Project Cortex Prometheus Viewer (project-cortex-prometheus-viewer) role.

Create the following role bindings using the kubectl commands to Cortex prometheus for each persona:

Infrastructure Operator (IO) Root-Admin - Project Cortex Prometheus Viewer:

kubectl --kubeconfig $HOME/root-admin-kubeconfig create rolebinding 
io-cortex-prometheus-viewer-binding -n infra-obs 
--user=fop-infrastructure-operator@example.com 
--role=project-cortex-prometheus-viewer

Platform-Admin (PA) Root-Admin - Project Cortex Prometheus Viewer:

kubectl --kubeconfig $HOME/root-admin-kubeconfig create rolebinding 
pa-cortex-prometheus-viewer-binding -n platform-obs 
--user=fop-platform-admin@example.com --role=project-cortex-prometheus-viewer

Application Operator (AO) Root-Admin - Project Cortex Prometheus Viewer :
```
kubectl --kubeconfig $HOME/root-admin-kubeconfig create rolebinding 
project-cortex-prometheus-viewer-binding -n PROJECT_NAME 
--user=USER_NAME --role=project-cortex-prometheus-viewer
```
Replace the following:
- PROJECT_NAME: The name of your project.
- USER_NAME: The account name of the user that requires the role binding.

After the role binding is created, you can access Prometheus API with your login username.

HTTP API endpoint

The following URL is the HTTP API endpoint for accessing metrics in the platform-obs project:

https://GDC_URL/PROJECT_NAME/cortex/prometheus/

Replace the following:

GDC_URL: The URL of your organization in GDC.
PROJECT_NAME: The name of your project.

Call the API endpoint

Work through the following steps to reach the Cortex API endpoint from the CLI and query metrics:

Ensure you meet the prerequisites.
Open the CLI.
Use the curl tool to call the Cortex endpoint and extend the URL using the standard https://prometheus.io/docs/prometheus/latest/querying/api/ to query metrics. For example:
```
curl https://console.org-1.zone1.google.gdch.test/alice/cortex/prometheus/api/v1/query?query=my_metric{cluster="org-1-system"}&time=2015-07-01T20:10:51.781Z
```

You obtain the output in the CLI following the command. The API response format is JSON.

Query and view metrics Stay organized with collections Save and categorize content based on your preferences.

Before you begin

Obtain and filter your metrics

Grafana dashboards

Identify your Grafana endpoint

View metrics in the Grafana user interface

Cortex endpoint

Identify your Cortex endpoint

Authenticate the curl request

Call the Cortex endpoint

Sample queries and labels

Query metrics from the Cortex API

Before you begin

HTTP API endpoint

Call the API endpoint

Query and view metrics

Authenticate the `curl` request