Query and view logs

After the log collector pulls data from various project components, the Observability platform aggregates logs from all log sources, adds indexes, and associates logs with labels according to the configuration for data observability. You can then query and view logs on the logs panel of the system monitoring instance.

Query and view logs on the logs panel

You can query logs through the user interface (UI) of the system monitoring instance of your project. The system monitoring instance includes project-level measures to monitor infrastructure components, such as CPU utilization, storage consumption, network monitoring, server monitoring, logs, alerts, and metrics from the GDC components.

You can view logs on the logs panel of the system monitoring instance of your project. Then, you can make queries to view specific logs for data observability of your use case needs.

Before you begin

Before querying and viewing logs on the logs panel, you must obtain access to the system monitoring instance. For more information, see Get access to dashboards.

To sign in and view logs in Grafana, ask your Project IAM Admin to grant you the Project Grafana Viewer (project-grafana-viewer) role. This role-based access control process lets you access data visualizations safely.

Build queries for your logs

Querying logs from the UI of the system monitoring instance lets you choose the data source to retrieve operational logs from your project and get an integrated view of your queries. To filter results, search for logs by labels using query language expressions. For example, you can set the cluster and namespace labels in your queries to the values that identify your cluster and project namespace, respectively.

System monitoring instance endpoint

For Application Operator (AO)

The following URL is the endpoint of the monitoring instance of your project:

https://GDC_URL/PROJECT_NAMESPACE/grafana

Replace the following:

GDC_URL: The URL of your organization in GDC.
PROJECT_NAMESPACE: The namespace of your project.

For Platform Admin (PA)

The following URL is the endpoint of the monitoring instance of your project:

https://GDC_URL/platform-obs/grafana

Replace GDC_URL with the URL of your organization in GDC.

Query operational logs

In the GDC console, select a project.
On the navigation menu, click Operations > Logging.
Click View all in Grafana Loki.

A new page opens the URL of the monitoring instance of your project.
On the UI of the monitoring instance, click explore Explore from the navigation menu to open the Explore page.
From the drop-down menu at the Explore bar, select Operational Logs as the data source to retrieve operational logs.
Enter a query to search for logs by labels using LogQL (Log Query Language) expressions. You can do this step in either of the following ways:
- Use the interactive query-builder UI. Then, click Run query.
- Enter your query using LogQL expressions directly in the text field and press Shift+Enter to run the query.
The page displays the logs matching your query.

After querying logs, you can export them. Click Export to download logs in plain text or CSV formats.

The Operational Logs option is selected on the Explore page of the UI to obtain operational logs.

Figure 1. Menu option to query logs from the UI of the system monitoring instance.

In figure 1, the Operational Logs option displays the interface that lets you build log queries from the UI to retrieve operational logs.

To query audit logs instead of operational logs, select Audit Logs as the data source from the drop-down menu at the Explore bar in the previous step.

The Audit Logs option is selected on the Explore page of the UI to obtain audit logs.

Figure 2. Menu option to query audit logs from the UI of the monitoring instance.

In figure 2, the Audit Logs option displays the interface that lets you build queries from the UI to retrieve audit logs.

After querying logs, you can export them. Click the Export button to download logs in plain text or CSV formats.

Complete the following steps to retrieve operational logs:

In the GDC console, select a project.
On the navigation menu, click Operations > Logging.
Click View all in Grafana Loki.

A new page opens the URL of the monitoring instance of your project.
On the UI of the monitoring instance, click explore Explore from the navigation menu to open the Explore page.
From the drop-down menu at the Explore bar, select Operational Logs as the data source to retrieve operational logs.
Enter a query to search for logs by labels using LogQL (Log Query Language) expressions. You can do this step in either of the following ways:
- Use the interactive query-builder UI. Then, click Run query.
- Enter your query using LogQL expressions directly in the text field and press Shift+Enter to run the query.
The page displays the logs matching your query.

After querying logs, you can export them. Click Export to download logs in plain text or CSV formats.

Query audit logs

Complete the following steps to retrieve audit logs:

In the GDC console, select a project.
On the navigation menu, click Operations > Audit Logging.
Click View all in Grafana Loki.

A new page opens the URL of the monitoring instance of your project.
On the UI of the monitoring instance, click explore Explore from the navigation menu to open the Explore page.
From the drop-down menu at the Explore bar, select Audit Logs as the data source to retrieve audit logs.
Enter a query to search for logs by labels using LogQL (Log Query Language) expressions. You can do this step in either of the following ways:
- Use the interactive query-builder UI. Then, click Run query.
- Enter your query using LogQL expressions directly in the text field and press Shift+Enter to run the query.
The page displays the logs matching your query.

After querying logs, you can export them. Click Export to download logs in plain text or CSV formats.

Select a time range for your logs

To query logs in a time range from the UI of the monitoring instance, follow these steps:

Click the Time Picker drop-down menu.
From the menu, perform one of the following actions:
- Select relative time range options, for example, the last 30 minutes.
- Set custom absolute time ranges by choosing specific dates and times from the calendar and clicking Apply time range.
Optionally, click Change time settings to change the Timezone and Fiscal year settings from the time range controls.

Time settings are saved on a per-dashboard basis. For more information about queries over a range of time, see https://grafana.com/docs/loki/latest/reference/api/#query-loki-over-a-range-of-time.

Sample queries and labels

The following are some of the default labels that you can use to query logs:

cluster: The name of the cluster
resources: The list of resources
namespace: Your project namespace
node: The node name prefix
pod: The pod name prefix
container: The container name prefix

The following table describes each of the default labels that you can use to query logs from the Observability platform:

Log labels
Log type	Label	Value
Kubernetes container logs	`cluster`	`CLUSTER_NAME`
	`resources`	`k8s_container` or `k8s_audit`
	`namespace`	`NAMESPACE_NAME`
	`node`	`NODE_NAME`
	`pod`	`POD_NAME`
	`container`	`CONTAINER_NAME`
	Other Kubernetes labels
Node journald logs	`cluster`	`CLUSTER_NAME`
	`resources`	`k8s_node`
	`node`	`NODE_NAME`
	`identifier`	`SYSLOG_IDENTIFIER`
	`machine`	`MACHINE_ID`

Queries use the LogQL query language https://grafana.com/docs/loki/latest/logql/. The following code samples show the use of labels and values to query different logs:

Select all logs from a project:
```
{cluster="user-vm-1-cluster", namespace="PROJECT_NAMESPACE"}
```
Replace PROJECT_NAMESPACE with the namespace of your project.
Select project logs from a service:
```
{cluster="user-vm-1-cluster", namespace="PROJECT_NAMESPACE", container="my-service"}
```
Replace PROJECT_NAMESPACE with the namespace of your project.

Select Kubernetes API server logs:

{cluster="admin", namespace="kube-system", resources="k8s_container", container="kube-apiserver"}

Select cluster bootstrap logs:
```
{cluster="admin", namespace="PROJECT_NAMESPACE"}
```
Replace PROJECT_NAMESPACE with the name of your project namespace. For example, cluster-user-test.

Select node journald logs:

{cluster="admin", resources="k8s_node"}

Select cluster audit logs:

{cluster="admin", resources="k8s_audit"}