Collect logs from your workflows

This page outlines the process for collecting logs from your workflows in Google Distributed Cloud (GDC) air-gapped environments to facilitate logging and data observability.

Logs record conditions and actions as you manage operations in GDC. You can scrape logs that your components produce to record events and activities. The logging platform offers a custom API to collect project-level logs generated by your workflows through logging targets.

To start collecting log data, deploy a LoggingTarget custom resource to your project namespace in the Management API server. After the log collector pulls data from your workflows, the logging platform aggregates logs from all log sources, adds indexes, and associates logs with labels according to the configuration from the LoggingTarget custom resource.

Access your collected logs using the Grafana user interface or the Log Query API, as detailed in Query and view logs.

For best practices on implementing logging for data observability, refer to the Kubernetes community guidelines:

https://github.com/kubernetes/community/blob/master/contributors/devel/sig-instrumentation/logging.md.

Before you begin

To get the permissions that you need to manage LoggingTarget custom resources, ask your Organization IAM Admin or Project IAM Admin to grant you one of the associated LoggingTarget roles.

Depending on the level of access and permissions you need, you might obtain creator, editor, or viewer roles for this resource in an organization or a project. For more information, see Prepare IAM permissions.

Configure the `LoggingTarget` custom resource

The LoggingTarget custom resource instructs the logging pipeline to collect logs within your GDC project. You can specify the targets for which you are collecting logs, a log parser, access levels, and any additional settings.

This resource defines the following configurations:

Targets: The criteria for selecting the pods and containers from which you want to collect logs. You can specify cluster names, pod name prefixes, and container name prefixes.
Log analyzer: A predefined parser to interpret log entries, map log outputs to labels, and extract relevant fields.
Service identification: The service name applied as a label to the log entries for easier identification and filtering.
Authorization: The access level for log entries, controlling who can view the collected logs.

Note: By default, all operational log entries are saved for the tenant ID of the project namespace. To overwrite this behavior, provide a logAccessLevel value in the LoggingTarget custom resource.

By configuring these parameters within the LoggingTarget custom resource, you can precisely control the collection and organization of logs.

Follow these steps to collect operational logs:

Determine the GDC project from which you want to collect logs.
In your LoggingTarget configuration, specify the pods for collecting your operational logs, the project namespace, and any additional settings.

The following YAML file shows an example of a LoggingTarget configuration in my-project-namespace, where the pod name prefix to collect logs from is my-pod-prefix, the access level for log entries is granted to Application Operators (ao), the service name is my-service-name, and the log format is JSON:
```
# Configures a log scraping job
apiVersion: logging.gdc.goog/v1
kind: LoggingTarget
metadata:
  # Choose a namespace that matches the namespace of the workload pods.
  namespace: my-project-namespace
  name: my-logging-target
spec:
  selector:
    matchPodNames:
      - my-pod-prefix
  parser: json
  logAccessLevel: ao
  serviceName: my-service-name
```
Refer to the complete LoggingTarget specification and the API reference documentation for additional fields and options.
Apply the LoggingTarget configuration to the Management API server within the same namespace as your target pods:
```
kubectl --kubeconfig KUBECONFIG_PATH apply -f LOGGING_TARGET_NAME.yaml
```
Replace the following:
- KUBECONFIG_PATH: the path to the kubeconfig file for the Management API server.
- LOGGING_TARGET_NAME: the name of the LoggingTarget definition file, such as my-logging-target.
The logging pipeline begins collecting logs from the additional components of your project.

You can query your collected logs using the Grafana user interface or the Log Query API. For more information, see Query and view logs.

If you use Grafana to query your logs, you can use the built-in color coding feature for different log levels. For more information on setting log-level values, see https://grafana.com/docs/grafana/latest/explore/logs-integration/.

Complete `LoggingTarget` specification

The following YAML file shows an example for the complete specification of the LoggingTarget custom resource. For more information and a complete description of fields, see the API reference documentation.

# Configures a log scraping job
apiVersion: logging.gdc.goog/v1
kind: LoggingTarget
metadata:
  # Choose a namespace that matches the namespace of the workload pods.
  namespace: PROJECT_NAMESPACE
  name: LOGGING_TARGET_NAME
spec:
  # Choose a matching pattern that identifies the pods for this job.
  # Optional.
  # Relationship between different selectors: 'AND'
  selector:
    # The clusters to collect logs from.
    # The default configuration is to collect logs from all clusters.
    # The relationship between different clusters is an 'OR' relationship.
    # Optional
    matchClusters:
    - my-cluster
    - another-cluster

    # The pod name prefixes to collect logs from.
    # The logging platform scrapes all pods with names
    # that start with the specified prefixes.
    # The values must contain '[a-z0-9-]' characters only.
    # The relationship between different list elements is an 'OR' relationship.
    # Optional.
    matchPodNames:
      - my-pod-prefix
      - another-pod-prefix

    # The container name prefixes to collect logs from.
    # The logging platform scrapes all containers with names
    # that start with the specified prefixes.
    # The values must contain '[a-z0-9-]' characters only.
    # The relationship between different list elements is an 'OR' relationship.
    # Optional.
    matchContainerNames:
      - my-container-prefix
      - another-container-prefix

  # Choose the predefined parser for log entries.
  # Use parsers to map the log output to labels and extract fields.
  # Specify the log format.
  # Optional.
  # Options: klog_text, klog_json, klogr, gdch_json, json
  parser: klog_text

  # Specify an access level for log entries.
  # The default value is 'ao'.
  # Optional.
  # Options: ao, pa, io
  logAccessLevel: ao

  # Specify a service name to be applied as a label.
  # For user workloads consider this field as a workload name.
  # Required.
  serviceName: my-service-name

  # The additional static fields to apply to log entries.
  # The field is a key-value pair, where the field name is the key and
  # the field value is the value.
  # Optional.
  additionalFields:
    app: workload2
    key: value

Replace the following:

PROJECT_NAMESPACE: your project namespace.
LOGGING_TARGET_NAME: the name of the LoggingTarget definition file.