Configure observability components

This document contains instructions for the Application Operator (AO) to configure values such as storage size and retention time with the ObservabilityPipeline CR. The instructions override the default values for the particular AO project.

The Observability pipeline of Google Distributed Cloud (GDC) air-gapped appliance manages the deployment and configuration of Observability stack components such as Prometheus, Cortex, Loki, and Fluent Bit.

The API to the Observability pipeline is the ObservabilityPipeline custom resource (CR) definition. The CR describes how to configure Observability for your deployment. It includes the properties of volumes to store your metrics and logs, alert rules for Alertmanager, Cortex configurations for metrics, and Grafana configurations for dashboards.

The ObservabilityPipeline CR references multiple Kubernetes ConfigMap objects for further configuration. For example, you can configure Alertmanager rules in a separate object and reference it in the ObservabilityPipeline CR definition by name. For more information on this example and configuring Alertmanager rules, see Alerting.

Deploy an ObservabilityPipeline CR named observability-config into your project namespace. To view the current configuration for your Observability deployment, run the following command:

kubectl get observabilitypipeline -n PROJECT_NAMESPACE observability-config -o yaml

Replace PROJECT_NAMESPACE with the name of your project namespace.

Configure storage size

Work through the following steps to configure the storage size for alerts, dashboards, and metrics in the project:

Open your ObservabilityPipeline CR in a command-line editor:
```
kubectl --kubeconfig=KUBECONFIG_FILE -n PROJECT_NAMESPACE edit observabilitypipeline/observability-config
```
Replace the following:
- KUBECONFIG_FILE: The path of the kubeconfig file for the admin cluster.
- PROJECT_NAMESPACE: The name of your project namespace.

In the ObservabilityPipeline custom resource, update localStorageSize and storageSize in the spec field for alerts, metrics, and dashboards:

  # Configure observability pipeline
apiVersion: observability.gdc.goog/v1alpha1
kind: ObservabilityPipeline
metadata:
  # Don't change the name.
  namespace: PROJECT_NAMESPACE
  name: observability-config
spec:
  alerting:
    localStorageSize: 1Gi # Configure the new storage size for alerts data in the project.
    ...
  monitoring:
    localStorageSize: 20Gi # Configure the new storage size for metrics data in the project.
    grafana:
      storageSize: 1Gi # Configure the new storage size for dashboards in the project.
    ...

Replace PROJECT_NAMESPACE with the name of your project namespace.

Save the change and exit your command-line editor.

Configure retention time

Work through the following steps to configure the retention time for logs and metrics in the project:

Open your ObservabilityPipeline CR in a command-line editor:
```
kubectl --kubeconfig=KUBECONFIG_FILE -n PROJECT_NAMESPACE edit observabilitypipeline/observability-config
```
Replace the following:
- KUBECONFIG_FILE: The path of the kubeconfig file for the admin cluster.
- PROJECT_NAMESPACE: The name of your project namespace.

In the ObservabilityPipeline custom resource, update retentionTime in the spec field for logs and metrics:

  # Configure observability pipeline
apiVersion: observability.gdc.goog/v1alpha1
kind: ObservabilityPipeline
metadata:
  # Don't change the name.
  namespace: PROJECT_NAMESPACE 
  name: observability-config
spec:
  monitoring:
    retentionTime: 1000h # Configure the new retention time in hours for metrics in the project.
    ...

Replace PROJECT_NAMESPACE with the name of your project namespace.

Save the change and exit your command-line editor.

Sample configuration

The following YAML file shows a sample configuration in the ObservabilityPipeline custom resource.

  # Configure observability pipeline
apiVersion: observability.gdc.goog/v1alpha1
kind: ObservabilityPipeline
metadata:
  # Don't change the name.
  namespace: PROJECT_NAMESPACE
  name: observability-config
spec:
  alerting:
    localStorageSize: 1Gi
    ...
    grafana:
      storageSize: 1Gi
    ...

Replace PROJECT_NAMESPACE with the name of your project namespace.