Configure notification channels for AO alerts

The Observability platform lets you configure notification channels for the alerts you set up. The Observability system sends an alert to these customized channels whenever an event causes an alert. This way, you can notify the recipients you need according to your preferred methods.

Google Distributed Cloud Hosted (GDCH) does not predefine notification channels intended for sending alerts. You must define at least one channel to receive alerts. If you don't define a notification channel configuration, the GDCH Observability system takes the configuration from the platform resources as a default. Consequently, if you don't configure notification channels but an administrator did it for the organization, then the system sends alerts to those channels.

Work through the following steps to configure notification channels for the alerts you set up for system monitoring:

  1. Define the Observability alerting configuration following the specification in https://prometheus.io/docs/alerting/latest/configuration/.
  2. Create a ConfigMap object and deploy it into your project namespace. The object must contain the configuration rules in the alertmanager.yml file.

    apiVersion: v1
    kind: ConfigMap
    metadata:
      # The name must match the ConfigMap name specified in the ObservabilityPipeline custom resource.
      name: CUSTOMIZED_CONFIGMAP_NAME
      # The namespace must match the project namespace
      namespace: PROJECT_NAMESPACE
    data:
      # The file name must be alertmanager.yml
      alertmanager.yml: |
        # Add the customized configuration in the following lines of this file.
    

    Replace the following:

    • CUSTOMIZED_CONFIGMAP_NAME: the name of your configuration file. You use this name in the ObservabilityPipeline custom resource during the following step.
    • PROJECT_NAMESPACE: the namespace of your project
  3. Configure the ObservabilityPipeline custom resource in a command-line editor to import the configuration to the alertmanagerConfig field.

     # Configure observability pipeline
    apiVersion: observability.gdc.goog/v1
    kind: ObservabilityPipeline
    metadata:
      # The namespace must match the project namespace
      namespace: PROJECT_NAMESPACE
      name: observability-config
    spec:
      # Configure alerts
      alerting:
        # The alerting configuration must be in the key "alertmanager.yml" file of the configMap object
        alertmanagerConfig: CUSTOMIZED_CONFIGMAP_NAME
      ...
    

    Replace the following:

  4. Save the changes in the ObservabilityPipeline custom resource and exit your command-line editor.