Configure a Simple Mail Transfer Protocol (SMTP) notification channel for Prometheus alerts

This page describes how to set up and configure a Simple Mail Transfer Protocol (SMTP) notification channel for Prometheus alerts. If you choose not to set up this notification channel, you can still view alerts at infra-obs or platform-obs.

Work through the following steps to configure Slack or email notification channels:

1. To create a custom alert notification channel, host the SMTP server in your network and have the SMTP configuration details (IP, username, password, receiver email) ready to create the alertmanager config. You can validate the correctness of the SMTP host IP by pinging it from bm03.

   Note: Perform the following steps for both of the namespaces (infra-obs and platform-obs) and any additional namespaces created by the user.

2. Create a file cm.yml with the following content:

   The smtp_smarthost must have the IP address of the SMTP server hosted by the customer.

     apiVersion: v1
     kind: ConfigMap
     metadata:
       # The name must match the ConfigMap name specified in the ObservabilityPipeline custom resource.
       name: appl-alert-notification
       # The namespace must match the project namespace
       namespace: PROJECT_NAMESPACE
     data:
       # The file name must be alertmanager.yml
       alertmanager.yml: |
         global:
           smtp_smarthost: SMTP HOST IP ADDRESS
           smtp_from: FROM_EMAIL_ADDRESS
           # Give empty string for unsecured smtp server in username and password and false in smtp_require_tls.
           smtp_auth_username: USERNAME
           smtp_auth_password: PASSWORD
           smtp_require_tls: true
         route:
           group_by: ['alertname', 'cluster', 'service']
           group_wait: 30s
           group_interval: 5m
           repeat_interval: 3h
           receiver: team-X-mails
         receivers:
           - name: 'team-X-mails'
             email_configs:
               - to: **RECIEVER_EMAIL_ID**
   

3. Update the required fields with the correct content.

4. Run kubectl apply -f cm.yaml.

5. Create an ObservabilityPipeline custom resource with the following:

       # Configure observability pipeline
     apiVersion: observability.gdc.goog/v1
     kind: ObservabilityPipeline
     metadata:
       # Choose namespace that matches the project's namespace
       namespace: PROJECT_NAMESPACE
       name: observability-config
     spec:
       # Configure Alertmanager
       alerting:
         # Permission: PA and AO
         # The alerts configuration must be in the key "alertmanager.yml" of the configMap
         alertmanagerConfig: CUSTOMIZED_CONFIGMAP_NAME
       ...
   

   Replace the following:

   • PROJECT_NAMESPACE: The namespace of your project.

   • CUSTOMIZED_CONFIGMAP_NAME: The name you used for your configuration file in the ConfigMap object must match exactly with the name of the ConfigMap object. For example in step 2, if you used appl-alert-notification as the ConfigMap name, then use the same name for your configuration file in the ObservabilityPipeline object.

     Note: You must create a CR for both the namespaces, infra-obs as well as platform-obs. There must be a separate configmap and ObservabilityPipeline for each namespace.

6. Update the namespace to the value used in the previous step.

7. Run kubectl apply -f ob.yaml.

8. All the alerts visible on the Grafana dashboard of the GDC trigger an email if they are in a firing state. Launch Grafana for the namespace which you used in the previous steps and check for alerts in a firing state. The alerts come to your email.

Note: Refer to Configure notification channels for alerts for more information on configuring notification channels for the alerts you set up.