Configure cluster notifications for third-party services

Stay organized with collections Save and categorize content based on your preferences.

This tutorial shows you how to configure third-party messaging services to receive Google Kubernetes Engine (GKE) cluster notifications.

Services such as Slack offer incoming webhooks, a simple way to post messages from apps into Slack. Cloud Functions is a lightweight Compute Engine solution to create single-purpose, stand-alone functions that respond to Google Cloud events, such as cluster notifications, without the need to manage a server or runtime environment. When GKE sends a cluster notification using Pub/Sub, a trigger responds by executing an action, such as sending a Slack notification.

There are many third-party services built with cross-application messaging functionality, such as IFTTT. You can use this tutorial as a template for connecting with these services.

In this tutorial, you use Cloud Functions and Pub/Sub to send notifications about GKE cluster events to Slack.

Objectives

  • Deploy a Slack application to receive external notifications from GKE.
  • Write a Cloud Function that sends Pub/Sub notifications to Slack.

Costs

This tutorial uses the following billable components of Google Cloud:

To generate a cost estimate based on your projected usage, use the pricing calculator. New Google Cloud users might be eligible for a free trial.

When you finish this tutorial, you can avoid continued billing by deleting the resources you created. For more information, see Clean up.

Before you begin

  1. Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.
  2. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

    Go to project selector

  3. Make sure that billing is enabled for your Cloud project. Learn how to check if billing is enabled on a project.

  4. Enable the GKE, Cloud Functions, Cloud Build, and Pub/Sub APIs.

    Enable the APIs

  5. Install and initialize the Google Cloud CLI.
  6. Update and install gcloud components:
    gcloud components update &&
    gcloud components install alpha beta
  7. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

    Go to project selector

  8. Make sure that billing is enabled for your Cloud project. Learn how to check if billing is enabled on a project.

  9. Enable the GKE, Cloud Functions, Cloud Build, and Pub/Sub APIs.

    Enable the APIs

  10. Install and initialize the Google Cloud CLI.
  11. Update and install gcloud components:
    gcloud components update &&
    gcloud components install alpha beta
  12. Enable GKE cluster notifications.

Slack notifications

To set up Slack notifications, you must create a Slack application, activate Incoming Webhooks for the application, and install the application to a Slack workspace.

Create the Slack application

  1. Join a Slack workspace, either by registering with your email or by using an invitation sent by a Workspace Admin.

  2. Sign in to Slack using your workspace name and your Slack account credentials.

  3. Create a new Slack app:

    1. In the Create an app dialog, click From scratch.
    2. Specify an App Name and choose your Slack workspace.
    3. Click Create App.
    4. Under Add features and functionality, click Incoming Webhooks.
    5. Click the Activate Incoming Webhooks toggle.
    6. In the Webhook URLs for Your Workspace section, click Add New Webhook to Workspace.
    7. On the authorization page that opens, select a channel to receive notifications.
    8. Click Allow.
    9. A webhook for your Slack application is displayed in the Webhook URLs for Your Workspace section. Save the URL for later.

Write the Cloud Function

When GKE publishes a cluster notification to a Pub/Sub topic, the event triggers Cloud Functions to send a Slack notification.

  1. Create a new directory named gke_slack and change directory into it:

    mkdir ~/gke_slack && cd $_
    
  2. Create the following files in the gke_slack directory:

    index.js

    const { IncomingWebhook } = require('@slack/webhook');
    const url = process.env.SLACK_WEBHOOK;
    
    const webhook = new IncomingWebhook(url);
    
    // Optionally filter what notification types to forward to Slack.
    // If empty, all types will be allowed.
    const allowedTypeURLs = [];
    
    // slackNotifier is the main function called by Cloud Functions
    module.exports.slackNotifier = (pubSubEvent, context) => {
      const data = decode(pubSubEvent.data);
    
      // Send message to Slack.
      if (isAllowedType(pubSubEvent.attributes)) {
        const message = createSlackMessage(data, pubSubEvent.attributes);
        webhook.send(message);
      }
    };
    
    // decode decodes a pubsub event message from base64.
    const decode = (data) => {
      return Buffer.from(data, 'base64').toString();
    }
    
    // isAllowedType can be used to filter out messages that don't match the
    // allowed type URLs. If allowedTypeURLs is empty, it allows all types.
    const isAllowedType = (attributes) => {
      if (allowedTypeURLs.length == 0) {
        return true;
      }
      for (var x in allowedTypeURLs) {
        if (attributes['type_url'] == allowedTypeURLs[x]) {
          return true;
        }
      }
      return false;
    }
    
    // createSlackMessage creates a message from a data object.
    const createSlackMessage = (data, attributes) => {
      // Write the message data and attributes.
      text = `${data}`
      for (var key in attributes) {
        if (attributes.hasOwnProperty(key)) {
          text = text + `\n\t\`${key}: ${attributes[key]}\``
        }
      }
      const message = {
        text: text,
        mrkdwn: true,
      };
      return message;
    }
    

    SLACK_WEBHOOK is a Cloud Functions environment variable specifying the webhook URL created for your Slack application. You define the environment variable when you deploy the function.

    The webhook listens for and receives messages from Cloud Functions. When GKE sends a cluster notification to Pub/Sub (the event), the function sends a message (the trigger) to the webhook URL, which in turn sends the message to the configured Slack workspace.

    You can expand the message in the createSlackMessage function to include much more, including text formatting and images. The isAllowedType function is provided to enable basic filtering of notifications by the type URL. You can specify what type URLs to allow in allowedTypeURLs. This function isn't necessary if you already filtered notifications, either in GKE or in your Pub/Sub subscription.

    package.json

    {
      "name": "gke-slack",
      "version": "0.0.1",
      "description": "Slack integration for GKE, using Cloud Functions",
      "main": "index.js",
      "dependencies": {
        "@slack/webhook": "5.0.1"
      }
    }
    

    package.json describes the following attributes of the program:

    • Name, version, and description
    • Primary runtime file(s)
    • Dependencies

    You can add more dependencies, requirements, and other information as needed.

You should now have the index.js and package.json files in the gke_slack directory.

Deploy the Cloud Function

You can deploy the Cloud Function using either the Google Cloud CLI or the Google Cloud console.

gcloud

To deploy the function, run the following command in the gke_slack directory:

gcloud functions deploy slackNotifier \
  --trigger-topic=TOPIC_NAME \
  --runtime=nodejs10 \
  --set-env-vars="SLACK_WEBHOOK=WEBHOOK_URL"

Replace the following:

The output is similar to the following:

Deploying function…
availableMemoryMb: 256
entryPoint: slackNotifier
environmentVariables:
  SLACK_WEBHOOK: https://hooks.slack.com/services/…
eventTrigger:
  eventType: google.pubsub.topic.publish
  failurePolicy: {}
  resource: projects/PROJECT_ID/topics/TOPIC_NAME
  service: pubsub.googleapis.com
labels:
  deployment-tool: cli-gcloud
name: projects/PROJECT_ID/locations/us-central1/functions/slackNotifier
runtime: nodejs10
serviceAccountEmail: PROJECT_ID@appspot.gserviceaccount.com
sourceUploadUrl: https://storage.googleapis.com/…
status: ACTIVE
timeout: 60s
updateTime: 'YYYY-MM-DDThh:mm:ssZ'
versionId: '1'

Console

  1. Go to the Cloud Functions page in the Google Cloud console.

    Go to Google Kubernetes Engine

  2. Click Create function.

  3. On the Configuration page, perform the following steps:

    1. For Function name, specify slackNotifier.
    2. For Region, specify a Compute Engine region.
    3. In the Trigger section, from the Trigger type drop-down list, select Cloud Pub/Sub.
    4. Select the Pub/Sub topic you created when enabling cluster notifications.
    5. Click Save.
    6. Expand the Runtime, build, connections and security settings section.
    7. Under Runtime environment variables, click Add Variable.
    8. For Name, specify SLACK_WEBHOOK.
    9. For Value, specify the URL of the Internal Webhook created in Create the Slack application.
    10. Click Next.
  4. On the Code page, perform the following steps:

    1. Select Node.js 10 from the Runtime drop-down list.
    2. For Entry point, specify slackNotifier.
    3. In the navigation pane, select index.js and replace the code with the sample code in Write the Cloud Function.
    4. In the navigation pane, select package.json and replace the code with the sample code in Write the Cloud Function.
    5. Click Deploy.

After you've completed deployment of the Cloud Function, you will receive a Slack notification whenever GKE sends a cluster notification.

Verify Slack notifications

You can test that cluster notifications work correctly by manually upgrading a node pool in your cluster to the same GKE version.

To start a node pool upgrade, run the following command:

gcloud container clusters upgrade CLUSTER_NAME \
    --node-pool=NODE_POOL_NAME \
    --cluster-version=VERSION

Replace the following:

  • CLUSTER_NAME: the name of your cluster.
  • NODE_POOL_NAME: the name of the node pool you want to upgrade.
  • VERSION: the GKE version you want for the node pool. If you omit this flag, GKE upgrades the node pool to the same version as the control plane.

When you run the command, you should receive a Slack notification similar to the following:

Master is upgrading to version 1.20.10-gke.301.
    cluster_location: us-central1
    cluster_name: pubsub-cluster
    payload: {"resourceType":"MASTER", "operation":"operation-1632775054313-45128f4f", "operationStartTime":"2021-09-27T20:37:34.313742491Z", "currentVersion":"1.20.9-gke.1001", "targetVersion":"1.20.10-gke.301"}
    project_id: 729788050015
    type_url: type.googleapis.com/google.container.v1beta1.UpgradeEvent

Clean up

To avoid incurring charges to your Google Cloud account for the resources used in this tutorial, either delete the project that contains the resources, or keep the project and delete the individual resources.

Delete the project

  1. In the Google Cloud console, go to the Manage resources page.

    Go to Manage resources

  2. In the project list, select the project that you want to delete, and then click Delete.
  3. In the dialog, type the project ID, and then click Shut down to delete the project.

Delete individual resources

  1. Delete the Cloud Function you deployed in this tutorial:

    gcloud functions delete slackNotifier
    

    You can also delete Cloud Functions from the Google Cloud console.

  2. Delete the Pub/Sub topic.

  3. Delete the Slack app.

What's next