Enabling real-time email and chat notifications

Overview

Security Command Center provides real-time notifications of findings in the Google Cloud console. This guide describes how to use Google Cloud services and third-party APIs to extend that functionality and receive near real-time notifications in email and chat apps. When you complete the guide, you get alerts to new findings in configured third-party services without logging into the Google Cloud console, allowing for faster triaging of vulnerabilities and threats. Learn more about the different types of vulnerabilities and threats in Security Command Center.

Topology

In this guide, you create the configuration that is illustrated in the following diagram.

Near real-time notifications workflow (click to enlarge)
Near real-time notifications workflow for Security Command Center (click to enlarge)

Objectives

In this guide, you do the following:

  1. Set up a Pub/Sub topic.
  2. Set up Slack, WebEx Teams, and SendGrid Email.
  3. Write code in Cloud Run functions.
  4. Configure Pub/Sub and Cloud Run functions to send notifications to Slack, WebEx Teams, or SendGrid Email whenever a new high or critical severity finding is written to Security Command Center.
  5. Troubleshoot notification problems.

Costs

This tutorial uses billable components of Google Cloud, including:

  • Pub/Sub
  • Cloud Run functions
  • Cloud Build

Use the pricing calculator to generate a cost estimate based on your projected usage.

Before you begin

To complete this guide, you must have the following Identity and Access Management (IAM) roles:

  • Organization Admin (roles/resourcemanager.organizationAdmin)
  • Security Center Admin (roles/securitycenter.admin)
  • Security Admin (roles/iam.securityAdmin)
  • A role with the serviceusage.services.use permission, such as Owner (roles/owner), Editor (roles/editor), or a custom role
  • Create Service Accounts (roles/iam.serviceAccountCreator)
  • Pub/Sub Editor (roles/pubsub.editor)
  • Billing Account Administrator (roles/billing.admin)

The IAM roles for Security Command Center can be granted at the organization, folder, or project level. Your ability to view, edit, create, or update findings, assets, and security sources depends on the level for which you are granted access. To learn more about Security Command Center roles, see Access control.

Setting up a project

Complete the following steps to create or select a project.

  1. Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.
  2. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

    Go to project selector

  3. Make sure that billing is enabled for your Google Cloud project.

  4. Enable the Cloud Build API.

    Enable the API

  5. Install the Google Cloud CLI.
  6. To initialize the gcloud CLI, run the following command:

    gcloud init
  7. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

    Go to project selector

  8. Make sure that billing is enabled for your Google Cloud project.

  9. Enable the Cloud Build API.

    Enable the API

  10. Install the Google Cloud CLI.
  11. To initialize the gcloud CLI, run the following command:

    gcloud init

Estimated time: it takes approximately one hour to set up and test notifications for a single service.

Setting up a Pub/Sub topic

Pub/Sub is a real-time messaging service that enables messages to be sent and received between independent applications. Learn more about Pub/Sub.

In this section, you configure Security Command Center to publish findings to a Pub/Sub topic.

To set up and subscribe to a Pub/Sub topic:

  1. Specify your Google Cloud project in an environment variable.

    export PROJECT_ID=PROJECT_ID
    

    Replace PROJECT_ID with your project ID.

  2. Specify your Google Cloud organization in an environment variable.

    export ORG_ID=ORG_ID
    

    Replace ORG_ID with your organization ID.

  3. Set the project ID for gcloud commands.

    gcloud config set project PROJECT_ID
    
  4. Create the Pub/Sub topic where notifications are published.

    gcloud pubsub topics create scc-critical-and-high-severity-findings-topic
    
  5. Specify the topic in an environment variable.

    export TOPIC=projects/$PROJECT_ID/topics/scc-critical-and-high-severity-findings-topic
    
  6. Create the subscription that notifies Cloud Run functions to send an email or chat message when messages are published to the topic.

    gcloud pubsub subscriptions create scc-critical-and-high-severity-findings-sub \
      --topic scc-critical-and-high-severity-findings-topic
    
  7. Configure Security Command Center to publish notifications to the topic. Any filter compatible with the ListFindings API can be used.

    The following filter publishes notifications for active high and critical severity findings. Learn more about filtering findings.

     gcloud scc notifications create scc-critical-and-high-severity-findings-notify \
       --pubsub-topic $TOPIC \
       --organization $ORG_ID \
       --filter "(severity=\"HIGH\" OR severity=\"CRITICAL\") AND state=\"ACTIVE\""
    

Next, you create or configure your email or chat app to receive notifications from Pub/Sub.

Setting up a messaging app

This section describes how to use Pub/Sub and Cloud Run functions or Cloud Run functions (1st gen) to enable near real-time notifications for SendGrid Email API, Slack, and WebEx Teams.

SendGrid Email

To enable email notifications, you do the following:

  • Create a SendGrid Email API account and obtain an API key.
  • Create and deploy a Cloud Run function that sends emails when notifications are received from Pub/Sub.

Create SendGrid Email API account

In this section, you create a SendGrid Email API account and obtain an API key. If you already have SendGrid enabled, skip to Obtain a Sendgrid Email API Key and ensure your existing API key has adequate permissions.

  1. Go to the Google Cloud console.
    Go to the Google Cloud console
  2. In the search box at the top of the page, search for SendGrid Email API. SendGrid Search
  3. On the next page, select the plan that fits your needs.

    • The free plan allows up to 12,000 emails per month and is sufficient for this guide, but large organizations might require more. If you anticipate a larger volume of email notifications, consider adding additional filters to notifications to exclude noisy findings.
    • You might be asked to select a project to associate with SendGrid. Proceed by selecting a project. You might need adequate permissions to manage purchases for the project's associated billing account.
  4. Review the terms and if you're comfortable, click Subscribe.

  5. Activate the SendGrid service by clicking Register with SendGrid.

  6. On the registration screen, enter a username, password, and email address. Accept the terms of service and click Continue.

  7. At the confirmation dialog, click Return to Google.

Obtain a SendGrid Email API key

  1. Click Manage API keys on SendGrid website. A new tab opens for the SendGrid website. Manage API keys

  2. Complete the form or sign in, if prompted. Then, click Get Started!

  3. In the menu panel, expand Settings and click API Keys.

  4. On the next screen, click the Create API Key button.

  5. Under API Key Name, enter "SCC Email Notifications," select Full Access, and then click the Create & View button.

    API key name

  6. You are shown the API key. Record the value. You need it in the next section.

  7. Click Done. You are shown the current set of API keys. Close the tab and return to the Google Cloud console.

Next, you deploy a Cloud Run function to send notifications to an email address.

Create the SendGrid Cloud Run function

In this section, you deploy a function that sends notifications to your email account.

  1. Go to Cloud Run functions.
    Go to Cloud Run functions

  2. Ensure that you're using the same PROJECT_ID you used to create the Pub/Sub topic.

  3. Click Create Function. SendGrid function

  4. Set Function name to send-high-and-critical-finding-email-notification and the Trigger type to Pub/Sub.

  5. Select the Pub/Sub topic that you created in Setting up a Pub/Sub topic.

  6. Click Save, and then click Next.

  7. On the next page, set Runtime to Python 3.8. The code sample in this section is written in Python, but you can use any language supported by Cloud Run functions.

  8. In the file list, click requirements.txt and add the following to the text field: sendgrid.

    SendGrid requirements

  9. Click main.py and replace the contents with the following code snippet.

    import base64
    import json
    from sendgrid import SendGridAPIClient
    from sendgrid.helpers.mail import Mail
    
    def send_email_notification(event, context):
        """Triggered from a message on a Pub/Sub topic.
        Args:
             event (dict): Event payload.
             context (google.cloud.functions.Context): Metadata for the event.
        """
        pubsub_message = base64.b64decode(event['data']).decode('utf-8')
        message_json = json.loads(pubsub_message)
        message = Mail(
            from_email='noreply@yourdomain.com',
            to_emails='$EMAIL_ADDRESS',
            subject='New High or Critical Severity Finding Detected',
            html_content='A new high or critical severity finding was
            detected: ' + ''.join(message_json['finding']['category']))
        try:
            sg = SendGridAPIClient('$SENDGRID_EMAIL_API_KEY')
            response = sg.send(message)
            print(response.status_code)
            print(response.body)
            print(response.headers)
        except Exception as e:
            print(e)
    
        print(pubsub_message)
    
  10. Replace the following:

    • Change noreply@yourdomain.com to the email address from which you want messages to originate.
    • Change$EMAIL_ADDRESS to the intended recipient's email address. Note: This variable can hold an array of email addresses (['user1@yourdomain.com', 'user2@yourdomain.com']) or you can write custom code to make a dynamic variable set to, for example, a rotating list of individuals who are on-call.
    • Change $SENDGRID_EMAIL_API_KEY to the API key you already have or the one you created in the previous section.
  11. Navigate to the Entry point field and enter the name of the function in the code snippet (send_email_notification, in this example).

  12. Click Deploy. You are returned to the Cloud Run functions list where you should see your new function. When a green check mark appears next to the function name, it has been successfully deployed. The process could take a few minutes. SendGrid function deploy

Slack

To send notifications to a Slack channel, you do the following:

  • Create a new Slack app with sufficient privileges to post messages to a public Slack channel.
  • Create and deploy a Cloud Run function that posts chat messages to Slack when notifications are received from Pub/Sub.

Create a new Slack app

In this section, you create a new Slack app to receive notifications.

  1. Navigate to Slack API apps. The page opens in a new tab.
  2. Sign in or create an account.

  3. Select Create an App.

  4. Set App Name to "SCC Finding Notifier."

  5. Select the Development Slack Workspace where you'd like the Slack Bot to post messages, and then click Create App.

  6. In the navigation panel, select OAuth & Permissions.

    Slack permissions

  7. Navigate to the Scopes section. Scopes fall into two categories:

    • Bot Token Scopes
    • User Token Scopes
  8. For this exercise, you don't need to add a User Token Scope. Under Bot Token Scopes, click Add an OAuth Scope and enter:

    • chat:write
    • chat:write.public

      Slack scopes
  9. Scroll to the top of the OAuth & Permissions page and click Install App to Workspace.

  10. In the confirmation dialog, click Allow.

  11. Copy the Bot User OAuth Access Token for use in the Cloud Function.

Next, you deploy a Cloud Run function to send notifications to a Slack group.

Create the Slack Cloud Run function

In this section, you deploy a function to send notifications to your Slack account.

  1. Go to Cloud Run functions.
    Go to Cloud Run functions

  2. Ensure that you're using the same PROJECT_ID where the Pub/Sub topic was created.

  3. Click Create Function. SendGrid function

  4. Set Function name to slack-chat-high-and-critical-findings and Trigger type to Pub/Sub.

  5. Select the Pub/Sub topic that you created in Setting up a Pub/Sub topic.

  6. Click Save, and then click Next.

  7. On the next page, set Runtime to Python 3.8. The code sample in this section is written in Python, but you can use any language supported by Cloud Run functions.

  8. Navigate to the file list. Click requirements.txt and add the following: requests. Slack requirements

  9. Click main.py and replace its contents with the following code snippet.

    import base64
    import json
    import requests
    
    TOKEN = "BOT_ACCESS_TOKEN"
    
    def send_slack_chat_notification(event, context):
        pubsub_message = base64.b64decode(event['data']).decode('utf-8')
        message_json = json.loads(pubsub_message)
        finding = message_json['finding']
    
        requests.post("https://slack.com/api/chat.postMessage", data={
            "token": TOKEN,
            "channel": "#general",
            "text": f"A high severity finding {finding['category']} was detected!"
        })
    
  10. Replace BOT_ACCESS_TOKEN with the Bot User OAuth Access Token you created with the Slack app.

  11. Navigate to the Entry point field and enter the name of the function in the code snippet (send_slack_chat_notification, in this example).

  12. Click Deploy. You are returned to the Cloud Run functions list where you should see your new function. When a green check mark appears next to the function name, it has been successfully deployed. The process could take a couple of minutes. Messages appear in the #general Slack channel. Slack function

WebEx

To send notifications to your WebEx Teams account, you do the following:

  • Create a new service account that has permissions to retrieve assets from Security Command Center.
  • Create a new WebEx Bot with sufficient privileges to post messages to your workspace.
  • Create and deploy a Cloud Run function that subscribes to Pub/Sub and posts chat messages to WebEx when notifications are received from the Pub/Sub topic.

Create a service account

Cloud Run functions, by default, cannot retrieve assets from Security Command Center. In this section, you provision a service account that allows Cloud Run functions to retrieve assets associated with findings.

  1. Name your service account and specify it as an environment variable.

    export SERVICE_ACCOUNT=ACCOUNT_NAME
    
  2. Create the service account for your project.

    gcloud iam service-accounts create $SERVICE_ACCOUNT \
     --display-name "Service Account for SCC Finding Notifier WebEx Cloud Function" \
     --project $PROJECT_ID
    
  3. Grant the service account the securitycenter.assetsViewer role at the organization level.

     gcloud organizations add-iam-policy-binding $ORG_ID \
       --member="serviceAccount:$SERVICE_ACCOUNT@$PROJECT_ID.iam.gserviceaccount.com" \
       --role='roles/securitycenter.assetsViewer'
    

Create a Webex Bot

In this section, you create a WebEx Bot that posts messages to your workspace.

  1. Sign in to your WebEx Teams account and navigate to the New Bot page.

  2. Set Bot Name to "SCC Finding Notifier".

  3. Select a descriptive but unique Bot Username (your-name-scc-finding-notifier-demo).

  4. Under Icon, select Default 1.

  5. Set Description to "A bot that notifies the team when a new high or critical severity finding is published to Security Command Center."

  6. Click Add Bot.

  7. On the confirmation page, copy the Bot Access Token and save it for use in the Cloud Run function.

Add Webex Bot to workspace

In this section, you add the WebEx Bot to a workspace.

  1. In the WebEx Space where you want the Bot to post notifications, expand the activity panel. WebEx panel

  2. Select Add People.

  3. Type "SCC Finding Notifier" in the text field and select the bot you created from the drop-down menu. WebEx add

  4. Select the Add button, and then close the panel.

  5. Obtain the Room ID of the workspace for the Cloud Run function. On a desktop computer, go to https://developer.webex.com/docs/api/v1/rooms/list- rooms and log in, if necessary. This page uses the WebEx API to list rooms to which you belong. WebEx rooms

  6. Navigate to the search panel.

  7. Select the tab labeled Try it at the top of the panel.

  8. Leave all options with their default values and click Run.

  9. In the Response tab, you receive a JSON-formatted response with a list of items, or rooms. Find the title of the room in which you want notifications to appear and record the associated id value. WebEx
response

Next, you deploy a Cloud Run function to send notifications to your WebEx workspace.

Create the WebEx Cloud Run function

In this section, you deploy a function to send notifications to your WebEx account.

  1. Go to Cloud Run functions.
    Go to Cloud Run functions

  2. Select the same PROJECT_ID in which the Pub/Sub topic was created.

  3. Click Create Function. WebEx function

  4. Set the Function name to webex-teams-high-and-critical-findings and Trigger type to Pub/Sub.

  5. Select the Pub/Sub topic that you created in Setting up a Pub/Sub topic.

  6. Expand the Variables, Networking and Advanced Settings field.

  7. Under Service account, filter for and select the webex-cloud-function-sa service account you created. WebEx variables

  8. Click Save, and then click Next.

  9. On the next page, set Runtime to Python 3.8. The code sample in this section is written in Python, but you can use any language supported by Cloud Run functions.

  10. Navigate to the file list. Click requirements.txt and add the following:

    • requests==2.25.1
    • google-cloud-securitycenter==1.1.0 WebEx requirements
  11. Click main.py and replace the contents with the following code snippet.

    #!/usr/bin/env python3
    import base64
    import json
    
    import requests
    from google.cloud import securitycenter_v1
    
    WEBEX_TOKEN = "WEBEX_TOKEN"
    ROOM_ID = "ROOM_ID"
    
    TEMPLATE = """
    **Severity:** {severity}\n
    **Asset:** {asset}\n
    **SCC Category:** {category}\n
    **Project:** {project}\n
    **First observed:** {create_time}\n
    **Last observed:** {event_time}\n
    **Link to finding:** {finding_link}
    """
    
    PREFIX = "https://console.cloud.google.com/security/command-center/findings"
    
    def get_finding_detail_page_link(finding_name):
        """Constructs a direct link to the finding detail page."""
        org_id = finding_name.split("/")[1]
        return f"{PREFIX}?organizationId={org_id}&resourceId={finding_name}"
    
    def get_asset(parent, resource_name):
        """Retrieves the asset corresponding to `resource_name` from SCC."""
        client = securitycenter_v1.SecurityCenterClient()
        resp = client.list_assets(
            securitycenter_v1.ListAssetsRequest(
                parent=parent,
                filter=f'securityCenterProperties.resourceName="{resource_name}"',
            )
        )
        page = next(resp.pages)
        if page.total_size == 0:
            return None
        asset = page.list_assets_results[0].asset
        return json.loads(securitycenter_v1.Asset.to_json(asset))
    
    def send_webex_teams_notification(event, context):
        """Send the notification to WebEx Teams."""
        pubsub_message = base64.b64decode(event["data"]).decode("utf-8")
        message_json = json.loads(pubsub_message)
        finding = message_json["finding"]
    
        parent = "/".join(finding["parent"].split("/")[0:2])
        asset = get_asset(parent, finding["resourceName"])
    
        requests.post(
            "https://webexapis.com/v1/messages",
            json={
                "roomId": ROOM_ID,
                "markdown": TEMPLATE.format(
                    severity=finding["severity"],
                    asset=asset["securityCenterProperties"]["resourceDisplayName"],
                    category=finding["category"],
                    project=asset["resourceProperties"]["project"],
                    create_time=finding["createTime"],
                    event_time=finding["eventTime"],
                    finding_link=get_finding_detail_page_link(finding["name"]),
                ),
            },
            headers={"Authorization": f"Bearer {WEBEX_TOKEN}"},
        )
    
  12. Replace the following:

    • WEBEX_TOKEN with the Bot Access Token from the Create a WebEx Bot section.
    • ROOM_ID with the Room ID from the Add WebEx Bot to workspace section.
  13. Navigate to the Entry point field and enter the name of the function in the code snippet (send_webex_teams_notification, in this example).

  14. Click Deploy. You are returned to the Cloud Run functions list where you should see your new function. When a green check mark appears next to the function name, it has been successfully deployed. The process can take a few minutes. WebEx deploy

If the preceding steps for your selected service completed without errors, the setup is complete and you start receiving notifications. Keep in mind:

  • You receive a separate email or chat message for each individual critical or high severity finding. The frequency or number of notifications depends on the resources within your organization.
  • Notifications are published and sent in near-real time. However, the immediacy of emails or messages is not guaranteed and multiple factors can cause delays, including issues with SendGrid, your email system, Slack, or WebEx.

To change the notifications workflow, you can do the following:

  • Change recipients by updating your Cloud Run function.
  • Change which findings trigger notifications by updating the filter for the Pub/Sub topic.

Testing notifications

To test whether notifications are properly configured, follow the instructions below to toggle high severity findings between active and inactive states.

  1. Go to the Security Command Center Findings page.
    Go to the Findings page
  2. Select your organization, if prompted.
  3. In the Quick filters panel, scroll down to the Severity section and select either High or Critical. The Findings query results panel updates to show findings of only the selected severity.
  4. In the Findings query results panel, select a finding by checking the box next to its name.
  5. From the Change active state menu in the Findings query results action bar, select Inactive. If the current findings query shows only active findings, the finding is removed from the query results.
  6. In the Quick filters panel, scroll down to the State section and change the selections so that only Inactive is selected. The Findings query results panel updates to show only inactive findings.
  7. In the Findings query results panel, select the finding that you marked inactive.
  8. From the Change active state menu in the Findings query results action bar, select Active.
  9. Check your email or messaging service and you should see a message similar to the images below.

Email:

Email notification

Slack:

Slack notification

Messages sent to WebEx, which contain more information in this guide, resemble the following image.

WebEx notification

Troubleshooting

If emails or chat messages are not being sent or received, follow the steps below to identify and resolve potential issues.

  • SendGrid Email:

    • To stop emails from going to the spam folder, add the from_email value to your email allowlist or configure sender authentication on SendGrid.
    • Be sure you are not exceeding the rate limit for your SendGrid plan.
    • Failed emails can be detected through SendGrid reporting.
      • Your domain's or email provider's DMARC policy might block email from unauthenticated senders. Learn how SendGrid manages sender identity. If there is an error, try another email address in the from_email value.
  • SendGrid Email, Slack, and WebEx:

    1. Check Stackdriver logs for your Cloud Run function to determine whether the function is being invoked. If it is not being invoked, ensure that notifications are set up correctly.

    2. If the Cloud Run function is being invoked, then it might be crashing. Check for errors in Cloud Run functions using Google Cloud console Error Reporting.

Cleaning up

To avoid incurring charges to your Google Cloud account for the resources used in this tutorial, either delete the project that contains the resources, or keep the project and delete the individual resources.

Deleting the project

  1. In the Google Cloud console, go to the Manage resources page.

    Go to Manage resources

  2. In the project list, select the project that you want to delete, and then click Delete.
  3. In the dialog, type the project ID, and then click Shut down to delete the project.

Deleting individual resources

  1. Delete your notification configuration:

    gcloud scc notifications delete organizations/ORG_ID/notificationConfigs/scc-critical-and-high-severity-findings-notify
    

    Replace ORG_ID with your organization ID.

    To maintain your configuration and temporarily pause notifications, comment out the send or post calls in your Cloud Run function.

  2. Delete your Cloud Run function:

    1. Go to Cloud Run functions.
      Go to Cloud Run functions
    2. Click the checkbox next to the function you want to delete.
    3. Click Delete .
  3. Delete the service account:

    1. Go to the Service accounts page.
      Go to the Service accounts page
    2. Select a project.
    3. Select the service account you want to delete, and then click Delete .

What's next