Enabling real-time email and chat notifications

Overview

Security Command Center provides real-time notifications of findings in the Google Cloud console. This guide describes how to use Google Cloud services and third-party APIs to extend that functionality and receive near real-time notifications in email and chat apps. When you complete the guide, you get alerts to new findings in configured third-party services without logging into the Google Cloud console, allowing for faster triaging of vulnerabilities and threats. Learn more about the different types of vulnerabilities and threats in Security Command Center.

Topology

In this guide, you create the configuration that is illustrated in the following diagram.

Near real-time notifications workflow for Security Command Center (click to enlarge)

Objectives

In this guide, you do the following:

1. Set up a Pub/Sub topic.
2. Set up Slack, WebEx Teams, and SendGrid Email.
3. Write code in Cloud Run functions.
4. Configure Pub/Sub and Cloud Run functions to send notifications to Slack, WebEx Teams, or SendGrid Email whenever a new high or critical severity finding is written to Security Command Center.
5. Troubleshoot notification problems.

Costs

This tutorial uses billable components of Google Cloud, including:

• Pub/Sub
• Cloud Run functions
• Cloud Build

Use the pricing calculator to generate a cost estimate based on your projected usage.

Before you begin

To complete this guide, you must have the following Identity and Access Management (IAM) roles:

• Organization Admin (roles/resourcemanager.organizationAdmin)
• Security Center Admin (roles/securitycenter.admin)
• Security Admin (roles/iam.securityAdmin)
• A role with the serviceusage.services.use permission, such as Owner (roles/owner), Editor (roles/editor), or a custom role
• Create Service Accounts (roles/iam.serviceAccountCreator)
• Pub/Sub Editor (roles/pubsub.editor)
• Billing Account Administrator (roles/billing.admin)

The IAM roles for Security Command Center can be granted at the organization, folder, or project level. Your ability to view, edit, create, or update findings, assets, and security sources depends on the level for which you are granted access. To learn more about Security Command Center roles, see Access control.

Setting up a project

Complete the following steps to create or select a project.

1. Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.

2. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

   Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

   Go to project selector

3. Make sure that billing is enabled for your Google Cloud project.

4. Enable the Cloud Build API.

   Enable the API

5. Install the Google Cloud CLI.

6. To initialize the gcloud CLI, run the following command:

   gcloud init

7. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

   Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

   Go to project selector

8. Make sure that billing is enabled for your Google Cloud project.

9. Enable the Cloud Build API.

   Enable the API

10. Install the Google Cloud CLI.

11. To initialize the gcloud CLI, run the following command:

    gcloud init

Estimated time: it takes approximately one hour to set up and test notifications for a single service.

Setting up a Pub/Sub topic

Important: If you already configured Security Command Center to publish findings in Pub/Sub, you can skip to Setting up a messaging app. Otherwise, follow the instructions in this section to set up a Pub/Sub topic.

Pub/Sub is a real-time messaging service that enables messages to be sent and received between independent applications. Learn more about Pub/Sub.

In this section, you configure Security Command Center to publish findings to a Pub/Sub topic.

To set up and subscribe to a Pub/Sub topic:

1. Specify your Google Cloud project in an environment variable.

   Note: This project ID is used throughout this document.

   export PROJECT_ID=PROJECT_ID
   

   Replace PROJECT_ID with your project ID.

2. Specify your Google Cloud organization in an environment variable.

   export ORG_ID=ORG_ID
   

   Replace ORG_ID with your organization ID.

3. Set the project ID for gcloud commands.

   gcloud config set project PROJECT_ID
   

4. Create the Pub/Sub topic where notifications are published.

   gcloud pubsub topics create scc-critical-and-high-severity-findings-topic
   

5. Specify the topic in an environment variable.

   export TOPIC=projects/$PROJECT_ID/topics/scc-critical-and-high-severity-findings-topic
   

6. Create the subscription that notifies Cloud Run functions to send an email or chat message when messages are published to the topic.

   gcloud pubsub subscriptions create scc-critical-and-high-severity-findings-sub \
     --topic scc-critical-and-high-severity-findings-topic
   

   Note: By default, Pub/Sub subscriptions expire after 31 days of inactivity, defined as subscriptions without active connections, pull requests, or push successes. To keep subscriptions longer, set the --expiration-period flag to a higher value or the special value never, indicating the subscription never expires. See Using subscription properties for more information.

7. Configure Security Command Center to publish notifications to the topic. Any filter compatible with the ListFindings API can be used.

   The following filter publishes notifications for active high and critical severity findings. Learn more about filtering findings.

    gcloud scc notifications create scc-critical-and-high-severity-findings-notify \
      --pubsub-topic $TOPIC \
      --organization $ORG_ID \
      --filter "(severity=\"HIGH\" OR severity=\"CRITICAL\") AND state=\"ACTIVE\""
   

   Note: If you use VPC Service Controls, you might need to grant access to protected project resources before creating a NotificationConfig and in order for notifications to work. For more information, see Granting perimeter access in VPC Service Controls.

Next, you create or configure your email or chat app to receive notifications from Pub/Sub.

Setting up a messaging app

This section describes how to use Pub/Sub and Cloud Run functions or Cloud Run functions (1st gen) to enable near real-time notifications for SendGrid Email API, Slack, and WebEx Teams.

Note: The SendGrid and Slack examples show how to send notifications with basic information (the finding category). The WebEx solution adds the additional step of creating a service account to include additional information, such as discovery times, remediation steps, and links to findings.

You can customize any of the examples to include as much or as little information as you want.

SendGrid Email

To enable email notifications, you do the following:

• Create a SendGrid Email API account and obtain an API key.
• Create and deploy a Cloud Run function that sends emails when notifications are received from Pub/Sub.

Create SendGrid Email API account

In this section, you create a SendGrid Email API account and obtain an API key. If you already have SendGrid enabled, skip to Obtain a Sendgrid Email API Key and ensure your existing API key has adequate permissions.

1. Go to the Google Cloud console.
   Go to the Google Cloud console
   
2. In the search box at the top of the page, search for SendGrid Email API.

3. On the next page, select the plan that fits your needs.

   • The free plan allows up to 12,000 emails per month and is sufficient for this guide, but large organizations might require more. If you anticipate a larger volume of email notifications, consider adding additional filters to notifications to exclude noisy findings.
   • You might be asked to select a project to associate with SendGrid. Proceed by selecting a project. You might need adequate permissions to manage purchases for the project's associated billing account.

4. Review the terms and if you're comfortable, click Subscribe.

5. Activate the SendGrid service by clicking Register with SendGrid.

6. On the registration screen, enter a username, password, and email address. Accept the terms of service and click Continue.

7. At the confirmation dialog, click Return to Google.

Obtain a SendGrid Email API key

1. Click Manage API keys on SendGrid website. A new tab opens for the SendGrid website.

2. Complete the form or sign in, if prompted. Then, click Get Started!

3. In the menu panel, expand Settings and click API Keys.

4. On the next screen, click the Create API Key button.

5. Under API Key Name, enter "SCC Email Notifications," select Full Access, and then click the Create & View button.
   
   

6. You are shown the API key. Record the value. You need it in the next section.

7. Click Done. You are shown the current set of API keys. Close the tab and return to the Google Cloud console.

Next, you deploy a Cloud Run function to send notifications to an email address.

Create the SendGrid Cloud Run function

In this section, you deploy a function that sends notifications to your email account.

1. Go to Cloud Run functions.
   Go to Cloud Run functions
   

2. Ensure that you're using the same PROJECT_ID you used to create the Pub/Sub topic.

3. Click Create Function.

4. Set Function name to send-high-and-critical-finding-email-notification and the Trigger type to Pub/Sub.

5. Select the Pub/Sub topic that you created in Setting up a Pub/Sub topic.

6. Click Save, and then click Next.

7. On the next page, set Runtime to Python 3.8. The code sample in this section is written in Python, but you can use any language supported by Cloud Run functions.

8. In the file list, click requirements.txt and add the following to the text field: sendgrid.
   
   

9. Click main.py and replace the contents with the following code snippet.

   Note: This code sample requires you to hardcode user or application credentials. For added security, and to safeguard credentials, consider using Secret Manager with Cloud Run functions. For instructions, see Using secrets in Cloud Run functions documentation.

   import base64
   import json
   from sendgrid import SendGridAPIClient
   from sendgrid.helpers.mail import Mail
   
   def send_email_notification(event, context):
       """Triggered from a message on a Pub/Sub topic.
       Args:
            event (dict): Event payload.
            context (google.cloud.functions.Context): Metadata for the event.
       """
       pubsub_message = base64.b64decode(event['data']).decode('utf-8')
       message_json = json.loads(pubsub_message)
       message = Mail(
           from_email='[e-mail]',
           to_emails='$EMAIL_ADDRESS',
           subject='New High or Critical Severity Finding Detected',
           html_content='A new high or critical severity finding was
           detected: ' + ''.join(message_json['finding']['category']))
       try:
           sg = SendGridAPIClient('$SENDGRID_EMAIL_API_KEY')
           response = sg.send(message)
           print(response.status_code)
           print(response.body)
           print(response.headers)
       except Exception as e:
           print(e)
   
       print(pubsub_message)
   

10. Replace the following:

    • Change [e-mail] to the email address from which you want messages to originate.
    • Change$EMAIL_ADDRESS to the intended recipient's email address. Note: This variable can hold an array of email addresses (['[e-mail]', '[e-mail]']) or you can write custom code to make a dynamic variable set to, for example, a rotating list of individuals who are on-call.
    • Change $SENDGRID_EMAIL_API_KEY to the API key you already have or the one you created in the previous section.

11. Navigate to the Entry point field and enter the name of the function in the code snippet (send_email_notification, in this example).

12. Click Deploy. You are returned to the Cloud Run functions list where you should see your new function. When a green check mark appears next to the function name, it has been successfully deployed. The process could take a few minutes.

Slack

To send notifications to a Slack channel, you do the following:

• Create a new Slack app with sufficient privileges to post messages to a public Slack channel.
• Create and deploy a Cloud Run function that posts chat messages to Slack when notifications are received from Pub/Sub.

Create a new Slack app

In this section, you create a new Slack app to receive notifications.

1. Navigate to Slack API apps. The page opens in a new tab.

2. Sign in or create an account.

3. Select Create an App.

4. Set App Name to "SCC Finding Notifier."

5. Select the Development Slack Workspace where you'd like the Slack Bot to post messages, and then click Create App.

6. In the navigation panel, select OAuth & Permissions.
   
   

7. Navigate to the Scopes section. Scopes fall into two categories:

   • Bot Token Scopes
   • User Token Scopes

8. For this exercise, you don't need to add a User Token Scope. Under Bot Token Scopes, click Add an OAuth Scope and enter:

   • chat:write
   • chat:write.public
     
     

9. Scroll to the top of the OAuth & Permissions page and click Install App to Workspace.

10. In the confirmation dialog, click Allow.

11. Copy the Bot User OAuth Access Token for use in the Cloud Function.

Next, you deploy a Cloud Run function to send notifications to a Slack group.

Create the Slack Cloud Run function

In this section, you deploy a function to send notifications to your Slack account.

1. Go to Cloud Run functions.
   Go to Cloud Run functions
   

2. Ensure that you're using the same PROJECT_ID where the Pub/Sub topic was created.

3. Click Create Function.

4. Set Function name to slack-chat-high-and-critical-findings and Trigger type to Pub/Sub.

5. Select the Pub/Sub topic that you created in Setting up a Pub/Sub topic.

6. Click Save, and then click Next.

7. On the next page, set Runtime to Python 3.8. The code sample in this section is written in Python, but you can use any language supported by Cloud Run functions.

8. Navigate to the file list. Click requirements.txt and add the following: requests.

9. Click main.py and replace its contents with the following code snippet.

   Note: This code sample requires you to hardcode user or application credentials. For added security, and to safeguard credentials, consider using Secret Manager with Cloud Run functions. For instructions, see Using secrets in Cloud Run functions documentation.

   import base64
   import json
   import requests
   
   TOKEN = "BOT_ACCESS_TOKEN"
   
   def send_slack_chat_notification(event, context):
       pubsub_message = base64.b64decode(event['data']).decode('utf-8')
       message_json = json.loads(pubsub_message)
       finding = message_json['finding']
   
       requests.post("https://slack.com/api/chat.postMessage", data={
           "token": TOKEN,
           "channel": "#general",
           "text": f"A high severity finding {finding['category']} was detected!"
       })
   

10. Replace BOT_ACCESS_TOKEN with the Bot User OAuth Access Token you created with the Slack app.

11. Navigate to the Entry point field and enter the name of the function in the code snippet (send_slack_chat_notification, in this example).

12. Click Deploy. You are returned to the Cloud Run functions list where you should see your new function. When a green check mark appears next to the function name, it has been successfully deployed. The process could take a couple of minutes. Messages appear in the #general Slack channel.

WebEx

To send notifications to your WebEx Teams account, you do the following:

• Create a new service account that has permissions to retrieve assets from Security Command Center.
• Create a new WebEx Bot with sufficient privileges to post messages to your workspace.
• Create and deploy a Cloud Run function that subscribes to Pub/Sub and posts chat messages to WebEx when notifications are received from the Pub/Sub topic.

Create a service account

Cloud Run functions, by default, cannot retrieve assets from Security Command Center. In this section, you provision a service account that allows Cloud Run functions to retrieve assets associated with findings.

1. Name your service account and specify it as an environment variable.

   export SERVICE_ACCOUNT=ACCOUNT_NAME
   

2. Create the service account for your project.

   gcloud iam service-accounts create $SERVICE_ACCOUNT \
    --display-name "Service Account for SCC Finding Notifier WebEx Cloud Function" \
    --project $PROJECT_ID
   

3. Grant the service account the securitycenter.assetsViewer role at the organization level.

    gcloud organizations add-iam-policy-binding $ORG_ID \
      --member="serviceAccount:$SERVICE_ACCOUNT@$PROJECT_ID.iam.gserviceaccount.com" \
      --role='roles/securitycenter.assetsViewer'
   

Create a Webex Bot

In this section, you create a WebEx Bot that posts messages to your workspace.

1. Sign in to your WebEx Teams account and navigate to the New Bot page.

2. Set Bot Name to "SCC Finding Notifier".

3. Select a descriptive but unique Bot Username (your-name-scc-finding-notifier-demo).

4. Under Icon, select Default 1.

5. Set Description to "A bot that notifies the team when a new high or critical severity finding is published to Security Command Center."

6. Click Add Bot.

7. On the confirmation page, copy the Bot Access Token and save it for use in the Cloud Run function.

Add Webex Bot to workspace

In this section, you add the WebEx Bot to a workspace.

1. In the WebEx Space where you want the Bot to post notifications, expand the activity panel.

2. Select Add People.

3. Type "SCC Finding Notifier" in the text field and select the bot you created from the drop-down menu.

4. Select the Add button, and then close the panel.

5. Obtain the Room ID of the workspace for the Cloud Run function. On a desktop computer, go to https://developer.webex.com/docs/api/v1/rooms/list- rooms and log in, if necessary. This page uses the WebEx API to list rooms to which you belong.

6. Navigate to the search panel.

7. Select the tab labeled Try it at the top of the panel.

8. Leave all options with their default values and click Run.

9. In the Response tab, you receive a JSON-formatted response with a list of items, or rooms. Find the title of the room in which you want notifications to appear and record the associated id value.

Next, you deploy a Cloud Run function to send notifications to your WebEx workspace.

Create the WebEx Cloud Run function

In this section, you deploy a function to send notifications to your WebEx account.

1. Go to Cloud Run functions.
   Go to Cloud Run functions
   

2. Select the same PROJECT_ID in which the Pub/Sub topic was created.

3. Click Create Function.

4. Set the Function name to webex-teams-high-and-critical-findings and Trigger type to Pub/Sub.

5. Select the Pub/Sub topic that you created in Setting up a Pub/Sub topic.

6. Expand the Variables, Networking and Advanced Settings field.

7. Under Service account, filter for and select the webex-cloud-function-sa service account you created.

8. Click Save, and then click Next.

9. On the next page, set Runtime to Python 3.8. The code sample in this section is written in Python, but you can use any language supported by Cloud Run functions.

10. Navigate to the file list. Click requirements.txt and add the following:

    • requests==2.25.1
    • google-cloud-securitycenter==1.1.0

11. Click main.py and replace the contents with the following code snippet.

    Note: This code sample requires you to hardcode user or application credentials. For added security, and to safeguard credentials, consider using Secret Manager with Cloud Run functions. For instructions, see Using secrets in Cloud Run functions documentation.

    #!/usr/bin/env python3
    import base64
    import json
    
    import requests
    from google.cloud import securitycenter_v1
    
    WEBEX_TOKEN = "WEBEX_TOKEN"
    ROOM_ID = "ROOM_ID"
    
    TEMPLATE = """
    **Severity:** {severity}\n
    **Asset:** {asset}\n
    **SCC Category:** {category}\n
    **Project:** {project}\n
    **First observed:** {create_time}\n
    **Last observed:** {event_time}\n
    **Link to finding:** {finding_link}
    """
    
    PREFIX = "https://console.cloud.google.com/security/command-center/findings"
    
    def get_finding_detail_page_link(finding_name):
        """Constructs a direct link to the finding detail page."""
        org_id = finding_name.split("/")[1]
        return f"{PREFIX}?organizationId={org_id}&resourceId={finding_name}"
    
    def get_asset(parent, resource_name):
        """Retrieves the asset corresponding to `resource_name` from SCC."""
        client = securitycenter_v1.SecurityCenterClient()
        resp = client.list_assets(
            securitycenter_v1.ListAssetsRequest(
                parent=parent,
                filter=f'securityCenterProperties.resourceName="{resource_name}"',
            )
        )
        page = next(resp.pages)
        if page.total_size == 0:
            return None
        asset = page.list_assets_results[0].asset
        return json.loads(securitycenter_v1.Asset.to_json(asset))
    
    def send_webex_teams_notification(event, context):
        """Send the notification to WebEx Teams."""
        pubsub_message = base64.b64decode(event["data"]).decode("utf-8")
        message_json = json.loads(pubsub_message)
        finding = message_json["finding"]
    
        parent = "/".join(finding["parent"].split("/")[0:2])
        asset = get_asset(parent, finding["resourceName"])
    
        requests.post(
            "https://webexapis.com/v1/messages",
            json={
                "roomId": ROOM_ID,
                "markdown": TEMPLATE.format(
                    severity=finding["severity"],
                    asset=asset["securityCenterProperties"]["resourceDisplayName"],
                    category=finding["category"],
                    project=asset["resourceProperties"]["project"],
                    create_time=finding["createTime"],
                    event_time=finding["eventTime"],
                    finding_link=get_finding_detail_page_link(finding["name"]),
                ),
            },
            headers={"Authorization": f"Bearer {WEBEX_TOKEN}"},
        )
    

12. Replace the following:

    • WEBEX_TOKEN with the Bot Access Token from the Create a WebEx Bot section.
    • ROOM_ID with the Room ID from the Add WebEx Bot to workspace section.

13. Navigate to the Entry point field and enter the name of the function in the code snippet (send_webex_teams_notification, in this example).

14. Click Deploy. You are returned to the Cloud Run functions list where you should see your new function. When a green check mark appears next to the function name, it has been successfully deployed. The process can take a few minutes.

If the preceding steps for your selected service completed without errors, the setup is complete and you start receiving notifications. Keep in mind:

• You receive a separate email or chat message for each individual critical or high severity finding. The frequency or number of notifications depends on the resources within your organization.
• Notifications are published and sent in near-real time. However, the immediacy of emails or messages is not guaranteed and multiple factors can cause delays, including issues with SendGrid, your email system, Slack, or WebEx.

To change the notifications workflow, you can do the following:

• Change recipients by updating your Cloud Run function.
• Change which findings trigger notifications by updating the filter for the Pub/Sub topic.

Testing notifications

To test whether notifications are properly configured, follow the instructions below to toggle high severity findings between active and inactive states.

Note: Event Threat Detection only returns iam_anomalous_grant findings for Security Command Center users with a gmail.com email address.

1. Go to the Security Command Center Findings page.
   Go to the Findings page
   
2. Select your organization, if prompted.
3. In the Quick filters panel, scroll down to the Severity section and select either High or Critical. The Findings query results panel updates to show findings of only the selected severity.
4. In the Findings query results panel, select a finding by checking the box next to its name.
5. From the Change active state menu in the Findings query results action bar, select Inactive. If the current findings query shows only active findings, the finding is removed from the query results.
6. In the Quick filters panel, scroll down to the State section and change the selections so that only Inactive is selected. The Findings query results panel updates to show only inactive findings.
7. In the Findings query results panel, select the finding that you marked inactive.
8. From the Change active state menu in the Findings query results action bar, select Active.

9. Check your email or messaging service and you should see a message similar to the images below.

   Note: Email messages might appear in your spam folder.

Email:

Slack:

Messages sent to WebEx, which contain more information in this guide, resemble the following image.

Troubleshooting

If emails or chat messages are not being sent or received, follow the steps below to identify and resolve potential issues.

• SendGrid Email:

  • To stop emails from going to the spam folder, add the from_email value to your email allowlist or configure sender authentication on SendGrid.
  • Be sure you are not exceeding the rate limit for your SendGrid plan.
  • Failed emails can be detected through SendGrid reporting.
    • Your domain's or email provider's DMARC policy might block email from unauthenticated senders. Learn how SendGrid manages sender identity. If there is an error, try another email address in the from_email value.

• SendGrid Email, Slack, and WebEx:

  1. Check Stackdriver logs for your Cloud Run function to determine whether the function is being invoked. If it is not being invoked, ensure that notifications are set up correctly.

  2. If the Cloud Run function is being invoked, then it might be crashing. Check for errors in Cloud Run functions using Google Cloud console Error Reporting.

Cleaning up

To avoid incurring charges to your Google Cloud account for the resources used in this tutorial, either delete the project that contains the resources, or keep the project and delete the individual resources.

Deleting the project

Caution: Deleting a project has the following effects:

• Everything in the project is deleted. If you used an existing project for the tasks in this document, when you delete it, you also delete any other work you've done in the project.
• Custom project IDs are lost. When you created this project, you might have created a custom project ID that you want to use in the future. To preserve the URLs that use the project ID, such as an appspot.com URL, delete selected resources inside the project instead of deleting the whole project.

If you plan to explore multiple architectures, tutorials, or quickstarts, reusing projects can help you avoid exceeding project quota limits.

1. In the Google Cloud console, go to the Manage resources page.

   Go to Manage resources

2. In the project list, select the project that you want to delete, and then click Delete.
3. In the dialog, type the project ID, and then click Shut down to delete the project.

Deleting individual resources

1. Delete your notification configuration:

   gcloud scc notifications delete organizations/ORG_ID/notificationConfigs/scc-critical-and-high-severity-findings-notify
   

   Replace ORG_ID with your organization ID.

   To maintain your configuration and temporarily pause notifications, comment out the send or post calls in your Cloud Run function.

2. Delete your Cloud Run function:

   1. Go to Cloud Run functions.
      Go to Cloud Run functions
      
   2. Click the checkbox next to the function you want to delete.
   3. Click Delete .

3. Delete the service account:

   1. Go to the Service accounts page.
      Go to the Service accounts page
   2. Select a project.
   3. Select the service account you want to delete, and then click Delete .

What's next

• Read more about notification errors.
• Learn about filtering notifications.
• Learn how to remediate Web Security Scanner and Security Health Analytics findings.