Sending Security Command Center data to Elastic Stack using Docker

This page explains how to use a Docker container to host your Elastic Stack installation, and automatically send Security Command Center findings, assets, audit logs, and security sources to Elastic Stack. It also describes how to manage the exported data.

Docker is a platform for managing applications in containers. Elastic Stack is a security information and event management (SIEM) platform that ingests data from one or more sources and lets security teams manage responses to incidents and perform real-time analytics. The Elastic Stack configuration discussed in this guide includes four components:

• Filebeat: a lightweight agent installed on edge hosts, such as virtual machines (VM), that can be configured to collect and forward data
• Logstash: a transformation service that ingests data, maps it into required fields, and forwards the results to Elasticsearch
• Elasticsearch: a search database engine that stores data
• Kibana: powers dashboards that let you visualize and analyze data

In this guide, you set up Docker, ensure that the required Security Command Center and Google Cloud services are properly configured, and use a custom module to send findings, assets, audit logs, and security sources to Elastic Stack.

The following figure illustrates the data path when using Elastic Stack with Security Command Center.

Security Command Center and Elastic Stack integration (click to enlarge)

Before you begin

Before connecting to Elastic Stack, do the following:

1. Create a service account with the following Identity and Access Management (IAM) roles:

   • Security Center Admin (roles/securitycenter.admin)
   • Organization Viewer (roles/resourcemanager.organizationViewer)
   • Cloud Asset Owner (roles/cloudasset.owner)
   • Pub/Sub Admin (roles/pubsub.admin)
   • Logs Configuration Writer (roles/logging.configWriter)

   For instructions on granting roles, see Granting, changing, and revoking access to resources.

2. Complete one of the following:

   1. If you are hosting the Docker container in Google Cloud, add the service account to the VM that will host your Kubernetes nodes.

   2. If you are hosting the Docker container in your on-premises environment, create a service account key. You will need the service account JSON from this task to complete this guide.

      Note: To learn about best practices for storing your service account keys securely, see Best practices for managing service account keys.

   3. If you are installing the Docker container in another cloud, configure workload identity federation and download the credentials configuration file.

3. Set up finding notifications as follows:

   1. Enable the Security Command Center API.
   2. Create a filter to export desired findings and assets.
   3. Create four Pub/Sub topics for findings, resources, audit logs, and assets. The notificationConfig must use the Pub/Sub topic you create for findings.

   You will need your organization ID, project ID, and Pub/Sub topic names from this task to configure Elastic Stack.

4. Enable the Cloud Asset API for your project.

Install the Docker and Elasticsearch components

Follow these steps to install the Docker and Elastic search components in your environment.

Install Docker Engine and Docker Compose

You can install Docker for use on-premises or with a cloud provider. To get started, complete the following guides in Docker's product documentation:

• Install Docker Engine (Linux)
• Install Docker Compose

Install Elasticsearch and Kibana

The Docker image that you installed in Install Docker includes Logstash and Filebeat. If you don't already have Elasticsearch and Kibana installed, use the following guides to install the applications:

• Installing Elasticsearch

• Install Kibana

You need the following information from those tasks to complete this guide:

• Elastic Stack: host, port, username, and password
• Kibana: host, port, username, and password

Download the GoApp module

This section explains how to download the GoApp module, a Go program maintained by Security Command Center. The module automates the process of scheduling Security Command Center API calls and regularly retrieves Security Command Center data for use in Elastic Stack.

To install GoApp, do the following:

1. In a terminal window, install wget, a free software utility used to retrieve content from web servers.

   For Ubuntu and Debian distributions, run the following:

     # apt-get install wget
   

   For RHEL, CentOS, and Fedora distributions, run the following:

     # yum install wget
   

2. Install unzip, a free software utility used to extract the contents of ZIP files.

   For Ubuntu and Debian distributions, run the following:

     # apt-get install unzip
   

   For RHEL, CentOS, and Fedora distributions, run the following:

     # yum install unzip
   

3. Create a directory for the GoogleSCCElasticIntegration installation package:

     mkdir GoogleSCCElasticIntegration
   

4. Download the GoogleSCCElasticIntegration installation package:

   wget -c https://storage.googleapis.com/security-center-elastic-stack/GoogleSCCElasticIntegration-Installation.zip
   

5. Extract the contents of the GoogleSCCElasticIntegration installation package into the GoogleSCCElasticIntegration directory:

   unzip GoogleSCCElasticIntegration-Installation.zip -d GoogleSCCElasticIntegration
   

6. Create a working directory to store and run GoApp module components:

     mkdir WORKING_DIRECTORY
   

   Replace WORKING_DIRECTORY with the directory name.

7. Navigate to the GoogleSCCElasticIntegration installation directory:

     cd ROOT_DIRECTORY/GoogleSCCElasticIntegration/
   

   Replace ROOT_DIRECTORY with the path to the directory that contains the GoogleSCCElasticIntegration directory.

8. Move install.sh, config.env, and dashboards.ndjson into your working directory.

     mv install.sh config.env 'Kibana Dashboards'/dashboards.ndjson WORKING_DIRECTORY
   

   Replace WORKING_DIRECTORY with the path to your working directory.

Install the Docker container

Note: To complete the tasks in this section, be sure Docker, Docker Compose, Elasticsearch, and Kibana are running.

To set up the Docker container, you download and install a preformatted image from Google Cloud that contains Logstash and Filebeat. For information about the Docker image, go to the Container Registry repository in the Google Cloud console.

Go to Container Registry

During installation, you configure the GoApp module with Security Command Center and Elastic Stack credentials.

1. Navigate to your working directory:

     cd /WORKING_DIRECTORY
   

   Replace WORKING_DIRECTORY with the path to your working directory.

2. Run the following commands to install the Docker image and configure the GoApp module.

     chmod +x install.sh
     ./install.sh
   

3. During the installation process, enter the requested variables:

   Variable	Description
   UPDATE	Whether you are upgrading from a previous version, either N for no or Y for yes
   CLIENT_CREDENTIAL_PATH	One of: The path to your service account JSON, if you are using service account keys The credential configuration file, if you are using workload identity federation, as described in Before you begin
   PROJECT_ID	The ID for project that contains the Pub/Sub topic
   ORGANIZATION_ID	Your organization ID
   FINDING_TOPIC_NAME	The name of the Pub/Sub topic for findings
   FINDING_SUBSCRIPTION_NAME	The name of the Pub/Sub subscription for findings
   ASSET_TOPIC_NAME_RESOURCE	The name of the Pub/Sub topic for resources
   ASSET_SUBSCRIPTION_NAME_RESOURCE	The name of the Pub/Sub subscription for resources
   ASSET_TOPIC_NAME_IAMPOLICY	The name of the Pub/Sub topic for IAM policies
   ASSET_SUBSCRIPTION_NAME_IAMPOLICY	The name of the Pub/Sub subscription for IAM policies
   AUDITLOG_TOPIC_NAME	The name of the Pub/Sub topic for audit logs
   AUDITLOG_SUBSCRIPTION_NAME	The name of the Pub/Sub subscription for audit logs
   ELASTIC_HOST	The IP address of your Elastic Stack host
   ELASTIC_PORT	The port for your Elastic Stack host
   KIBANA_HOST	The IP address or hostname to which the Kibana server will bind
   KIBANA_PORT	The port for the Kibana server
   HTTP_PROXY	An optional link with the username, password, IP address, and port for your proxy host, for example, http://USER:PASSWORD@PROXY_IP:PROXY_PORT
   ELASTIC_USERNAME	Your Elasticsearch username (optional)
   ELASTIC_PASSWORD	Your Elasticsearch password (optional)
   KIBANA_USERNAME	Your Kibana username (optional)
   KIBANA_PASSWORD	Your Kibana password (optional)
   FINDINGS_START_DATE	The optional date to start migrating findings, for example, 2021-04-01T12:00:00+05:30

   The GoApp module downloads the Docker image, installs the image, and sets up the container.

4. When the process is finished, copy the email address of the WriterIdentity service account from the installation output.

   Your working directory should have the following structure:

     ├── config.env
     ├── dashboards.ndjson
     ├── docker-compose.yml
     ├── install.sh
     └── main
         ├── client_secret.json
         ├── filebeat
         │          └── config
         │                   └── filebeat.yml
         ├── GoApp
         │       └── .env
         └── logstash
                    └── pipeline
                               └── logstash.conf
   

Update permissions for audit logs

To update permissions so that audit logs can flow to your SIEM:

1. Navigate to the Pub/Sub topics page.

   Go to Pub/Sub

2. Select your project that includes your Pub/Sub topics.

3. Select the Pub/Sub topic that you created for audit logs.

4. In Permissions, add the WriterIdentity service account (that you copied in step 4 of the installation procedure) as a new principal and assign it the Pub/Sub Publisher role. The audit log policy is updated.

The Docker and Elastic Stack configurations are complete. You can now set up Kibana.

View Docker logs

1. Open a terminal, and run the following command to see your container information, including container IDs. Note the ID for the container where Elastic Stack is installed.

     docker container ls
   

2. To start a container and view its logs, run the following commands:

     docker exec -it CONTAINER_ID /bin/bash
     cat go.log
   

   Replace CONTAINER_ID with the ID of the container where Elastic Stack is installed.

Set up Kibana

Complete these steps when you are installing the Docker container for the first time.

1. Open kibana.yml in a text editor.

     sudo vim KIBANA_DIRECTORY/config/kibana.yml
   

   Replace KIBANA_DIRECTORY with the path to your Kibana installation folder.

2. Update the following variables:

   • server.port: the port to use for Kibana's back-end server; default is 5601
   • server.host: the IP address or hostname to which the Kibana server will bind
   • elasticsearch.hosts: the IP address and port of the Elasticsearch instance to use for queries
   • server.maxPayloadBytes: the maximum payload size in bytes for incoming server requests; default is 1,048,576
   • url_drilldown.enabled: a Boolean value that controls the ability to navigate from Kibana dashboard to internal or external URLS; default is true

   The completed configuration resembles the following:

     server.port: PORT
     server.host: "HOST"
     elasticsearch.hosts: ["http://ELASTIC_IP_ADDRESS:ELASTIC_PORT"]
     server.maxPayloadBytes: 5242880
     url_drilldown.enabled: true
   

Import Kibana dashboards

1. Open the Kibana application.
2. In the navigation menu, go to Stack Management, and then click Saved Objects.
3. Click Import, navigate to the working directory and select dashboards.ndjson. The dashboards are imported and index patterns are created.

Update links to Google Cloud console

Complete these steps when you are installing the Docker container for the first time.

The findings and assets dashboards can include links to assets and findings in Google Cloud.

To update links in the findings dashboard in the Google Cloud console, do the following:

1. In the navigation menu, go to Stack Management.
2. Under Kibana, select Index Pattern.
3. Click gccfindings.
4. Search for the finding.name.keyword field.
5. Open finding.name.keyword in edit mode.

6. Enter the following URL:

   https://https://console.cloud.google.com/security/command-center/findings?organizationId=ORGANIZATION_ID&orgonly=true&supportedpurview=organizationId&view_type=vt_finding_type&vt_severity_type=All&columns=category,resourceName,eventTime,createTime,parent,securityMarks.marks&vt_finding_type=All&resourceId={value}
   

   Replace ORGANIZATION_ID with the ID for your organization.

7. Click Save field.

8. Open the findings dashboard, and click on any value under Finding Name to check whether the link configuration was successful. If there's an issue, modify the URL.

To update the link to the assets dashboard in the Google Cloud console, do the following:

1. In the navigation menu, go to Stack Management.
2. Under Kibana, select Index Pattern.
3. Click gccassets.
4. Search for the Asset Name field.
5. Open Asset Name in edit mode.

6. Enter the following URL:

   https://https://console.cloud.google.com/security/command-center/assets?organizationId=ORGANIZATION_ID&orgonly=true&supportedpurview=organizationId&view_type=vt_asset_type&vt_asset_type=All&columns=securityCenterProperties.resourceType,securityCenterProperties.resourceOwners,securityMarks.marks&pageState=(%22cscc-asset-inventory%22:(%22f%22:%22%255B%257B_22k_22_3A_22securityCenterProperties.resourceName_22_2C_22t_22_3A10_2C_22v_22_3A_22_5C_22\{value}_5C_22_22_2C_22s_22_3Atrue_2C_22i_22_3A_22securityCenterProperties.resourceName_22%257D%255D%22))
   

   Replace ORGANIZATION_ID with the ID for your organization.

7. Click Save field.

8. Open the assets dashboard, and click on any value under Asset Name to check whether the link works. If there's an issue, modify the URL.

Upgrade the Docker container

If you deployed a previous version of the GoApp module, you can upgrade to a newer version. When you upgrade the Docker container to a newer version, you can keep your existing service account setup, Pub/Sub topics, and ElasticSearch components.

If you are upgrading from an integration that didn't use a Docker container, see Upgrade to the latest release.

Note: To complete the tasks in this section, verify that Docker, Docker Compose, Elasticsearch, and Kibana are running.

1. Add the Logs Configuration Writer (roles/logging.configWriter) role to the service account.

2. Create a Pub/Sub topic for your audit logs.

3. If you are installing the Docker container in another cloud, configure workload identity federation and download the credentials configuration file.

4. Optionally, to avoid issues when importing the new dashboards, remove the existing dashboards from Kibana:

   1. Open the Kibana application.
   2. In the navigation menu, go to Stack Management, and then click Saved Objects.
   3. Search for Google SCC.
   4. Select all the dashboards that you want to remove.
   5. Click Delete.

5. Remove the existing Docker container:

   1. Open a terminal and stop the container:

      docker stop CONTAINER_ID
      

      Replace CONTAINER_ID with the ID of the container where Elastic Stack is installed.

   2. Remove the Docker container:

      docker rm CONTAINER_ID
      

      If necessary, add -f before the container ID to remove the container forcefully.

6. Complete the steps in Download the GoApp module.

7. Complete the steps in Install Docker.

8. Complete the steps in Update permissions for audit logs.

9. Import the new dashboards, as described in Import Kibana dashboards. This step will overwrite your existing Kibana dashboards.

View and edit Kibana dashboards

You can use custom dashboards in Elastic Stack to visualize and analyze your findings, assets, and security sources. The dashboards display critical findings and help your security team prioritize fixes.

Overview dashboard

The Overview dashboard contains a series of charts that displays the total number of findings in your organization by severity level, category, and state. Findings are compiled from Security Command Center's built-in services—Security Health Analytics, Web Security Scanner, Event Threat Detection, and Container Threat Detection—and any integrated services you enable.

To filter content by criteria such as misconfigurations or vulnerabilities, you can select the Finding class.

Additional charts show which categories, projects, and assets are generating the most findings.

Note: You must be a Security Command Center Premium customer to receive Event Threat Detection and Container Threat Detection findings.

Assets dashboard

The Assets dashboard displays tables that show your Google Cloud assets. The tables show asset owners, asset counts by resource type and projects, and your most recently added and updated assets.

You can filter asset data by asset name, asset type, and parents, and quickly drill down to findings for specific assets. If you click an asset name, you are redirected to Security Command Center's Assets page in the Google Cloud console and shown details for the selected asset.

Audit logs dashboard

The Audit logs dashboard displays a series of charts and tables that show audit log information. The audit logs that are included in the dashboard are the administrator activity, data access, system events, and policy denied audit logs. The table includes the time, severity, log type, log name, service name, resource name, and resource type.

You can filter the data by source (such as a project), severity, log type, and resource type.

Findings dashboard

The Findings dashboard includes charts showing your most recent findings. The charts provide information about the number of findings, their severity, category, and state. You can also view active findings over time, and which projects or resources have the most findings.

You can filter the data by finding class.

If you click a finding name, you are redirected to Security Command Center's Findings page in the Google Cloud console and shown details for the selected finding.

Sources dashboard

The Sources dashboard shows the total number of findings and security sources, the number of findings by source name, and a table of all your security sources. Table columns include name, display name, and description.

Add columns

1. Navigate to a dashboard.
2. Click Edit, and then click Edit visualization.
3. Under Add sub-bucket, select Split rows.
4. In the list, select Aggregation.
5. In the Descending drop-down menu, select ascending or descending. In the Size field, enter the maximum number of rows for the table.
6. Select the column you want to add.
7. Save the changes.

Remove columns

1. Navigate to the dashboard.
2. Click Edit.
3. To hide columns, next to the column name, click the visibility, or eye, icon. To remove a column, next to the colume name, click on the X, or delete, icon.

Uninstall Docker

1. To see your container information, including container IDs, open the terminal and run the following command:

     docker container ls
   

2. Stop the container:

     docker stop CONTAINER_ID
   

   Replace CONTAINER_ID with the ID of the container where Elastic Stack is installed.

3. Remove the Docker container:

     docker rm CONTAINER_ID
   

   If necessary, add -f before the container ID to remove the container forcefully.

4. Remove the Docker image:

     docker rmi us.gcr.io/security-center-gcr-host/googlescc_elk:latest
   

5. Remove Pub/Sub feeds for assets, findings, IAM policies, and audit logs.

6. Delete the working directory.

What's next

• Learn more about setting up finding notifications in Security Command Center.

• Read about filtering finding notifications in Security Command Center.