>

Installing the Creator app

This page provides the Creator example of the Cloud Security Command Center (Cloud SCC) application package. Creator can do queries on Cloud SCC data at regular intervals and send the results to a Cloud Pub/Sub topic.

This guide was written for tools version 3.3.0. If you're using a different version, please see the README file included with the tools version you downloaded. As of May 22, 2019, the most recent release version is 4.0.1.

Before you begin

Before you start this guide, you must complete the prerequisites and installation setup in Setting up Cloud SCC tools.

To install and run the Creator package, you will also need the following:

  • An active GCP Organization
  • The following Cloud Identity and Access Management (Cloud IAM) roles at the organization level:
    • Organization Administrator - roles/resourcemanager.organizationAdmin
    • Project Creator - roles/resourcemanager.projectCreator
    • Billing Account User - roles/billing.user
    • Viewer - roles/viewer
    • Service Usage Admin - roles/serviceusage.serviceUsageAdmin
    • API Keys Admin - roles/serviceusage.apiKeysAdmin
    • Pub/Sub Admin - roles/pubsub.admin
    • Cloud Datastore Owner - roles/datastore.owner
    • Cloud Scheduler Admin - roles/cloudscheduler.admin
    • App Engine Admin - roles/appengine.appAdmin
    • Cloud Build Editor - roles/cloudbuild.builds.editor
    • Service Account Admin - roles/iam.serviceAccountAdmin
    • Service Account Key Admin - roles/iam.serviceAccountKeyAdmin
    • Project IAM Admin - roles/resourcemanager.projectIamAdmin
    • Storage Admin - roles/storage.admin
    • Storage Object Admin - roles/storage.objectAdmin
    • Pub/Sub Publisher - roles/pubsub.publisher
  • An active Cloud Billing account
  • A service account created on the project with Cloud SCC enabled on the organization.

Setting up environment variables

  1. Go to the Google Cloud Platform Console.
    Go to the GCP Console page
  2. Click Activate Cloud Shell.
  3. Run the following commands to set environment variables. Use the tools release version you downloaded during setup. This guide was written for tools version 3.3.0. For other tools versions, see the README included with the files you downloaded.

    # Release version you downloaded during setup
    export VERSION=[RELEASE_VERSION]
    
    # Directory to unzip the installation files
    export WORKING_DIR=${HOME}/scc-tools-install
    
    # Organzation ID where the script will run
    export ORGANIZATION_ID=[YOUR_ORG_ID]
    
    # Project ID to be created
    export CREATOR_PROJECT_ID=[YOUR_CREATOR_PROJECT_ID]
    
    # A valid billing account ID
    export BILLING=[YOUR_BILLING_ACCOUNT_ID]
    
    # A selected location from the App Engine Locations:
    # https://cloud.google.com/appengine/docs/locations) list
    export GAE_LOCATION=[YOUR_LOCATION]
    
  4. On the Cloud Shell menu bar, click Upload file on the More devshell settings menu.

  5. Upload the scc-creator-${VERSION}.zip file you downloaded during the installation setup.

  6. Unzip the file you uploaded by running:

    unzip -qo scc-creator-${VERSION}.zip -d ${WORKING_DIR}
    
  7. Go to the installation working directory:

    cd ${WORKING_DIR}
    

Installing the Creator app package

In any of the following sections, you can simulate executions of the commands by using the option --simulation.

Step 1: Creating the project

Create the project in which you'll install the Creator app, and then link it to your billing account by running:

gcloud projects create ${CREATOR_PROJECT_ID} \
  --organization ${ORGANIZATION_ID}

gcloud beta billing projects link ${CREATOR_PROJECT_ID} \
  --billing-account ${BILLING}

Step 2: Enabling App Engine for the app

The Creator app uses App Engine as its execution environment. To select a App Engine location and create an app to install to, run:

gcloud app create \
 --region  ${GAE_LOCATION} \
 --project ${CREATOR_PROJECT_ID}

Note that you can't change the location after you set it.

Step 3: Enabling Google APIs

The Creator app uses Google APIs to function. Enable the APIs needed by the project by running:

gcloud services enable \
  securitycenter.googleapis.com \
  servicemanagement.googleapis.com \
  cloudresourcemanager.googleapis.com \
  appengine.googleapis.com \
  compute.googleapis.com \
  pubsub.googleapis.com \
  cloudbuild.googleapis.com \
  storage-component.googleapis.com \
  --project ${CREATOR_PROJECT_ID}

Step 4: Creating a Cloud SCC client service account

To deploy the Creator app, you need a service account that has the following roles:

  • Security Center Assets Viewer - roles/securitycenter.assetsViewer
  • Security Center Findings Viewer - roles/securitycenter.findingsViewer

If you installed the Cloud SCC tools with an earlier version of the Cloud SCC API, like alpha or beta, you must create a new service account for the new API version.

To create a service account and grant it the necessary roles, run the following commands:

  1. Create the service account:

    gcloud iam service-accounts create scc-viewer  \
     --display-name "SCC Viewer SA"  \
     --project ${CREATOR_PROJECT_ID}
    
  2. Download the service account key file:

    (cd setup; \
     gcloud iam service-accounts keys create \
     service_accounts/scc-viewer-${CREATOR_PROJECT_ID}-service-account.json \
     --iam-account scc-viewer@${CREATOR_PROJECT_ID}.iam.gserviceaccount.com)
    
  3. Grant the Organization Level roles:

    gcloud beta organizations add-iam-policy-binding ${ORGANIZATION_ID} \
     --member="serviceAccount:scc-viewer@${CREATOR_PROJECT_ID}.iam.gserviceaccount.com" \
     --role='roles/securitycenter.assetsViewer'
    gcloud beta organizations add-iam-policy-binding ${ORGANIZATION_ID} \
    --member="serviceAccount:scc-viewer@${CREATOR_PROJECT_ID}.iam.gserviceaccount.com" \
    --role='roles/securitycenter.findingsViewer'
    
  4. Export the absolute path to the service account key file:

    export SCC_SA_FILE=[PATH_TO_SERVICE_ACCOUNT_FILE]
    

Step 4: Creating an API key

Before you run the Creator app setup, Create an API key and restrict it to the Cloud SCC application, then export its value to an environment variable:

  1. Go to the APIs & Services > Credentials page in the GCP Console.
    Go to the Credentials page
  2. On the Create credentials drop-down list, click API key.
  3. On the API key created dialog that appears, copy your API key. You will need this in the steps below.
  4. Export the API key as an environment variable by running:

    export API_KEY=[YOUR_API_KEY]
    

Step 5: Creating a publisher service account

The Creator app uses a publisher service account with the Cloud Pub/Sub Publisher role to publish to topics. Create this service account by running the follwing:

  1. Create the service account:

    gcloud iam service-accounts create publisher  \
     --display-name "Publisher"  \
     --project ${CREATOR_PROJECT_ID}
    
  2. Grant the publisher role:

    gcloud beta organizations add-iam-policy-binding ${ORGANIZATION_ID} \
     --member="serviceAccount:publisher@${CREATOR_PROJECT_ID}.iam.gserviceaccount.com" \
     --role='roles/pubsub.publisher'
    
  3. Export the absolute path to the service account file to an environment variable:

    export PUBLISHER_SA_FILE=[YOUR_PUBLISHER_SERVICE_ACCOUNT]
    

Step 6: Running the application setup

Start the application setup to create the remaining infrastructure and Cloud Pub/Sub topics, and deploy Creator by running:

(cd setup; \
export PUBLISHER_SA_FILE=./service_accounts/${CREATOR_PROJECT_ID}_publisher.json; \
pipenv run python3 run_setup_creator.py \
  --organization_id ${ORGANIZATION_ID} \
  --creator_project ${CREATOR_PROJECT_ID} \
  --creator_sa_file ${SCC_SA_FILE} \
  --scc_api_key ${API_KEY} \
  --publisher_sa_file ${PUBLISHER_SA_FILE} \
  --no-simulation)

During deployment, a token is created to validate the requests for the URLs in the app used internally for the Cloud Pub/Sub integration. You can check this value in the following file on your home directory: .google_code_lab_creator_cloud_function_secret_client_key_demomode.

Configuring the application

The Creator app has configuration options to select the operation mode and the queries you want to run.

Configuring the operation mode

The Creator app can run in DEMO or PRODUCTION mode:

  • In PRODUCTION mode, a App Engine cron job periodically triggers the execution of the stored queries.
  • In DEMO mode, the cron job is ignored and a Cloud Pub/Sub topic message is used to trigger the execution of the stored queries. DEMO mode should be used for validation only.

To change the operation mode, you must have the Cloud IAM Pub/Sub Publisher role on the organization or project creator level.

The first time you run the Creator app, it's best to set the app operation mode to DEMO for validation. To set the operation mode, run the load.py helper script:

  1. Install Python dependencies:

    (cd creator/cli; \
      pipenv --python 3.5.3; \
      pipenv install --ignore-pipfile)
    
  2. Set the application default login:

    gcloud auth application-default login
    
  3. Set mode to PRODUCTION or DEMO:

    (cd creator/cli; \
    pipenv run python3 load.py mode \
      --mode <PRODUCTION|DEMO> \
      --project ${CREATOR_PROJECT_ID})
    

Configuring the executable query

To add a query to be executed, run the following script that uses a sample query from creator/samples:

(cd creator/cli; \
pipenv run python3 load.py event_queries \
  --yaml_file ../samples/one_step_asset_for_testing.yaml \
  --project ${CREATOR_PROJECT_ID})

For more examples of queries that you can use in the app, see the creator/samples folder in the scc-creator-[VERSION].zip file you downloaded earlier. For information about creating queries, see the Managing queries section below.

Validating the installation

Step 1: Verifying the query upload

(cd creator/cli; \
pipenv run python3 load.py list_queries \
  --project ${CREATOR_PROJECT_ID})

This should return a single query on the list with the name: find Assets of resourceType google.cloud.resourcemanager.Organization.

Step 2: Creating a Cloud Pub/Sub test topic

gcloud pubsub topics create projects/${CREATOR_PROJECT_ID}/topics/testtopic \
  --project ${CREATOR_PROJECT_ID}

Step 3: Creating a subscription

gcloud pubsub subscriptions create projects/${CREATOR_PROJECT_ID}/subscriptions/testsubscription \
  --topic testtopic \
  --topic-project ${CREATOR_PROJECT_ID} \
  --project ${CREATOR_PROJECT_ID}

Step 4: Running the query

(cd creator/cli; \
pipenv run python3 load.py execute_queries \
  --project ${CREATOR_PROJECT_ID})

Step 5: Getting results from the subscription

gcloud pubsub subscriptions pull projects/${CREATOR_PROJECT_ID}/subscriptions/testsubscription \
  --auto-ack \
  --limit 1 \
  --format json \
  --project ${CREATOR_PROJECT_ID}

Queries

This section includes information about how to manage queries, like uploading a new query or deleting a query, query format, and provides some example queries.

Managing queries

You can use the helper script to manage queries and complete the following tasks:

  • Upload queries
  • Execute all queries
  • List all queries
  • Delete a queries
  • Delete all queries

Uploading queries

(cd creator/cli; \
pipenv run python3 load.py event_queries \
  --yaml_file [PATH_TO_QUERY_FILE] \
  --project ${CREATOR_PROJECT_ID})

Executing all queries

(cd creator/cli; \
pipenv run python3 load.py execute_queries \
  --project ${CREATOR_PROJECT_ID})

Listing all queries

(cd creator/cli; \
pipenv run python3 load.py list_queries \
  --project ${CREATOR_PROJECT_ID})

Deleting a query

(cd creator/cli; \
pipenv run python3 load.py delete_query \
  --project ${CREATOR_PROJECT_ID} --key [KEY_TO_BE_DELETED])

Where [KEY_TO_BE_DELETED] is the value read from the list_queries results.

Deleting all queries

(cd creator/cli; \
pipenv run python3 load.py delete_all_queries \
  --project ${CREATOR_PROJECT_ID})

Query format

The Creator app uses the list_assets and list_findings APIs to get information from Cloud SCC. These APIs are restricted to return items from a single organization.

The Creator app maps the available parameters passed to the API on the .yaml files that contain the queries the system executes.

Following is a sample .yaml with a two-step query example:

- name: "Find FORSETI findings on projects owned by [USER]"
  joins:
    - field: securityCenterProperties.resourceName
      kind: ASSET
      order: 1
      type: SINGLE
    - field: resourceName
      kind: FINDING
      order: 2
      type: MULTI
  steps:
    - duration: 43w
      kind: ASSET
      order: 1
      referenceTime:
        type: timestamp
        value: "2018-04-02T10:30:42+0400"
      where: "securityCenterProperties.resourceType = \"google.cloud.resourcemanager.Project\" AND securityCenterProperties.resourceOwners : \"[USER]\""
    - kind: FINDING
      order: 2
      referenceTime:
        type: from_now
        value: 1d+7h+20m
      where: "parent : \"organizations/000000000000/sources/0000000000000000000\""
  threshold:
    operator: gt
    value: 2
  topic: cloudflare_topic

This example includes a query "Find FORSETI findings on projects owned by [USER]" that has two steps:

  1. Search projects owned by a user with login "[USER]".
  2. Search findings from FORSETI on projects found in step 1.

Following is a description of each part of the example:

  • name: an identifier in human readable form for the query.
  • joins: a list that explains which field from the result of the first step should be used to on the second step. Each element contains:
    • order: identifies if it is related to the first or second step
    • kind: identifies the entity it's related to, currently one of ASSET or FINDING.
    • field: the field that is extracted to construct the join information. In queries with Assets and Findings, this is usually the asset_id.
    • type: indicates if the field is a scalar value or an array.
    • If you're extracting information from a MULTI field, the system needs to split the array in individual values.
    • If you're building the where for a step on a MULTI field, the system needs to use the contains operator ", instead of the equals operator =.
  • steps: a list with the actual information to be used when calling the asset or findings APIs. Each one contains:
    • order: indicates if it is the first or the second step.
    • kind: idenifites it's related to, currently one of ASSET or FINDING. These will be queried against the Cloud SCC API.
    • where: has the actual query that will be sent to the API for this step. It can query Attributes, Properties or marks. You can use use the contains operator " or the equals operator =. You can also use AND and OR.
    • referenceTime: an optional field to represent the moment in time of the search. It has two modes:
    • TIMESTAMP mode where you pass a fixed moment in time as a String.
    • FROM_NOW mode where you pass a time amount.
      • The format is {#weeks}w+{#days}d+{#hours}h+{#minutes}m+{#seconds}s.
      • The moment of the query will be the current date minus the time amount informed.
      • 4w would mean 4*7*24*60*60 seconds before the current date.
      • 3d+4h+15s is 3*24*60*60 seconds before the current date.
    • DURATION: an optional field, only valid for the ASSET kind. This represents a time amount in the form {#weeks}w+{#days}d+{#hours}h+{#minutes}m+{#seconds}s. It will represent an interval before the reference time when the asset existence will be validated.
    • threshold: an <operator,value> pair to set a condition that defines when the result of a query should trigger a message to be posted to the Notifier app. This comparison is executed against the size of the response of the last step of the query. The operator can be one of the following:
    • lt lower than
    • le lower or equal
    • eq equal
    • ne not equal
    • ge greater or equal
    • gt greater than
  • topic: an optional field with the name of a Cloud Pub/Sub topic where the results of these queries should be posted.
    • If not present, results will be posted to the default topic.
    • If the topic does not exist, it will be created by the application.

If the size of a query doesn't satisfy the threshold condition, nothing is posted to the Cloud Pub/Sub topic.

Query examples

Working with datetime type query params

The following example query will help check if any firewall was created or deleted in a one-hour interval starting 5 minutes before the query execution. This could be used to periodically query the API.

order: 1
kind: ASSET
where: "securityCenterProperties.resourceType = \"google.compute.Firewall\""
duration: "1h"
referenceTime:
  type: from_now
  value: "5m"

The following example query will check if there was any network created or deleted in the first week of February.

order: 1
kind: ASSET
where: "securityCenterProperties.resourceType = \"google.compute.Network\""
duration: "1w"
referenceTime:
  type: timestamp
  value: "2018-02-08T00:00:00+0400"

Using Scalar and Multi-value fields

order: 1
kind: ASSET
where: "securityCenterProperties.resourceType = \"google.cloud.resourcemanager.Project\" AND securityCenterProperties.resourceOwners : \"[USER]\""

securityCenterProperties.resourceType is an example of a scalar field. You can use both comparison operators, : and =. The equality operator will do a full match and the : will do a substring match in the original value.

securityCenterProperties.resourceOwners is an example of a multi-value field. A sample value could be:

securityCenterProperties.resourceOwners = ["useralpha@example.com","userbeta@example.com","userga@example.com"]

To query the field value for the user useralpha, you could use the following example as a way to query a multi-value attribute for individual values:

where: "securityCenterProperties.resourceOwners : \"useralpha@example.com\""

Changing the cron job interval

The default cron job interval for the Creator app is configured to run every 15 minutes. When the job triggers, it calls the Creator app to run the currently uploaded queries if the app is on PRODUCTION mode.

To change the interval at which the cron job runs the Creator app, edit the schedule field of cron.yaml in the creator\appflex folder:

cron:
- description: test task
  url: /events/processor
  schedule: [INTERVAL]

After you edit cron.yaml, upload the change by running:

(cd creator/appflex; \
gcloud app deploy cron.yaml)

For more information, see Scheduling Jobs with cron.yaml

¿Te sirvió esta página? Envíanos tu opinión:

Enviar comentarios sobre…

Cloud Security Command Center
¿Necesitas ayuda? Visita nuestra página de asistencia.