Verify data ingestion using test rules

Supported in:

Google SecOps SIEM

Google Security Operations curated detections include a set of test rule sets that help you verify that data required for each rule set is in the correct format.

These test rules are under the Managed Detection Testing category. Each rule set validates that data received by the test device is in a format expected by rules for that specified category.

Rule set name	Description
GCP Managed Detection Testing	Verifies that Google Cloud data is successfully ingested from devices supported by the Cloud Threats category. See Verify Google Cloud data ingestion for Cloud Threats category for more information.
AWS Managed Detection Testing	Verifies that AWS data is successfully ingested from devices supported by the Cloud Threats category. See Verify AWS data ingestion for Cloud Threats category for more information.
Linux Managed Detection Testing	Verifies that data is successfully ingested from devices supported by the Linux Threats category. See Verify data ingestion for Linux Threats category for more information.
Windows Managed Detection Testing	Verifies that data is successfully ingested from devices supported by the Windows Threats category. See Verify data ingestion for Windows Threats category for more information.

Follow the steps in this document to test and verify that incoming data is ingested correctly and is in the correct format.

Verify Google Cloud data ingestion for Cloud Threats category

These rules help verify whether log data is being ingested as expected for Google Security Operations Curated Detections.

The following steps describes how to test data using the:

• Cloud Audit Metadata Testing rule: To trigger this rule, add a unique and expected Custom Metadata key to any Compute Engine virtual machine that is sending data to Google Security Operations.

• Cloud DNS Testing rule: To trigger this rule, perform a DNS lookup to the domain (chronicle.security) within any virtual machine that has access to the internet and is sending log data to Google Security Operations.

• SCC Managed Detection Testing rules: To trigger these rules, perform multiple actions in the Google Cloud console.

• Cloud Kubernetes Node Testing rule: To trigger this rule, create a test project that is sending log data to Google Security Operations, and create a unique node pool in an existing Google Kubernetes Engine cluster.

Step 1. Enable the test rules

1. Log in to Google Security Operations.
2. Open the Curated Detections page.
3. Click Rules & Detections > Rule Sets.
4. Expand the Managed Detection Testing section. You may need to scroll the page.
5. Click GCP Managed Detection Testing in the list to open the detail page.
6. Enable both Status and Alerting for the Cloud Managed Detection Testing rules.

Step 2. Send data for Cloud Audit Metadata Testing rule

Important: The following steps must be performed as an IAM user in the chosen project that has permission to edit the Instance Metadata of a virtual machine, The user account must have setInstanceMetadata permission.

To trigger the test, complete the following steps:

1. Choose a project within your organization.
2. Go to Compute Engine, and then choose a virtual machine within the project.
3. Within the virtual machine, click Edit, and then perform the following under the Custom MetaData section:
   • Click Add Item.
   • Enter the following information:
     • Key: GCTI_ALERT_VALIDATION_TEST_KEY
     • Value: works
   • Click Save.

4. Perform the following steps to verify the alert was triggered:

   1. Log in to Google Security Operations
   2. Open the Curated Detections page, and then click Dashboard
   3. Check that the tst_GCP_Cloud_Audit_Metadata rule was triggered in the detection list.

Step 3. Send data for Cloud DNS Testing rule

The following steps must be performed as an IAM user in the chosen project that has access to a Compute Engine virtual machine.

To trigger the test, complete the following steps:

1. Choose a project within your organization.
2. Go to Compute Engine, then choose a virtual machine within the project.
   • If it is a Linux virtual machine, make sure you have SSH access.
   • If it is a Windows virtual machine, make sure you have RDP access.
3. Click SSH (Linux) or RDP (Microsoft Windows) to access the virtual machine.

4. Send test data using one of the following steps:

   • Linux virtual machine: After accessing the virtual machine using SSH, run one of the following commands: nslookup chronicle.security or host chronicle.security

     If the command fails, install dnsutils on the virtual machine using one of the following commands:

     • sudo apt-get install dnsutils (for Debian/Ubuntu)
     • dnf install bind-utils (for RedHat/CentOS)
     • yum install bind-utils

   • Microsoft Windows virtual machine: After accessing the virtual machine using RDP, go to any installed browser and browse to the following URL: https://chronicle.security

5. Perform the following steps to verify the alert was triggered:

   1. Log in to Google Security Operations
   2. Open the Curated Detections page, and then click Dashboard
   3. Check that the tst_GCP_Cloud_DNS_Test_Rule rule was triggered in the detection list.

Step 4. Send data for Cloud Kubernetes Node Testing rules

The following steps must be performed as an IAM user in the chosen project that has access to Google Kubernetes Engine resources. For more detailed information about creating regional clusters and node pools, see Create a regional cluster with a single-zone node pool. These test rules are intended to verify data ingestion from the KUBERNETES_NODE log type.

To trigger the test rules, complete the following steps:

1. Create a project within your organization, named chronicle-kube-test-project. This project is used for testing only.
2. Navigate to the Google Kubernetes Engine page in the Google Cloud console.
   Go to the Google Kubernetes Engine page
3. Click Create to create a new regional cluster in the project. Configure the cluster per your organization requirements.
4. Click Add node pool.
5. Name the node pool kube-node-validation, and then adjust the pool size to 1 node per zone.
6. Delete the test resources:
   1. After the kube-node-validation node pool has been created, delete the node pool.
   2. Delete the chronicle-kube-test-project test project.

7. Log in to Google Security Operations.

8. Open the Curated Detections page, and then click Dashboard.

9. Check that the tst_GCP_Kubernetes_Node rule was triggered in the detection list.

10. Check that the tst_GCP_Kubernetes_CreateNodePool rule was triggered in the detection list.

Step 5. Send data for SCC Managed Detection Testing rules

The steps in the following section verify that Security Command Center findings, and related data, are ingested correctly and in the expected format.

The SCC Managed Detection Testing rule sets under the Managed Detection Testing category enable you to verify that data required for the CDIR SCC Enhanced rule sets is sent to Google Security Operations and is in the correct format.

Each test rule validates that data is received in a format expected by rules. You perform actions in your Google Cloud environment to send data that will generate a Google Security Operations alert.

Make sure to complete the following sections of this document required to configure logging in Google Cloud services, collect Security Command Center Premium findings, and send Security Command Center findings to Google Security Operations:

• Required data sets and log types
• Enable and configure CDIR SCC Enhanced rule sets

To learn more about the Security Command Center alerts described in this section, see the Security Command Center document Investigating and Responding to Threats.

Trigger the CDIR SCC Persistence test rule

To send data that triggers this alert in Google Security Operations, perform the following steps:

1. In the Google Cloud console, create a new VM instance and temporarily assign the Compute Engine default service account with Editor privileges. You will remove this after the test is complete.

2. When the new instance is available, assign the Access Scope to Allow Full Access to all APIs.

3. Create a new service account with following information:

   • Set Service account name to scc-test.
   • Set Service account ID to scc-test.
   • Optionally, enter a Description for the service account.

   See the Create service accounts document for information about how to create service accounts.

4. Connect using SSH to the test instance created in the earlier step, and then execute the following gcloud command:

   gcloud projects add-iam-policy-binding PROJECT_NAME
   --member="serviceAccount:scc-test@PROJECT_NAME.iam.gserviceaccount.com"
   --role="roles/owner`"
   

   Replace PROJECT_NAME with the name of the project where the Compute Engine instance is running and where the scc-test account was created.

   The Persistence: IAM Anomalous Grant Security Command Center alert should fire.

5. Sign in to Google Security Operations, and then open the Alerts & IOCs page.

6. You should see a Google Security Operations alert titled Test SCC Alert: IAM Anomalous Grant given to test account.

7. Open the Google Cloud console, and then do the following:

   • Remove the scc-test test account access from IAM and Admin Console.
   • Delete the service account using the Service Accounts portal.
   • Delete the VM instance that you just created.

Trigger the CDIR SCC Malware test rule

To send data that triggers this alert in Google Security Operations, perform the following steps:

1. In the Google Cloud console, connect using SSH to any VM instance where the curl command is installed.

2. Execute the following command:

     curl etd-malware-trigger.goog
   

   After you execute this command, the Malware: Bad Domain Security Command Center alert should fire.

3. Sign in to Google Security Operations, and then open the Alerts & IOCs page.

4. Verify that you see a Google Security Operations alert titled Test SCC Alert: Malware Bad Domain.

Trigger the CDIR SCC Defense Evasion test rule

To send data that triggers this alert in Google Security Operations, perform the following steps:

1. Sign in to Google Cloud console using an account that has access at the organization level to modify VPC Service Control Perimeters.

2. In the Google Cloud console, go to the VPC Service Controls page.

   Go to VPC Service Controls

3. Click +New Perimeter and configure the following fields in the Details page:

   • Perimeter Title: scc_test_perimeter.
   • Perimeter Type to Regular perimeter (default).
   • Config Type to Enforced.

4. In the left navigation, select 3 Restricted Services.

5. In the Specify services to restrict dialog, select Google Compute Engine API, and then click Add Google Compute Engine API.

6. In the left navigation, click Create Perimeter.

7. To modify the perimeter, go to the VPC Service Perimeters page. For more detailed information about how to access this page, see List and describe service perimeters.

8. Select scc_test_perimeter, and then select Edit Perimeter.

9. Under Restricted Services, click the Delete icon to remove the Google Compute Engine API service. This should trigger the Defense Evasion: Modify VPC Service Control Perimeter alert in SCC.

10. Sign in to Google Security Operations, and then open the Alerts & IOCs page.

11. Verify that you see a Google Security Operations alert titled Test SCC Alert: Modify VPC Service Control Test Alert.

Trigger the CDIR SCC Exfiltration test rule

To send data that triggers this alert in Google Security Operations, perform the following steps:

1. In the Google Cloud console, go to a Google Cloud project, and then open BigQuery.

   Go to BigQuery

2. Create a CSV file with the following data, and then save it to your home directory.

   column1, column2, column3
   data1, data2, data3
   data4, data5, data6
   data7, data8, data9
   

3. In the left navigation, choose Create Dataset.

4. Set the following configuration, and then click Create Dataset:

   • Dataset ID set to scc_test_dataset.
   • Location type set to Multi-region.
   • Enable Table expiration: do not select this option.

   

   For more detailed information about creating a dataset, see the BigQuery document Creating datasets.

5. In the left navigation, to the right of scc_test_dataset, click the icon, then select Create Table.

6. Create a table and set the following configuration:

   • Create table from: set to Upload.
   • Select file: browse to your home directory and select the CSV file you created earlier.
   • File format: set to CSV.
   • Dataset: set to css_test_dataset.
   • Table type: set to Native table.

7. Accept the default configuration for all other fields, and then click Create Table.

   

   For more detailed information about creating a table, see the BigQuery document Create and use tables.

8. In the resources list, select the css_test_dataset table, then click Query and choose in New Tab.

   

9. Run the following query:

   SELECT * FROM TABLE_NAME LIMIT 1000`
   

   Replace TABLE_NAME with the fully qualified table name.

10. After the query executes, click Save Results, and then choose CSV in Google Drive. This should trigger the Exfiltration: BigQuery Exfiltration to Google Drive Security Command Center alert. The Security Command Center finding should be sent to Google Security Operations and trigger a Google Security Operations alert.

    

11. Sign in to Google Security Operations, and then open the Alerts & IOCs page.

12. Verify that you see a Google Security Operations alert titled Test SCC Alert: BigQuery Exfiltration to Google Drive.

Step 6. Disable the test rules

When you are finished, disable the GCP Managed Detection Testing rules.

1. Log in to Google Security Operations.
2. Open the Curated Detections page.
3. Disable both Status and Alerting for the GCP Managed Detection Testing rules.

Verify AWS data ingestion for Cloud Threats category

You can use AWS Managed Detection Testing test rules to verify that AWS data is being ingested to Google Security Operations. These test rules help verify that AWS data was ingested and is in the expected format. After setting up the ingestion of AWS data, you perform actions in AWS that should trigger the test rules.

• The user who enables these rules in Detection Engine must have the curatedRuleSetDeployments.batchUpdate IAM permission.
• The user who performs the steps to send AWS data must have the AWS IAM permissions to edit the tags of an EC2 instance in the chosen account. For more information about tagging EC2 instances, see the AWS document Tag your Amazon EC2 resources.

Enable the AWS Managed Detection Testing test rules

1. In Google Security Operations, click Detections > Rules & Detections to open the Curated Detections page.
2. Select the Managed Detection Testing > AWS Managed Detection Testing.
3. Enabled both Status and Alerting for the Broad and Precise rules.

Verify that tag actions in AWS trigger the test rule

Perform the following steps to verify that the tag actions in AWS trigger the rule set.

Step 1. Generate a log event in AWS.

Important: The following steps must be performed as an IAM user in the chosen account that has permission to edit the tags of an EC2 instance.

1. Choose an account within your AWS environment.
2. Go to the EC2 Dashboard, and then choose an Instance within the account.
3. Within the EC2 Instance, click Actions, then Instance Settings, and perform the following under the Manage Tags section:
   1. Click Add new tag.
   2. Enter the following information:
   3. Key: GCTI_ALERT_VALIDATION_TEST_KEY
   4. Value: works
   5. Click Save.

For more detailed information, see Add or remove EC2 instance tags

Step 2. Verify that the test alerts are triggered.

After performing the task in the previous step, verify that the AWS CloudTrail Test Rule rule is triggered. This indicates that CloudTrail logs were recorded and sent to Google Security Operations as expected. Perform the following steps to verify the alert:

1. In Google Security Operations, click Detections > Rules & Detections to open the Curated Detections page.
2. Click Dashboard.
3. In the list of detections, check that the tst_AWS_Cloud_Trail_Tag rule was triggered.

Verify that AWS GuardDuty sample findings trigger test rules

To ensure GuardDuty alerts will work as intended in your environment, you can send GuardDuty sample findings to Google Security Operations.

Step 1. Generate GuardDuty sample findings data.

1. Navigate to AWS Console home.
2. Under Security, Identity, and Compliance, open GuardDuty.
3. Navigate to GuardDuty's Settings.
4. Click Generate Sample findings.

For more information about how to generate sample GuardDuty findings, see Generating sample findings in GuardDuty.

Step 2. Verify the test alerts were triggered.

1. In Google Security Operations, click Detection > Rules & Detections to open the Curated Detections page.
2. Click Dashboard.
3. Check that AWS CloudTrail Test Rule was triggered in the detection list.

Disable the AWS Managed Detection Testing rule sets

1. In Google Security Operations, click Detection > Rules & Detections to open the Curated Detections page.
2. Select the Managed Detection Testing > AWs Managed Detection Testing rules.
3. Disable both Status and Alerting for the Broad and Precise rules.

Verify data ingestion for Linux Threats category

The Linux Managed Detection Testing rules verify that logging on a Linux system is working correctly for Google Security Operations curated detections. The tests involve using the Bash prompt in a Linux environment to run various commands and can be performed by any user that has access to the Linux Bash prompt.

Step 1. Enable the test rules

1. Log in to Google Security Operations.
2. Open the Curated Detections page.
3. Click Rules & Detections > Rule Sets.
4. Expand the Managed Detection Testing section. You may need to scroll the page.
5. Click Linux Managed Detection Testing in the list to open the detail page.
6. Enable both Status and Alerting for the Linux Managed Detection Testing rules.

Step 2. Send test data from a Linux device

To trigger the Linux Managed Detection Testing test rules, perform the following steps:

1. Access any Linux device where data is being sent to Google Security Operations.
2. Open a new Linux Bash prompt command line interface as any user.

3. Enter the following command, and then press Enter:

   /bin/echo hello_chronicle_world!

You must use the echo binary, rather than the Linux shell built-in echo command.

1. Enter the following command, and then press Enter:

   sudo useradd test_chronicle_account

2. Remove the test account created in the previous step. Execute the command:

   sudo userdel test_chronicle_account

3. Enter the following command, and then press Enter:

   su

4. When prompted for the password, enter any random string. Notice that the su: Authentication failure message is displayed.

5. Close the Bash window.

Step 3. Verify that alerts were triggered in Google Security Operations

Verify that the command triggered the *tst_linux_echo, tst_linux_failed_su_login, and tst_linux_test_account_creation rules in Google Security Operations. This indicates that the Linux logs are written and sent as expected. To verify the alert in Google Security Operations, perform the following steps:

1. Log in to Google Security Operations.
2. Open the Curated Detections page.
3. Click Dashboard.

4. Verify that the tst_linux_echo, tst_linux_failed_su_login, and tst_linux_test_account_creation rules were triggered in the detection list.

   Note: There may be a slight delay before the alert is displayed in Google Security Operations.

Step 4. Disable the test rules

When you are finished, disable the Linux Managed Detection Testing rules.

1. Log in to Google Security Operations.
2. Open the Curated Detections page.
3. Disable both Status and Alerting for the Linux Managed Detection Testing rules.

Verify data ingestion for Windows Threats category

The Windows Echo Test Rule verifies that Microsoft Windows logging is working correctly for Google Security Operations curated detections. The test involves using the command prompt in a Microsoft Windows environment to run the echo command with an expected and unique string.

You can run the test while logged on as any user that has access to the Windows Command Prompt.

Step 1. Enable the test rules

1. Log in to Google Security Operations.
2. Open the Curated Detections page.
3. Expand the Managed Detection Testing section. You may need to scroll the page.
4. Click Windows Managed Detection Testing in the list to open the detail page.
5. Enable both Status and Alerting for the Windows Managed Detection Testing rules.

Step 2. Send test data from a Windows device

To trigger the Windows Echo Test Rule, perform the following steps:

1. Access any device that generates data that is to be sent to Google Security Operations.
2. Open a new Microsoft Windows Command Prompt window as any user.

3. Enter the following case-insensitive command, and then press Enter:

   cmd.exe /c "echo hello_chronicle_world!"
   

4. Close the Command Prompt window.

Step 3. Verify that an alert was triggered

Verify that the command triggered the tst_Windows_Echo rule in Google Security Operations. This indicates that Microsoft Windows logging is sending data as expected. To verify the alert in Google Security Operations, perform the following steps:

1. Log in to Google Security Operations.
2. Open the Curated Detections page.
3. Click Dashboard.

4. Verify that the tst_Windows_Echo rule was triggered in the detection list.

   Note: There will be a slight delay for the alert to display in Google Security Operations.

Step 4. Disable the test rules

When you are finished, disable the Windows Managed Detection Testing rules.

1. Log in to Google Security Operations.
2. Open the Curated Detections page.
3. Disable both Status and Alerting for the Windows Managed Detection Testing rules.