Feed management user guide

Overview and prerequisites

Chronicle feed management lets you create and manage data feeds to your Chronicle account. The feed management UI is built on the feed management API. See the Feed management API for additional information.

Each data feed has its own set of prerequisites that must be completed prior to setting up the feed in Chronicle. These prerequisites are outlined in the Feed configuration by type section of the Feed management API documentation. Search for the data feed type you need to setup and follow the instructions provided.

Delete Source Files

As mentioned in the Overview section, different types of feeds have different prerequisites.

For several feed types, including Cloud Storage, there is a field in the Add new or Edit feed workflow labeled SOURCE DELETION OPTION. It is a drop-down menu with three options:

1. Never delete files
2. Delete transferred files and empty directories
3. Delete transferred files

Options 2 and 3 involve deletions: one for files and one for files AND any empty directories. If you select either of those options, you need to add the permissions specific to your feed type, which can be found in the Feed configuration by type section of the Feed management API documentation.

This option lets you delete an object out of the storage system after you have transferred it. Feeds always remember which objects (or files) they have transferred and never transfer the same file twice (unless it has been updated), but you have to set this option if you want the system to actually delete the source object after it has been (successfully) transferred.

Creating and Editing Feeds

To access the Feed management interface, select Settings > Feeds from the navigation bar.

The data feeds listed on this page include all the feeds Google has configured for your account in addition to the feeds you have configured.

Figure 1. Feeds

Add a feed

To add a feed to your Chronicle account, complete the following steps. Be sure to complete the prerequisites for the data feed type you plan to add prior to attempting to add a new feed here. See the Overview and prerequisites section for more information.

You can add up to five feeds for each log type.

1. Click Add New. The ADD FEED window is displayed.

2. Starting from the Set Properties tab, select the SOURCE TYPE from the drop down menu. The source type is the mechanism by which you intend to bring data into Chronicle. You can select from the following feed source types:

   • Amazon S3
   • Amazon SQS
   • Google Cloud Storage
   • HTTP(S) Files (non-API)
   • Microsoft Azure Blob Storage
   • Third party API
   Note: For Amazon SQS, make sure you enable the permissions for Chronicle to be able to delete the Amazon SQS queue messages.

3. Select the Log Type from the drop-down menu. The logs available vary depending on which source type you selected previously. Click Next.

   If you had selected Google Cloud Storage as the source type, use the Get service account option to get a unique service account. In this document, see Google Cloud Storage feed setup example.

   

   Figure 2. Log Type Selection

4. Specify the parameters needed from the Input Parameters tab. The options presented here vary depending on the source and log type selected on the Set Properties tab. Hold the pointer over the question icon for each field to get additional information on what you need to provide.

5. (Optional) You can specify a namespace here. For more information about namespaces, see the asset namespace documentation.

6. Click Next.

   

   Figure 3. Input Parameter Configuration

7. Review your new feed configuration from the Finalize tab. Click Submit when you are ready. Chronicle completes a validation check of the new feed. If the feed passes the check, a name is generated for the feed, it is submitted to Chronicle, and Chronicle begins to attempt to fetch data.

   

   Figure 4. Finalize Feed Request

Google Cloud Storage feed setup example

1. From the Chronicle menu, select Settings, and then click Feeds.
2. Click Add New.
3. Select Google Cloud Storage for Source Type.
4. Select the Log type. For example, to create a feed for Google Kubernetes Engine audit logs, select Google Kubernetes Engine audit logs as the Log Type.
5. Click Get service account. Chronicle provides a unique service account that Chronicle uses to ingest data.
6. Configure access for the service account to access the Cloud Storage objects. In this document, see Grant access to the Chronicle service account.
7. Click Next.
8. Based on the Cloud Storage configuration that you created, specify values for the following fields:
   • Storage bucket URI
   • URI is a
   • Source deletion option
9. Click Next and then Submit.

Grant access to the Chronicle service account

1. In the Google Cloud console, go to the Cloud Storage Buckets page.
   

   Go to Buckets

2. Grant access to the service account to the relevant Cloud Storage objects.

   • To grant read permission to a specific file, complete the following steps:

     1. Select the file and click Edit access.
     2. Click Add principal.
     3. In the New principals field, enter the name of the Chronicle service account.
     4. Assign a role that contains the read permission to the Chronicle service account. For example, Storage Object Viewer (roles/storage.objectViewer). This can only be done if you have not enabled uniform bucket-level access.
     5. Click Save.

   • To grant read permission to multiple files, you must grant access at the bucket level. You must add the Chronicle service account as a principal to your storage bucket and grant it the IAM Storage Object Viewer (roles/storage.objectViewer) role.

     If you configure the feed to delete source files, you must add the Chronicle service account asa principal on your bucket and grant it the IAM Storage Object Admin (roles/storage.objectAdmin) role.

Configure VPC Service Controls

If VPC Service Controls is enabled, an ingress rule is required to provide access to the Cloud Storage bucket.

The following Cloud Storage methods must be allowed in the ingress rule:

• google.storage.objects.list. Required for a single file feed.
• google.storage.objects.get. Required for feeds that require directory or subdirectory access.
• google.storage.objects.delete. Required for feeds that require deletion of the source file.

Sample ingress rule

- ingressFrom:
  identities:
    - serviceAccount:[e-mail]
  sources:
  - accessLevel: "*"
  ingressTo:
  operations:
  - serviceName: storage.googleapis.com
    methodSelectors:
    - method: google.storage.objects.list
    - method: google.storage.objects.get
    - method: google.storage.objects.delete
  resources:
  - projects/PROJECT_ID

Feed status

You can monitor the status of the feed from the initial Feeds page. Feeds can have the following statuses:

• Active—Feed is configured and ready to ingest data into your Chronicle account.
• InProgress—Chronicle is now attempting to pull data from the configured third party.
• Completed—Data successfully retrieved by this feed.
• Archived—Disabled feed.
• Failed—Feed is failing to successfully fetch data. This is likely due to a configuration issue. Click the question to display the configuration error. Once you have corrected the error and resubmitted the feed, return to the Feeds page to determine whether or not the feed is now working.

Edit feeds

From the Feeds page, you can edit an existing feed:

1. Hold the pointer over an existing feed and click the three dot menu in the right column.

2. Click Edit Feed. You can now alter the Input Parameters for the Feed and resubmit it to Chronicle. Chronicle will attempt to use the edited feed.

Enable and disable feeds

In the Status column, enabled feeds are labeled as Active, InProgress, Completed, or Failed. Disabled fields are labeled as Archived. For a description, see the feed status.

From the Feeds page, you can enable or disable any of the existing feeds:

1. Hold the pointer over an existing feed and click the three dot menu in the right column.

2. To enable a feed, click the Enable Feed toggle.

3. To disable a feed, click the Disable Feed toggle. The feed is now labeled as Archived.

Delete feeds

From the Feeds page, you can also delete an existing feed:

1. Hold the pointer over an existing feed and click the three dot menu in the right column.

2. Click Delete Feed. The DELETE FEED window opens. To permanently delete the feed, click Yes, Delete It.