Feed management user guide

Overview and prerequisites

Chronicle Feed Management enables you to create and manage data feeds to your Chronicle account. The Feed Management UI is built on the Feed Management API. See here for additional information.

Each data feed has its own set of prerequisites that must be completed prior to setting up the feed in Chronicle. These prerequisites are outlined in the Feed configuration by type section of the Feed management API documentation. Search for the data feed type you need to setup and follow the instructions provided.

For example, if you setup a data feed to a Google Cloud Storage account, you would need to complete the following prerequisites:

1. Grant Chronicle access.

2. Add the email address [e-mail] to the permissions of the relevant Google Cloud Storage object(s).

3. Perform the following actions from the Cloud Storage section in the Google Cloud Console (console.cloud.google.com)

   • To grant read permission to a specific file, you can "Edit access" on that file and grant the above email "Reader" access. This can only be done if you have not enabled uniform bucket-level access.
   • To grant read permission to multiple files you must grant access at the bucket level. Specifically, you must add the above email as a principal to your storage bucket and grant it the IAM role of Storage Object Viewer.

4. If you configure the feed to delete source files (see below for how to do this), you must add the above email as a principal on your bucket and grant it the IAM role of Storage Object Admin.

Delete Source Files

As mentioned in the Overview section, different types of feeds have different prerequisites.

For several feed types, including Cloud Storage, there is a field in the Add new or Edit feed workflow labeled SOURCE DELETION OPTION. It is a drop-down menu with three options:

1. Never delete files
2. Delete transferred files and empty directories
3. Delete transferred files

Options 2 and 3 involve deletions: one for files and one for files AND any empty directories. If you select either of those options, you need to add the permissions specific to your feed type, which can be found in the Feed configuration by type section of the Feed management API documentation.

This option allows you to delete an object out of the storage system after you have transferred it. Feeds always remember which objects (or files) they have transferred and never transfer the same file twice (unless it has been updated), but you have to set this option if you want the system to actually delete the source object after it has been (successfully) transferred.

Creating and Editing Feeds

To access the Feed Management interface, select the Settings option from the main menu.

Figure 1. Settings

You can then navigate to the Feeds page. The data feeds listed on this page include all the feeds Google has configured for your account in addition to the feeds you have configured.

Figure 2. Feeds

Add a feed

To add a feed to your Chronicle account, complete the following steps. Be sure to complete the prerequisites for the data feed type you plan to add prior to attempting to add a new feed here. See the Overview and prerequisites section for more information.

You can add up to five feeds for each log type.

1. Click ADD NEW. The ADD FEED window is displayed.

2. Starting from the Set Properties tab, select the SOURCE TYPE from the drop down menu. The source type is the mechanism by which you intend to bring data into Chronicle. You can select from the following feed source types:

   • Amazon S3
   • Amazon SQS
   • Google Cloud Storage
   • HTTP(S) Files (non-API)
   • Microsoft Azure Blob Storage
   • Third party API

Note: For Amazon SQS, make sure you enable the permissions for Chronicle to be able to delete the Amazon SQS queue messages.

1. Select the LOG TYPE from the drop-down menu. The logs available vary depending on which source type you selected previously. Click NEXT.

   

   Figure 3. Log Type Selection

2. Specify the parameters needed from the Input Parameters tab. The options presented here vary depending on the source and log type selected on the Set Properties tab. Hover over the question icon above each field to get additional information on what you need to provide.

3. (Optional) You can specify a namespace here. For more information about namespaces, see the asset namespace documentation.

4. Click NEXT.

   

   Figure 4. Input Parameter Configuration

5. Review your new feed configuration from the Finalize tab. Click SUBMIT when you are ready. Chronicle completes a validation check of the new feed. If the feed passes the check, a name is generated for the feed, it is submitted to Chronicle, and Chronicle begins to attempt to fetch data.

   

   Figure 5. Finalize Feed Request

Feed status

You can monitor the status of the feed from the initial Feeds page. Feeds can have the following statuses:

• Active—Feed is configured and ready to ingest data into your Chronicle account.
• InProgress—Chronicle is now attempting to pull data from the configured third party.
• Completed—Data successfully retrieved by this feed.
• Archived—Disabled feed.
• Failed—Feed is failing to successfully fetch data. This is likely due to a configuration issue. Click the question to display the configuration error. Once you have corrected the error and resubmitted the feed, return to the Feeds page to determine whether or not the feed is now working.

Edit feeds

From the Feeds page, you can edit an existing feed:

1. Hover over an existing feed and click the three dot menu in the right column.

2. Click Edit Feed. You can now alter the Input Parameters for the Feed and resubmit it to Chronicle. Chronicle will attempt to use the edited feed.

Enable and disable feeds

In the Status column, enabled feeds are labeled as Active, InProgress, Completed, or Failed. Disabled fields are labeled as Archived. See here for a description of each feed status.

From the Feeds page, you can enable or disable any of the existing feeds:

1. Hover over an existing feed and click the three dot menu in the right column.

2. To enable a feed, click the Enable Feed toggle.

3. To disable a feed, click the Disable Feed toggle. The feed is now labeled as Archived.

Delete feeds

From the Feeds page, you can also delete an existing feed:

1. Hover over an existing feed and click the three dot menu in the right column.

2. Click Delete Feed. The DELETE FEED window opens. To permanently delete the feed, click Yes, Delete It.