Stay organized with collections Save and categorize content based on your preferences.

Feed management user guide

This guide explains how to create and manage data feeds using the Chronicle user interface (UI). For details about how to programmatically create and manage feeds, see the Feed Management API.

About data feeds

You can use Chronicle data feeds to ingest log data into your Chronicle instance from the following:

  • Chronicle-supported cloud storage services, such as Google Cloud Storage
  • Third-party data sources with a Chronicle-supported API, such as Microsoft 365
  • Files accessible directly using HTTP(S) requests

Each feed that you create is composed of a data source type and a log type. Google Cloud Storage, third-party APIs, and HTTP-accessible files are examples of source types. For each data source type that Chronicle supports, Chronicle also supports specific log types. For example, for the Google Cloud Storage source type, Chronicle supports the Carbon Black log type and many others. The list of supported log types varies by source type.

When you create a feed, you specify the source type, log type, required permissions, authentication details, and more.

If Chronicle provides a default parser for the log type, then the ingested log data is stored in both Chronicle Unified Data Model (UDM) format and raw log format.

Supported source types and log types

Chronicle supports the following source types:

Feed source type Description
Third-party API Ingest data from a third-party API.
Google Cloud Storage Ingest data from a Google Cloud Storage bucket.
Amazon S3 Ingest data from an Amazon Simple Storage Service bucket.
Amazon SQS Ingest data from an Amazon Simple Queue Service queue whose entries point to files stored in S3
Azure Blobstore Ingest data from Azure Blob Storage.
HTTP(S) Ingest data from files accessible by an HTTP(S) request. Do not use this source type to interact with third-party APIs. Use the API feed source type for third-party APIs supported by Chronicle.

There are several ways to view a list of currently supported log types:

  • Chronicle UI. To view the list of supported log types for each source type, go to Settings > Feeds > Add New and use the Source Type and Log Type menus. For details, see Creating and editing feeds .

  • API reference documentation. To view a list of supported log types for third-party API feeds, see Configuration by log type .

  • Feed Schema API. To view log types for any source type, you can also use the Feed Schema API .

Prerequisites

Each data feed has its own set of prerequisites that must be completed prior to setting up the feed in Chronicle. You can find the prerequisites as follows:

  • Prerequisites for each source type are listed in Configuration by source type .

  • Prerequisites for each log type ingested using the API feed source type are listed in Configuration by log type .

  • Prerequisites for all log types ingested using any source type are listed in the Chronicle UI. Go to Settings > Feeds > Add New, select a Source Type and Log Type, and review the required fields. For details, see Creating and editing feeds .

For example, if you set up a data feed from a Google Cloud Storage bucket, you would need to complete the following prerequisites:

You must first grant Chronicle access. You must add the email address 8911409095528497-0-account@partnercontent.gserviceaccount.com to the permissions of the relevant Google Cloud Storage objects. You must also perform the following actions from the Cloud Storage section in the Google Cloud console.

  • To grant read permission to a specific file, you can edit access on that file and grant the above email Reader access. This can only be done if you have not enabled uniform bucket-level access.
  • To grant read permission to multiple files, you must grant access at the bucket level. Specifically, you must add the email address 8911409095528497-0-account@partnercontent.gserviceaccount.com as a principal to your storage bucket and grant it the IAM role of Storage Object Viewer.
  • If you configure the feed to delete source files, you must add the email address as a principal on your bucket and grant it the IAM role of Storage Object Admin.

Creating and editing feeds

To access the Feed Management interface, select the Settings option from the main menu.

Settings

Figure 1. Settings

You can then navigate to the Feeds page. The data feeds listed on this page include all the feeds Google has configured for your account in addition to the feeds you have configured.

Feeds

Figure 2. Feeds

Add a feed

To add a feed to your Chronicle account, complete the following steps. Be sure to complete the prerequisites for the data feed type you plan to add prior to attempting to add a new feed here. See the Prerequisites section for more information.

  1. Click ADD NEW. The ADD FEED window is displayed.

  2. Select Directory type under the URI IS A dropdown menu. If it is a single file, select File. If you have multiple files, select Directory. In the case that files are stored in a directory structure under multiple directories, select Directory which include subdirectories. Amazon S3 or Cloud Storage bucket is considered as a directory.

  3. Starting from the Set Properties tab, select the SOURCE TYPE from the drop down menu. The source type is the mechanism by which you intend to bring data into Chronicle. You can select from the following feed source types:

    • Amazon S3
    • Amazon SQS
    • Google Cloud Storage
    • HTTP(S) Files (non-API)
    • Microsoft Azure Blob Storage
    • Third party API

    For Amazon SQS, make sure you enable the permissions for Chronicle to be able to delete the Amazon SQS queue messages.

  4. Select the LOG TYPE from the drop down menu. The logs available vary depending on which source type you selected previously. Click NEXT.

    Log type selection

    Figure 3. Log Type Selection

  5. Specify the parameters needed from the Input Parameters tab. The options presented here vary depending on the source and log type selected on the Set Properties tab. Hover over the question icon above each field to get additional information on what you need to provide.

  6. Click NEXT.

    Input parameter configuration

    Figure 4. Input Parameter Configuration

  7. Review your new feed configuration from the Finalize tab. Click SUBMIT when you are ready. Chronicle completes a validation check of the new feed. If the feed passes the check, a name is generated for the feed, it is submitted to Chronicle, and Chronicle begins to attempt to fetch data.

    Finalize feed request

    Figure 5. Finalize Feed Request

Feed status

You can monitor the status of the feed from the initial Feeds page. Feeds can have the following statuses:

  • Active—Feed is configured and ready to ingest data into your Chronicle account.
  • InProgress—Chronicle is now attempting to pull data from the configured third party.
  • Completed—Data successfully retrieved by this feed.
  • Archived—Disabled feed.
  • Failed—Feed is failing to successfully fetch data. This is likely due to a configuration issue. Click the question to display the configuration error. Once you have corrected the error and resubmitted the feed, return to the Feeds page to determine whether or not the feed is now working.

Edit feeds

From the Feeds page, you can edit an existing feed:

  1. Hover over an existing feed and click the three dot menu in the right column.

  2. Click Edit Feed. You can now alter the Input Parameters for the Feed and resubmit it to Chronicle. Chronicle will attempt to use the edited feed.

Enable and disable feeds

In the Status column, enabled feeds are labeled as Active, InProgress, Completed, or Failed. Disabled fields are labeled as Archived. See here for a description of each feed status.

From the Feeds page, you can enable or disable any of the existing feeds:

  1. Hover over an existing feed and click the three dot menu in the right column.

  2. To enable a feed, click the Enable Feed toggle.

  3. To disable a feed, click the Disable Feed toggle. The feed is now labeled as Archived.

Delete feeds

From the Feeds page, you can also delete an existing feed:

  1. Hover over an existing feed and click the three dot menu in the right column.

  2. Click Delete Feed. The DELETE FEED window opens. To permanently delete the feed, click Yes, Delete It.