Feed management API
This reference guide explains how to use the Google Security Operations feed management API to programmatically create, run, and manage data feeds that send logs to your Google Security Operations instance. For details about how to use the Google Security Operations UI to create and manage feeds, see the Feed management user guide.
Prerequisites
Each data feed has its own set of prerequisites that must be completed prior to setting up the feed in Google Security Operations. You can find the prerequisites as follows:
Prerequisites for each source type are listed in Configuration by source type .
Prerequisites for each log type ingested using the
API
feed source type are listed in Configuration by log type .Prerequisites for all log types ingested using any source type are listed in the Google Security Operations UI. Go to Settings > Feeds > Add New, select a Source Type and Log Type, and review the required fields. For details, see Creating and editing feeds .
For example, if you set up a data feed from a Google Cloud Storage bucket, you might need to complete the following tasks:
- Use the feed management
fetchFeedServiceAccount
method to get a Google Security Operations service account that Google Security Operations uses to ingest data. - Grant access to the Google Security Operations service account to the relevant Cloud Storage objects. For more information, see Grant access to the Google Security Operations service account.
How to authenticate with the Google Security Operations API
This Google Security Operations API uses the OAuth 2.0 protocol for authentication and authorization. Your application can complete these tasks using either of the following implementations:
Using the Google API Client Library for your computer language.
Directly interfacing with the OAuth 2.0 system using HTTP.
See the reference documentation for the Google Authentication library in Python.
Google Authentication libraries are a subset of the Google API client libraries. See other language implementations.
Getting API authentication credentials
Your Google Security Operations representative will provide you with a Google Developer Service Account Credential to enable the API client to communicate with the API.
You also must provide the Auth Scope when initializing your API client. OAuth 2.0 uses a scope to limit an application's access to an account. When an application requests a scope, the access token issued to the application is limited to the scope granted.
Use the following scope to initialize your Google API client:
https://www.googleapis.com/auth/chronicle-backstory
Python example
The following Python example demonstrates how to use the OAuth2 credentials
and HTTP client using google.oauth2
and googleapiclient
.
# Imports required for the sample - Google Auth and API Client Library Imports.
# Get these packages from https://pypi.org/project/google-api-python-client/ or run $ pip
# install google-api-python-client from your terminal
from google.auth.transport import requests
from google.oauth2 import service_account
SCOPES = ['https://www.googleapis.com/auth/chronicle-backstory']
# The apikeys-demo.json file contains the customer's OAuth 2 credentials.
# SERVICE_ACCOUNT_FILE is the full path to the apikeys-demo.json file
# ToDo: Replace this with the full path to your OAuth2 credentials
SERVICE_ACCOUNT_FILE = '/customer-keys/apikeys-demo.json'
# Create a credential using Google Developer Service Account Credential and Google Security Operations API
# Scope.
credentials = service_account.Credentials.from_service_account_file(SERVICE_ACCOUNT_FILE, scopes=SCOPES)
# Build a requests Session Object to make authorized OAuth requests.
http_session = requests.AuthorizedSession(credentials)
# Your endpoint GET|POST|PATCH|etc. code will vary below
# Reference List example (for US region)
url = 'https://backstory.googleapis.com/v2/lists/COLDRIVER_SHA256'
# You might need another regional endpoint for your API call; see
# https://cloud.google.com/chronicle/docs/reference/ingestion-api#regional_endpoints
# requests GET example
response = http_session.request("GET", url)
# POST example uses json
body = {
"foo": "bar"
}
response = http_session.request("POST", url, json=body)
# PATCH example uses params and json
params = {
"foo": "bar"
}
response = http_session.request("PATCH", url, params=params, json=body)
# For more complete examples, see:
# https://github.com/chronicle/api-samples-python/
Chronicle API query limits
The Chronicle API enforces limits on the volume of requests that can be made by any one customer against the Google Security Operations platform. If you reach or exceed the query limit, the Chronicle API server returns HTTP 429 (RESOURCE_EXHAUSTED) to the caller. When developing applications for the Chronicle API, Google Security Operations recommends that you enforce rate limits within your system to avoid resource exhaustion. These limits apply to all of the Chronicle APIs, including the feed management API.
The feed management API enforces the following limits, which are measured in queries per second (QPS):
Chronicle API | API Method | Limit |
Feed management | Create Feed | 1 QPS |
Get Feed | 1 QPS | |
List Feeds | 1 QPS | |
Update Feed | 1 QPS | |
Delete Feed | 1 QPS |
Control the rate of ingestion
When the data ingestion rate for a tenant reaches a certain threshold, Google Security Operations restricts the rate of ingestion for new data feeds to prevent a source with a high ingestion rate from affecting the ingestion rate of another data source. In this case, there is a delay but no data is lost. The ingestion volume and tenant's usage history determine the threshold.
You can request a rate limit increase by contacting Cloud Customer Care.
Limitations
Data feeds have a maximum log line size of 4 MB.
See the detailed list of the Chronicle API query limits.
Python example using OAuth2 credentials and HTTP client
The following Python example demonstrates how to use the OAuth2 credentials
and the HTTP client using google.oauth2
and googleapiclient
.
# Imports required for the sample - Google Auth and API Client Library Imports.
# Get these packages from https://pypi.org/project/google-api-python-client/ or
# run $ pip install google-api-python-client from your terminal
from google.auth.transport import requests
from google.oauth2 import service_account
SCOPES = ['https://www.googleapis.com/auth/chronicle-backstory']
# The apikeys-demo.json file contains the customer's OAuth 2 credentials.
# SERVICE_ACCOUNT_FILE is the full path to the apikeys-demo.json file
# ToDo: Replace this with the full path to your OAuth2 credentials
SERVICE_ACCOUNT_FILE = '/customer-keys/apikeys-demo.json'
# Create a credential using Google Developer Service Account Credential and
Chronicle API Scope.
credentials = service_account.Credentials.from_service_account_file(SERVICE_ACCOUNT_FILE, scopes=SCOPES)
# Build an HTTP session to make authorized OAuth requests.
http_session = requests.AuthorizedSession(credentials)
# <your code continues here>
Regional endpoints
Google Security Operations provides regional endpoints for each API.
- São Paulo—
https://southamerica-east1-backstory.googleapis.com
- Canada—
https://northamerica-northeast2-backstory.googleapis.com
- Dammam—
https://me-central2-backstory.googleapis.com
- Doha—
https://me-central1-backstory.googleapis.com
- Europe Multi-Region—
https://europe-backstory.googleapis.com
- Frankfurt—
https://europe-west3-backstory.googleapis.com
- London—
https://europe-west2-backstory.googleapis.com
- Mumbai—
https://asia-south1-backstory.googleapis.com
- Paris—
https://europe-west9-backstory.googleapis.com
- Singapore—
https://asia-southeast1-backstory.googleapis.com
- Sydney—
https://australia-southeast1-backstory.googleapis.com
- Tel Aviv—
https://me-west1-backstory.googleapis.com
- Tokyo—
https://asia-northeast1-backstory.googleapis.com
- Turin—
https://europe-west12-backstory.googleapis.com
- United States Multi-Region—
https://backstory.googleapis.com
- Zurich—
https://europe-west6-backstory.googleapis.com
Feed Schema API reference
The Feed Schema API returns information that is useful for constructing valid feed management API requests. For example, you can get the data structure representing the entire feed schema. This structure defines the specific fields to specify for each valid combination of feed source type and log type. Alternatively, you can get a list of all log types compatible with a particular feed source type.
Specifically, the feed schema contains:
- Information about each valid feed source type:
- A human-readable name
- A human-readable description
- Whether feeds with a given feed source type can be modified using the API, or are read-only
- Information about each log type:
- A human-readable name
- Whether feeds with a given log type can be modified using the API, or are read-only
- Which log types are compatible with which feed source types
- Information about the specific fields to specify for each valid combination
of log type and feed source type:
- A human-readable field name and description
- Compatibility with other fields
- Semantic type (e.g. URI, "secret", etc)
- Whether the field is required
- What a valid value for the field looks like
The schema can be accessed using a few different methods.
GetFeedSchema
This method returns a structure representing the entire feed schema. The schema is comprised of a list of "feed source type schemas" each of which describe the supported feed source types. Each feed source types' schema contains a list of "log type schemas" that correspond to the set of log types that are compatible with the feed source type and describe the log type. Each log type schema contains a list of "details field schemas" that describe those fields you would set when issuing a Create Feed request, for example, or expect to see as a result of a ListFeed or GetFeed response. The field schemas specified are unique to the particular combination of log type and feed source type.
Request
GET https://backstory.googleapis.com/v1/feedSchema
Response
{
"feedSourceTypeSchemas": [{
"name": "feedSourceTypeSchemas/AMAZON_S3",
"displayName": "Amazon S3",
"description": "Amazon Simple Storage Service, a service offered by Amazon Web Services that provides object storage through a web service interface",
"feedSourceType": "AMAZON_S3",
"logTypeSchemas": [{
"name": "feedSourceTypeSchemas/AMAZON_S3/logTypeSchemas/AWS_CLOUDTRAIL",
"displayName": "AWS Cloudtrail",
"logType": "AWS_CLOUDTRAIL",
"detailsFieldSchemas": [{
"fieldPath": "details.amazon_s3_settings.authentication.access_key_id",
"displayName": "Access key ID",
"description": "An account access key that is a 20-character alphanumeric string, for example AKIAIOSFODNN7EXAMPLE",
"type": "STRING",
"exampleInput": "AKIAIOSFODNN7EXAMPLE",
},
...
{
"fieldPath": "details.amazon_s3_settings.s3_uri",
"displayName": "S3 URI",
"description": "The S3 bucket source URI",
"type": "STRING_URI",
"isRequired": true,
"exampleInput": "s3://cs-prod-cannon-00afe0c847a8/data/",
}],
},
...
{
"name": "feedSourceTypeSchemas/AMAZON_S3/logTypeSchemas/ABNORMAL_SECURITY",
"displayName": "Abnormal Security",
"logType": "ABNORMAL_SECURITY",
...
}],
},
...
{
"name": "feedSourceTypeSchemas/AMAZON_SQS",
"displayName": "Amazon SQS",
"description": "Amazon Simple Queue Service, a service offered by Amazon Web Services that provides fully managed message queuing service to transfer messages asynchronously",
"feedSourceType": "AMAZON_SQS",
...
}],
}
ListFeedSourceTypeSchemas
This method returns information about all feed source types.
Request
GET https://backstory.googleapis.com/v1/feedSourceTypeSchemas
Sample Response
{
"feedSourceTypeSchemas": [{
"name": "feedSourceTypeSchemas/AMAZON_S3",
"displayName": "Amazon S3",
"description": "Amazon Simple Storage Service, a service offered by Amazon Web Services that provides object storage through a web service interface",
"feedSourceType": "AMAZON_S3",
},
...
{
"name": "feedSourceTypeSchemas/AMAZON_SQS",
"displayName": "Amazon SQS",
"description": "Amazon Simple Queue Service, a service offered by Amazon Web Services that provides fully managed message queuing service to transfer messages asynchronously",
"feedSourceType": "AMAZON_SQS",
}],
}
ListLogTypeSchemas
This method returns information about all log types compatible with a particular feed source type.
Request
GET https://backstory.googleapis.com/v1/feedSourceTypeSchemas/{feed source type}/logTypeSchemas
Sample Request
https://backstory.googleapis.com/v1/feedSourceTypeSchemas/AMAZON_S3/logTypeSchemas
Sample Response
{
"logTypeSchemas": [{
"name": "feedSourceTypeSchemas/AMAZON_S3/logTypeSchemas/AWS_CLOUDTRAIL",
"displayName": "AWS Cloudtrail",
"logType": "AWS_CLOUDTRAIL",
},
...
{
"name": "feedSourceTypeSchemas/AMAZON_S3/logTypeSchemas/ABNORMAL_SECURITY",
"displayName": "Abnormal Security",
"logType": "ABNORMAL_SECURITY",
...
}],
}
GetLogTypeSchema
This method returns detailed information about all the fields necessary to configure a feed for a particular source type and log type.
Request
GET https://backstory.googleapis.com/v1/feedSourceTypeSchemas/{feed source type}/logTypeSchemas/{log type}
Sample Request
https://backstory.googleapis.com/v1/feedSourceTypeSchemas/AMAZON_S3/logTypeSchemas/AWS_CLOUDTRAIL
Sample Response
{
"name": "feedSourceTypeSchemas/AMAZON_S3/logTypeSchemas/AWS_CLOUDTRAIL",
"displayName": "AWS Cloudtrail",
"logType": "AWS_CLOUDTRAIL",
"detailsFieldSchemas": [{
"fieldPath": "details.amazon_s3_settings.authentication.access_key_id",
"displayName": "Access key ID",
"description": "An account access key that is a 20-character alphanumeric string, for example AKIAIOSFODNN7EXAMPLE",
"type": "STRING",
"exampleInput": "AKIAIOSFODNN7EXAMPLE",
},
...
{
"fieldPath": "details.amazon_s3_settings.s3_uri",
"displayName": "S3 URI",
"description": "The S3 bucket source URI",
"type": "STRING_URI",
"isRequired": true,
"exampleInput": "s3://cs-prod-cannon-01abc2d345e6/data/",
}],
}
Feed management API reference
This section describes the endpoints for creating, enabling, and managing feeds.
When creating or editing a feed, you need to specify the feedSourceType
and logType
in the request body. For details about these fields, see Configuration by source type and Configuration by log type.
Create Feed
Creates a third party data feed in your Google Security Operations instance.
Request
POST https://backstory.googleapis.com/v1/feeds
Request body
This example shows how to collect authentication logs from Duo Security.
{
"display_name": "some feed name",
"details": {
"feedSourceType": "API",
"logType": "DUO_AUTH",
"duoAuthSettings": {
"authentication": {
"user": "ABCUSERNAMEDEF",
"secret": "aBcS3cReTdEf"
},
"hostname": "api-abc123.duosecurity.com"
},
"namespace": "my-asset-namespace",
"labels": [{
"key": "my-ingestion-label-key",
"value": "my-ingestion-label-value"
}]
}
}
Sample request
https://backstory.googleapis.com/v1/feeds
{
"display_name": "some feed name",
"details": {
"feedSourceType": "API",
"logType": "DUO_AUTH",
"duoAuthSettings": {
"authentication": {
"user": "ABCUSERNAMEDEF",
"secret": "aBcS3cReTdEf"
},
"hostname": "api-abc123.duosecurity.com"
}
}
}
Sample successful response
{
"name": "feeds/12a34567-bc8d-1111-e9f0-gh1ijk234567",
"display_name": "some feed name",
"details": {
"logType": "DUO_AUTH",
"feedSourceType": "API",
"duoAuthSettings": {
"hostname": "api-abc123.duosecurity.com"
}
},
"feedState": "ACTIVE"
}
If the response is unsuccessful, it returns an HTTP status code other than 200 (OK). Be sure to check the body of the response for details of the failure.
Asset namespace
To assign an
asset namespace
to all events that are ingested from a particular feed, set the "namespace"
field within details
. The namespace
field is a string.
Ingestion label
Ingestion labels are part of
Unified Data Model metadata.
They are repeated key and value pairs. To assign ingestion labels to all events
that are ingested from a particular feed, set the labels
field within
details
. The labels
field is an array of JSON objects with key
and value
fields.
DeleteFeed
Deletes a feed that was configured using the Google Security Operations feed management API.
Request
DELETE https://backstory.googleapis.com/v1/feeds/{feedID}
Sample request
DELETE https://backstory.googleapis.com/v1/feeds/12a34567-bc8d-1111-e9f0-gh1ijk234567
Sample response
If the operation is successful, Delete Feed returns an empty response with an HTTP status code 200 (OK).
{}
Enable Feed
Enables an INACTIVE
feed, which allows it to be executed.
Request
POST https://backstory.googleapis.com/v1/feeds/{feedID}:enable
Sample request
POST https://backstory.googleapis.com/v1/feeds/12a34567-bc8d-1111-e9f0-gh1ijk234567:enable
Sample response
{
"name": "feeds/12a34567-bc8d-1111-e9f0-gh1ijk234567",
"display_name": "some feed name",
"details": {
"logType": "DUO_AUTH",
"feedSourceType": "API",
"duoAuthSettings": {
"hostname": "api-abc123.duosecurity.com"
}
},
"feedState": "ACTIVE"
}
Disable Feed
Disables a feed. A disabled feed has a status of INACTIVE
. Disabled feeds will
no longer fetch data.
Request
POST https://backstory.googleapis.com/v1/feeds/{feedID}:disable
Sample request
POST https://backstory.googleapis.com/v1/feeds/12a34567-bc8d-1111-e9f0-gh1ijk234567:disable
Sample response
{
"name": "feeds/12a34567-bc8d-1111-e9f0-gh1ijk234567",
"display_name": "some feed name",
"details": {
"logType": "DUO_AUTH",
"feedSourceType": "API",
"duoAuthSettings": {
"hostname": "api-abc123.duosecurity.com"
}
},
"feedState": "INACTIVE"
}
Get Feed
Gets the details of the feed that was configured.
Request
GET https://backstory.googleapis.com/v1/feeds/{feedID}
Sample request
https://backstory.googleapis.com/v1/feeds/12a34567-bc8d-1111-e9f0-gh1ijk234567
Sample response
{
"name": "feeds/12a34567-bc8d-1111-e9f0-gh1ijk234567",
"display_name": "some feed name",
"details": {
"logType": "DUO_AUTH",
"feedSourceType": "API",
"duoAuthSettings": {
"hostname": "api-abc123.duosecurity.com"
}
},
"feedState": "FAILED",
"last_feed_initiation_time": "2024-01-15T01:30:15.01Z",
"failure_details": {
"error_code": "INVALID_ARGUMENT"
"http_error_code": 400,
"error_cause": "A connection to the source was established, but the feed failed because of invalid arguments",
"error_action":"Check the feed configuration. Learn more about setting up the feeds.\nIf the problem continues, contact Chronicle Support"
}
}
List Feeds
Retrieves all the feeds configured for a given Google Security Operations instance.
Request
GET https://backstory.googleapis.com/v1/feeds
Sample request
https://backstory.googleapis.com/v1/feeds
Sample response
{
"feeds": [
{
"name": "feeds/12a34567-bc8d-1111-e9f0-gh1ijk234567",
"details": {
"logType": "AZURE_AD_CONTEXT",
"feedSourceType": "API",
"azureAdContextSettings": {}
},
"feedState": "FAILED",
"last_feed_initiation_time": "2024-01-15T01:30:15.01Z",
"failure_details": {
"error_code": "INVALID_ARGUMENT"
"http_error_code": 400,
"error_cause": "A connection to the source was established, but the feed failed because of invalid arguments",
"error_action":"Check the feed configuration. Learn more about setting up the feeds.\nIf the problem continues, contact Chronicle Support"
}
},
{
"name": "feeds/12a34567-bc8d-1111-e9f0-gh1ijk234567",
"display_name": "some feed name",
"details": {
"logType": "PAN_PRISMA_CLOUD",
"feedSourceType": "API",
"panPrismaCloudSettings": {
"hostname": "api2.prismacloud.io"
}
},
"feedState": "ACTIVE",
"last_feed_initiation_time": "2024-01-15T01:30:15.01Z",
}
]
}
Read-only feeds
There may be feeds returned from a List Feeds request that have the field
readOnly
set to true
. Read-only feeds cannot be created, updated, or
deleted.
Feeds are read-only for a few reasons. For example:
- Some feed source types are not fully supported by feed management at the moment, and were created before the release of feed management.
- Some specialized log types are not available to every Google Security Operations user. If a feed exists with one of these types, it is considered read-only.
Update Feed
Updates the given feed with new details.
Request
PATCH https://backstory.googleapis.com/v1/feeds/{feedID}
Request body
The following examples shows how to update a Duo Auth feed.
Sample request
{
"display_name": "my feed",
"details": {
"feedSourceType": "API",
"logType": "DUO_AUTH",
"duoAuthSettings": {
"authentication": {
"user": "ABCUSERNAMEDEF",
"secret": "aBcS3cReTdEf"
},
"hostname": "api-abc123.duosecurity.com"
}
}
}
Sample response
{
"display_name": "my feed",
"name": "feeds/12a34567-bc8d-1111-e9f0-gh1ijk234567",
"details": {
"logType": "DUO_AUTH",
"feedSourceType": "API",
"duoAuthSettings": {
"hostname": "api-abc123.duosecurity.com"
}
},
"feedState": "ACTIVE"
}
Fetch service account
Gets a unique service account that Google Security Operations uses to ingest data. Use this method only if you're setting up a Cloud Storage feed.
Request
GET https://backstory.googleapis.com/v1/fetchFeedServiceAccount
Sample request
GET https://backstory.googleapis.com/v1/fetchFeedServiceAccount
Sample response
"serviceAccount": "xxxxxxxx-0-account@partnercontent.gserviceyesaccount.com"
Response message fields
This section describes the following fields that are returned in response messages:
feedState
failureMsg
Feed state
The feedState
field can be found in the response message of most operations.
feedState
gives some insight into the current state of a feed.
feedState |
Description |
---|---|
"ACTIVE" |
Feed successfully created and will begin fetching data. |
"INACTIVE" |
Feed has been disabled. |
"IN_PROGRESS"
|
Feed is attempting to fetch data. A feed will only have this status if it has not previously failed. |
"COMPLETED" |
Feed has recently fetched data successfully. |
"FAILED"
|
Feed has failed and has not successfully fetched data
since it failed. Mis-configuration is the typical cause of
feed failure. Please see the failureMsg
field for more information. |
Failure message
The failureMsg
field can be found in the response message of most
operations, but only for those feeds whose feedState
is FAILED
. It
provides information regarding the error code, cause of the error, and how
to troubleshoot the error. For information about error messages, see
Troubleshooting.
Refer to the following documentation for your particular feed type to
understand how to correctly configure the feed.
Generate secret key and API key to authenticate the feed
You need to generate the secret key and API key to authenticate the feed when you set up a feed that has webhook or Amazon Data Firehose as the source type. You can reuse your existing API key to authenticate to Google Security Operations. You must generate a secret key for every new feed and can't reuse the secret key.
To generate a secret key for a webhook or Amazon Data Firehose feed, run the following
curl
command that uses thegenerateSecret
Chronicle API.curl --location --request POST -H "Authorization: Bearer $(gcloud auth print-access-token)" 'https://REGIONAL_ENDPOINT/v1alpha/projects/PROJECT_NUMBER/locations/REGION_ID/instances/CUSTOMER_ID/feeds/FEED_ID:generateSecret
Replace the following:
REGIONAL_ENDPOINT
: the Google Security Operations regional endpoint, such asus-chronicle.googleapis.com
. For information about supported regional endpoints, see Google Security Operations regional endpoints section of this document.PROJECT_NUMBER
: an automatically generated unique identifier for your project. For information about project name, project ID, and project number, which are used to identify a project, see Creating and managing projects.REGION_ID
: the code that Google assigns based on the region. The following are the supported region IDs:Asia-Southeast1
,Australia-Southeast1
,Europe
,EU
,Europe-West2
,Europe-West3
,Europe-West6
,Govcloud-US
,Me-West1
, andUS
.CUSTOMER_ID
: the Google Security Operations customer ID.FEED_ID
: the Google Security Operations feed ID.
A secret key is returned. Copy and store the secret key because you cannot view this secret again. You can use the
generateSecret
API again to generate a new secret key, but regeneration of the secret key makes the previous secret key obsolete.To generate the API key, do the following:
- Go to the Google Cloud console Credentials page.
- Click Create credentials, and then select API key.
- Restrict the API key access to the Chronicle API.
Configuration by source type
This section provides information about configuring feed source types.
A feed source type defines where data is located and how it's accessed. Valid values for feedSourceType
are as follows:
feedSourceType |
Description |
---|---|
API |
Ingest data from a third-party API. |
HTTPS_PUSH_GOOGLE_CLOUD_PUBSUB |
Ingest data using a Pub/Sub push subscription. |
GOOGLE_CLOUD_STORAGE |
Ingest data from a Cloud Storage bucket. |
HTTPS_PUSH_AMAZON_KINESIS_FIREHOSE |
Ingest data from Amazon Data Firehose. |
AMAZON_S3 |
Ingest data from an Amazon Simple Storage Service bucket. |
AMAZON_SQS |
Ingest data from an Amazon Simple Queue Service queue whose entries point to files stored in S3 |
AZURE_BLOBSTORE |
Ingest data from Azure Blob Storage. |
HTTP |
Ingest data from files accessible by an HTTP(S) request. Note that this *should not* be used to interact with third-party APIs. Use the API feed source type for third-party APIs supported by Google Security Operations. |
HTTPS_PUSH_WEBHOOK |
Ingest data using an HTTPS webhook. |
API
Use the API
feed source type to ingest data from a third-party API. The
configuration settings for the API
feed source type are specific to the
log type that you specify for the feed.
For example, when you use the Feeds API endpoints to configure
settings for an API
feed data source, you specify API
as the value for
the details.feedSourceType
field. You also specify a supported log type
as the value for the details.logType
field. The log type determines the
additional feed configuration settings that you can specify. Details about
these settings are provided in the section for that specific log type in
Configuration by log type.
Google Cloud Pub/Sub
Data source | Ingest schedule | details.feedSourceType |
---|---|---|
Pub/Sub | Based on your implementation, it might take approximately a minute for a new feed to populate because Google Security Operations takes some time to process the data. | HTTPS_PUSH_GOOGLE_CLOUD_PUBSUB |
Prerequisites
- Ensure that a Google Cloud project for Google Security Operations is configured and the Chronicle API is enabled for the project.
- Link a Google Security Operations instance to Google Cloud services.
Set up push ingestion using Pub/Sub
Data can be sent to Google Security Operations using Pub/Sub. You must first create a feed with the appropriate log type before configuring Pub/Sub to send data.
To set up HTTPS push ingestion using Pub/Sub, do the following:
Create a Pub/Sub feed using the following
create
API request:{ "displayName": "FEED_NAME", "details": { "feedSourceType": "HTTPS_PUSH_GOOGLE_CLOUD_PUBSUB", "logType": "projects/PROJECT_ID/locations/REGION_ID/instances/CUSTOMER_ID/logTypes/LOG_TYPE" } }
Replace the following:
FEED_NAME
: specify a name for the feed.PROJECT_ID
: the project ID of the project that is bound to Google Security Operations.REGION_ID
: the region configured for your Google Security Operations instance. This was set when the tenant was provisioned. The following are the supported region IDs:Asia-Southeast1
,Australia-Southeast1
,Europe
,EU
,Europe-West2
,Europe-West3
,Europe-West6
,Govcloud-US
,Me-West1
, andUS
.CUSTOMER_ID
: the Google Security Operations customer ID.LOG_TYPE
: the type of data the feed ingests. Google Security Operations supports specific log types for the Pub/Sub feed.
If the data to be ingested contains a delimiter that separate log lines, such as
\\n
, include the following in thedetails
field of the feed request body:"httpsPushGoogleCLoudPubsubSettings": { "splitDelimiter": "LOG_DELIMITER" }
Replace
LOG_DELIMITER
with the delimiter that separates the log lines, such as\\n
.After you create a feed in Pub/Sub, create a push subscription, specify the HTTPS endpoint, and enable authentication. For more information about how to create a push subscription, see Create push subscriptions.
Specify the endpoint URL. The endpoint URL must have the following format:
https://REGIONAL_ENDPOINT/v1alpha/projects/PROJECT_ID/ locations/REGION_ID/instances/CUSTOMER_ID/feeds/FEED_ID:importPushLogs
REGIONAL_ENDPOINT
: the Google Security Operations regional endpoint, such asus-chronicle.googleapis.com
. For information about supported regional endpoints, see Google Security Operations regional endpoints section of this document.PROJECT_ID
: the project ID of the project that is bound to Google Security Operations.REGION_ID
: the region configured for your Google Security Operations instance. This was set when the tenant was provisioned. The following are the supported region IDs:Asia-Southeast1
,Australia-Southeast1
,Europe
,EU
,Europe-West2
,Europe-West3
,Europe-West6
,Govcloud-US
,Me-West1
, andUS
.CUSTOMER_ID
: the Google Security Operations customer ID.FEED_ID
: the Google Security Operations feed ID.
Select Enable authentication and select a service account.
GOOGLE_CLOUD_STORAGE
Data source | Ingest schedule | details.feedSourceType
|
details.logType
|
---|---|---|---|
Google Cloud Storage Bucket | Every 15 minutes | GOOGLE_CLOUD_STORAGE
|
See the Feed Schema API reference to get compatible log types. |
Prerequisites
Before you set up a Cloud Storage feed, you must get a Google Security Operations service account and provide access to the account so that Google Security Operations can ingest data.
- Use the feed management
fetchFeedServiceAccount
method to get a Google Security Operations service account. - Grant access to the Google Security Operations service account to the relevant Cloud Storage objects. For more information, see Grant access to the Google Security Operations service account.
- If VPC Service Controls is enabled, configure an ingress rule to provide access to the Cloud Storage bucket. In this document, see Configure VPC Service Controls.
Recommendation
If your Cloud Storage bucket contains many small files, it can increase the amount of time it takes to transfer the data. It is recommended to combine smaller files into one large file to accelerate the feed transfer process.
It is important to set a data retention policy to your Cloud Storage buckets so that the transferred files are deleted from the buckets and are not listed in any future transfer feeds. Alternatively, using the Google SecOps feed management UI, you can set the option to delete the source files from the storage buckets after they have been transferred.
As Google SecOps pulls files from Cloud Storage on a frequent basis, it is recommended to specify a storage class for the bucket that is the most cost-effective.
Configure VPC Service Controls
If VPC Service Controls is enabled, an ingress rule is required to provide access to the Cloud Storage bucket.
The following Cloud Storage methods must be allowed in the ingress rule:
google.storage.objects.list
. Required for a single file feed.google.storage.objects.get
. Required for feeds that require directory or subdirectory access.google.storage.objects.delete
. Required for feeds that require deletion of the source file.
Sample ingress rule
- ingressFrom:
identities:
- serviceAccount:8911409095528497-0-account@partnercontent.gserviceaccount.com
sources:
- accessLevel: "*"
ingressTo:
operations:
- serviceName: storage.googleapis.com
methodSelectors:
- method: google.storage.objects.list
- method: google.storage.objects.get
- method: google.storage.objects.delete
resources:
- projects/PROJECT_ID
Type-specific request fields
Field | Required | Description |
---|---|---|
details.gcsSettings.bucketUri |
Yes | The URI which corresponds to the Cloud Storage bucket. The format is the same format used by the gcloud storage command group to specify a resource. |
details.gcsSettings.sourceType |
Yes | The type of object indicated by bucketUri . Valid values are:
|
details.gcsSettings.sourceDeletionOption |
Yes | Whether to delete source files after they have been transferred to Google Security Operations. This reduces storage costs. Valid values are:
|
Sample Create Feed request
{
"details": {
"feedSourceType": "GOOGLE_CLOUD_STORAGE",
"logType": "LOGTYPE_YOU_WANT_TO_BRING",
"gcsSettings": {
"bucketUri": "gs://bucket/file",
"sourceType": "FOLDERS_RECURSIVE",
"sourceDeletionOption": "SOURCE_DELETION_NEVER"
}
}
}
Amazon Data Firehose
Data source | Ingest schedule | details.feedSourceType |
---|---|---|
Amazon Data Firehose | Based on your implementation, it might take approximately a minute for a new feed to populate because Google Security Operations takes some time to process the data. | HTTPS_PUSH_AMAZON_KINESIS_FIREHOSE |
Prerequisites
- Ensure that a Google Cloud project for Google Security Operations is configured and the Chronicle API is enabled for the project.
- Link a Google Security Operations instance to Google Cloud services.
Set up push ingestion using Amazon Data Firehose
Data can be sent to Google Security Operations using Amazon Data Firehose. You must first create a feed with the appropriate log type before configuring Amazon Data Firehose to send data.
To set up HTTPS push ingestion using Amazon Data Firehose, do the following:
Create an Amazon Data Firehose feed using the following
create
API request:{ "displayName": "FEED_NAME", "details": { "feedSourceType": "HTTPS_PUSH_AMAZON_KINESIS_FIREHOSE", "logType": "projects/PROJECT_NUMBER/locations/REGION_ID/instances/CUSTOMER_ID/logTypes/LOG_TYPE" } }
Replace the following:
FEED_NAME
: specify a name for the feed.PROJECT_NUMBER
: an automatically generated unique identifier for your project. For information about project name, project ID, and project number, which are used to identify a project, see Creating and managing projects.REGION_ID
: the region configured for your Google Security Operations instance. This was set when the tenant was provisioned. The following are the supported region IDs:Asia-Southeast1
,Australia-Southeast1
,Europe
,EU
,Europe-West2
,Europe-West3
,Europe-West6
,Govcloud-US
,Me-West1
, andUS
.CUSTOMER_ID
: the Google Security Operations customer ID.LOG_TYPE
: the type of data the feed ingests. Google Security Operations supports specific log types for the Amazon Data Firehose feed.
If the data to be ingested contains a delimiter that separate log lines, such as
\\n
, include the following in thedetails
field of the feed request body:"httpsPushAmazonKinesisFirehoseSettings": { "splitDelimiter": "LOG_DELIMITER" }
Replace
LOG_DELIMITER
with the delimiter that separates the log lines, such as\\n
.After you create a feed, generate a secret key for the feed and generate an API key to authenticate to Google Security Operations.
In Amazon Data Firehose, specify the HTTPS endpoint and access key.
Specify the endpoint URL. Here is a sample Amazon Data Firehose endpoint:
https://REGIONAL_ENDPOINT/v1alpha/projects/PROJECT_NUMBER/ locations/REGION_ID/instances/CUSTOMER_ID/feeds/FEED_ID%3AimportPushLogs?key=API_KEY
The endpoint includes the following values:
REGIONAL_ENDPOINT
: the Google Security Operations regional endpoint, such asus-chronicle.googleapis.com
. For information about supported regional endpoints, see Google Security Operations regional endpoints section of this document.PROJECT_NUMBER
: an automatically generated unique identifier for your project. For information about project name, project ID, and project number, which are used to identify a project, see Creating and managing projects.REGION_ID
: the region configured for your Google Security Operations instance. This was set when the tenant was provisioned. The following are the supported region IDs:Asia-Southeast1
,Australia-Southeast1
,Europe
,EU
,Europe-West2
,Europe-West3
,Europe-West6
,Govcloud-US
,Me-West1
, andUS
.CUSTOMER_ID
: the Google Security Operations customer ID.FEED_ID
: the Google Security Operations feed ID.API_KEY
: the API key value.
In the Access key field, specify the secret key that you obtained using the
generateSecret
API.
AMAZON_S3
Data source | Ingest schedule | details.feedSourceType
|
details.logType
|
---|---|---|---|
Amazon Simple Storage Service Bucket | Every 15 minutes | AMAZON_S3
|
See the Feed Schema API reference to get compatible log types. |
Prerequisites
- Create an S3 bucket.
- Create a security key for programmatic access.
To learn more about how to configure a feed to ingest data from an Amazon S3 bucket, see Ingest AWS logs.
Recommendation
If your S3 bucket contains many small files, it can increase the amount of time it takes to transfer the data. It is recommended to combine smaller files into one large file to accelerate the feed transfer process.
It is important to set a data retention policy to your S3 buckets so that the transferred files are deleted from the buckets and are not listed in any future transfer feeds. Alternatively, using the Google SecOps feed management UI, you can set the option to delete the source files from the storage buckets after they have been transferred.
As Google SecOps pulls files from S3 buckets on a frequent basis, it is recommended to specify a storage class for your buckets that is the most cost-effective.
Type-specific request fields
Field | Required | Description |
---|---|---|
details.amazonS3Settings.s3Uri |
Yes | The S3 URI to ingest. |
details.amazonS3Settings.sourceType |
Yes | The type of file indicated by the URI. Valid values are:
|
details.amazonS3Settings.sourceDeletionOption |
Yes | Whether to delete source files after they have been transferred to Google Security Operations. This reduces storage costs. Valid values are:
|
details.amazonS3Settings.authentication.region |
Yes | The region where the S3 bucket resides. For a list of regions, see Amazon S3 regions. |
details.amazonS3Settings.authentication.accessKeyId |
Yes | This is the 20 character ID associated with your Amazon IAM account. |
details.amazonS3Settings.authentication.secretAccessKey |
Yes | This is the 40 character access key associated with your Amazon IAM account. |
Sample Create Feed request
{
"details": {
"feedSourceType": "AMAZON_S3",
"logType": "LOGTYPE_YOU_WANT_TO_BRING",
"amazonS3Settings": {
"s3Uri": "s3://uri/to/file",
"sourceType": "FILES",
"sourceDeletionOption": "SOURCE_DELETION_NEVER",
"authentication": {
"region": "US_EAST_1",
"accessKeyId": "AKIAIOSFODNN7EXAMPLE",
"secretAccessKey": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
},
}
}
}
Amazon S3 regions
AWS Region | AWS Region code | authentication.region |
---|---|---|
Asia Pacific (Mumbai) | ap-south-1 | AP_SOUTH_1 |
Asia Pacific (Seoul) | ap-northeast-2 | AP_NORTHEAST_2 |
Asia Pacific (Singapore) | ap-southeast-1 | AP_SOUTHEAST_1 |
Asia Pacific (Sydney) | ap-southeast-2 | AP_SOUTHEAST_2 |
Asia Pacific (Tokyo) | ap-northeast-1 | AP_NORTHEAST_1 |
AWS GovCloud (US-East) | us-gov-east-1 | US_GOV_EAST_1 |
AWS GovCloud (US-West) | us-gov-west-1 | US_GOV_CLOUD |
Canada (Central) | ca-central-1 | CA_CENTRAL_1 |
China (Beijing) | cn-north-1 | CN_NORTH_1 |
China (Ningxia) | cn-northwest-1 | CN_NORTHWEST_1 |
Europe (Frankfurt) | eu-central-1 | EU_CENTRAL_1 |
Europe (Ireland) | eu-west-1 | EU_WEST_1 |
Europe (London) | eu-west-2 | EU_WEST_2 |
Europe (Paris) | eu-west-3 | EU_WEST_3 |
Europe (Stockholm) | eu-north-1 | EU_NORTH_1 |
South America (São Paulo) | sa-east-1 | SA_EAST_1 |
US East (N. Virginia) | us-east-1 | US_EAST_1 |
US East (Ohio) | us-east-2 | US_EAST_2 |
US West (N. California) | us-west-1 | US_WEST_1 |
US West (Oregon) | us-west-2 | US_WEST_2 |
AMAZON_SQS
Data source | details.feedSourceType |
details.logType |
---|---|---|
Amazon Simple Storage Service Bucket (S3) that sends notifications to an Amazon Simple Queue Service (SQS) | AMAZON_SQS
|
See the Feed Schema API reference to get compatible log types. |
You can ingest data from an Amazon SQS service whose entries point to files stored in an Amazon S3 bucket.
When you apply a policy, make sure that you have included sqs:DeleteMessage
.
Google Security Operations is not able to delete messages if the sqs:DeleteMessage
permission is not attached to the SQS queue. All the messages are accumulated on
the AWS side, which causes a delay as Google Security Operations repeatedly attempts to transfer the same files.
Amazon SQS regions
AWS Region | AWS Region code | authentication.region |
---|---|---|
Asia Pacific (Mumbai) | ap-south-1 | AP_SOUTH_1 |
Asia Pacific (Seoul) | ap-northeast-2 | AP_NORTHEAST_2 |
Asia Pacific (Singapore) | ap-southeast-1 | AP_SOUTHEAST_1 |
Asia Pacific (Sydney) | ap-southeast-2 | AP_SOUTHEAST_2 |
Asia Pacific (Tokyo) | ap-northeast-1 | AP_NORTHEAST_1 |
AWS GovCloud (US-East) | us-gov-east-1 | US_GOV_EAST_1 |
AWS GovCloud (US-West) | us-gov-west-1 | US_GOV_CLOUD |
Canada (Central) | ca-central-1 | CA_CENTRAL_1 |
China (Beijing) | cn-north-1 | CN_NORTH_1 |
China (Ningxia) | cn-northwest-1 | CN_NORTHWEST_1 |
Europe (Frankfurt) | eu-central-1 | EU_CENTRAL_1 |
Europe (Ireland) | eu-west-1 | EU_WEST_1 |
Europe (London) | eu-west-2 | EU_WEST_2 |
Europe (Paris) | eu-west-3 | EU_WEST_3 |
Europe (Stockholm) | eu-north-1 | EU_NORTH_1 |
South America (São Paulo) | sa-east-1 | SA_EAST_1 |
US East (N. Virginia) | us-east-1 | US_EAST_1 |
US East (Ohio) | us-east-2 | US_EAST_2 |
US West (N. California) | us-west-1 | US_WEST_1 |
US West (Oregon) | us-west-2 | US_WEST_2 |
Prerequisites
Create the S3 bucket and the SQS queue in the same region.
- Create an S3 bucket.
- Create an SQS queue.
- The queue must be a Standard queue, not a FIFO queue.
- Set up notifications on your S3 bucket
to write to your SQS queue.
- Be sure to attach an access policy.
- Create an access key which will be used to access both the SQS queue and the S3 bucket.
To learn more about how to configure a feed to ingest data from an Amazon SQS queue whose entries point to files stored in an Amazon S3 bucket, see Ingest AWS logs.
Recommendation
The use of Amazon SQS is preferred over Amazon S3 as the source type. When an Amazon SQS service is used, Google Security Operations reads the Amazon S3 notifications that are sent to the Amazon SQS service and pulls the corresponding files out of the Amazon S3 bucket. This is effectively a push-based version of an Amazon S3 feed and can be used to reduce the latency of ingestion.
Type-specific request fields
Field | Required | Description |
---|---|---|
details.amazonSqsSettings.queue |
Yes | The SQS queue name. |
details.amazonSqsSettings.region |
Yes | The region where the SQS queue and S3 bucket reside. For a list of regions, see Amazon S3 regions. |
details.amazonSqsSettings.accountNumber |
Yes | The account number for the SQS queue and S3 bucket. |
details.amazonSqsSettings.sourceDeletionOption |
Yes | Whether to delete the source files in the S3 bucket after they have been transferred to Google Security Operations. This reduces storage costs. Valid values are:
|
details.amazonSqsSettings.authentication.sqsAccessKeySecretAuth.accessKeyId |
Yes | This is the 20 character ID associated with your Amazon IAM account. |
details.amazonSqsSettings.authentication.sqsAccessKeySecretAuth.secretAccessKey |
Yes | This is the 40 character access key associated with your Amazon IAM account. |
details.amazonSqsSettings.authentication.additionalS3AccessKeySecretAuth.accessKeyId |
No | This is the 20 character ID associated with your Amazon IAM account. Only specify if using a different access key for the S3 bucket. |
details.amazonSqsSettings.authentication.additionalS3AccessKeySecretAuth.secretAccessKey |
No | This is the 40 character access key associated with your Amazon IAM account. Only specify if using a different access key for the S3 bucket. |
Sample Create Feed request
{
"details": {
"feedSourceType": "AMAZON_SQS",
"logType": "LOGTYPE_YOU_WANT_TO_BRING",
"amazonSqsSettings": {
"queue": "cs-prod-canon-queue-01234abc56de789f",
"region": "US_EAST_1",
"accountNumber": "123456789012",
"sourceDeletionOption": "SOURCE_DELETION_NEVER"
"authentication": {
"sqsAccessKeySecretAuth": {
"accessKeyId": "AKIAIOSFODNN7EXAMPLE",
"secretAccessKey": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
},
},
}
}
}
AZURE_BLOBSTORE
Data source | Ingest schedule | details.feedSourceType
|
details.logType
|
---|---|---|---|
Microsoft Azure Blob Storage Container | Every 15 minutes | AZURE_BLOBSTORE
|
See the Feed Schema API reference to get compatible log types. |
Prerequisites
You will need either:
- A shared key authorized to access an Azure Blob Storage container.
- A Shared Access Signature with authority to read an Azure Blob Storage container.
To learn more about how to configure a feed to ingest data from Azure Blob Storage, see Ingest Azure Activity logs.
Recommendation
If your Azure Blob Storage container contains many small files, it can increase the amount of time it takes to transfer the data. It is recommended to combine smaller files into one large file to accelerate the feed transfer process.
It is important to set a data retention policy to your Azure Blob Storage containers so that the transferred files are deleted from the containers and are not listed in any future transfer feeds.
As Google SecOps pulls files from containers on a frequent basis, it is recommended to specify a storage class for your containers that is the most cost-effective.
Type-specific request fields
Field | Required | Description |
---|---|---|
details.azureBlobStoreSettings.azureUri |
Yes | The URI pointing to a Azure Blob Storage blob or container. |
details.azureBlobStoreSettings.sourceType |
Yes | The type of object indicated by the URI. Valid values are:
|
details.azureBlobStoreSettings.sourceDeletionOption |
Yes | Source file deletion is not supported in Azure. This field's value must be set to SOURCE_DELETION_NEVER . |
details.azureBlobStoreSettings.authentication.sharedKey |
No | A shared key, a 512-bit random string in base64 encoding, authorized to access Azure Blob Storage. Required if not specifying an SAS Token. |
details.azureBlobStoreSettings.authentication.sasToken |
No | A Shared Access Signature authorized to access the Azure Blob Storage container. |
Azure URI source types
When specifying an Azure URI you must also indicate the type of object is indicated by the URI.
details.sourceType |
Source type |
---|---|
FILES
|
The URI points to a single blob that will be ingested with each execution of the feed. |
FOLDERS
|
The URI points to a directory. All files contained within the directory will be ingested with each execution of the feed. |
FOLDERS_RECURSIVE |
The URI points to a Blob Storage container. |
Sample Create Feed request
{
"details": {
"feedSourceType": "AZURE_BLOBSTORE",
"logType": "LOGTYPE_YOU_WANT_TO_BRING",
"azureBlobStoreSettings": {
"azureUri": "https://myaccount.blob.core.windows.net/logging",
"sourceType": "FOLDERS_RECURSIVE",
"sourceDeletionOption": "SOURCE_DELETION_NEVER"
"authentication": {
"sharedKey": "Ab12CyDEFG3HI45JklMnopQrs00TU6xVw7xYZ8AbcdeFgHioJkL0MnoPqRsTUvWxYZaBCdEFg9hijKlm0N12pqR==",
},
}
}
}
HTTP
Data source | Ingest schedule | details.feedSourceType
|
details.logType
|
---|---|---|---|
Files available over the open internet via an HTTP request. | Every 15 minutes | HTTP
|
See the Feed Schema API reference to get compatible log types. |
WARNING: The HTTP type should not be used to gather data from an API. Refer to the following supported API feed types.
Type-specific request fields
Field | Required | Description |
---|---|---|
details.httpSettings.uri |
Yes | The URI pointing to a file or collection of files. |
details.httpSettings.sourceType |
Yes | The type of file indicated by the URI. Valid values are:
|
details.httpSettings.sourceDeletionOption |
Yes | Whether to delete source files after they have been transferred to Google Security Operations. This reduces storage costs. Valid values are:
|
Sample Create Feed request
{
"details": {
"feedSourceType": "HTTP",
"logType": "LOGTYPE_YOU_WANT_TO_BRING",
"httpSettings": {
"uri": "https://url.com/myfile",
"sourceType": "FILES",
"sourceDeletionOption": "SOURCE_DELETION_NEVER"
}
}
}
Webhook
Data source | Ingest schedule | details.feedSourceType |
---|---|---|
Data that is streamed to an HTTPS webhook. | Based on your implementation, it might take approximately a minute for a new feed to populate because Google Security Operations takes some time to process the data. | HTTPS_PUSH_WEBHOOK |
Prerequisites
- Ensure that a Google Cloud project for Google Security Operations is configured and the Chronicle API is enabled for the project.
- Link a Google Security Operations instance to Google Cloud services.
Set up push ingestion using an HTTPS webhook
Data can be sent to Google Security Operations using an HTTPS webhook. You must first create a feed with the appropriate log type before configuring an HTTPS webhook to send data.
To set up HTTPS push ingestion using an HTTPS webhook, do the following:
Create an HTTPS webhook feed using the following
create
API request:{ "displayName": "FEED_NAME", "details": { "feedSourceType": "HTTPS_PUSH_WEBHOOK", "logType": "projects/PROJECT_NUMBER/locations/REGION_ID/instances/CUSTOMER_ID/logTypes/LOG_TYPE" } }
Replace the following:
FEED_NAME
: specify a name for the feed.PROJECT_NUMBER
: an automatically generated unique identifier for your project. For information about project name, project ID, and project number, which are used to identify a project, see Creating and managing projects.REGION_ID
: the code that Google assigns based on the region. The following are the supported region IDs:Asia-Southeast1
,Australia-Southeast1
,Europe
,EU
,Europe-West2
,Europe-West3
,Europe-West6
,Govcloud-US
,Me-West1
, andUS
.CUSTOMER_ID
: the Google Security Operations customer ID.LOG_TYPE
: the type of data the feed ingests. Google Security Operations supports specific log types for the HTTPS webhook feed.
If the data to be ingested contains a delimiter that separate log lines, such as
\\n
, include the following in thedetails
field of the feed request body:"httpsPushWebhookSettings": { "splitDelimiter": "LOG_DELIMITER" }
Replace
LOG_DELIMITER
with the delimiter that separates the log lines, such as\\n
.After you create a feed, generate a secret key for the feed and generate an API key to authenticate to Google Security Operations.
In your client application, specify the HTTPS endpoint. Here is a sample HTTPS webhook endpoint:
https://REGIONAL_ENDPOINT/v1alpha/projects/PROJECT_NUMBER/locations/REGION_ID/instances/CUSTOMER_ID/feeds/FEED_ID:importPushLogs
The endpoint includes the following values:
REGIONAL_ENDPOINT
: the Google Security Operations regional endpoint, such asus-chronicle.googleapis.com
. For information about supported regional endpoints, see Google Security Operations regional endpoints section of this document.PROJECT_NUMBER
: an automatically generated unique identifier for your project. For information about project name, project ID, and project number, which are used to identify a project, see Creating and managing projects.REGION_ID
: the code that Google assigns based on the region. The following are the supported region IDs:Asia-Southeast1
,Australia-Southeast1
,Europe
,EU
,Europe-West2
,Europe-West3
,Europe-West6
,Govcloud-US
,Me-West1
, andUS
.CUSTOMER_ID
: the Google Security Operations customer ID.FEED_ID
: the Google Security Operations feed ID.
Enable authentication by specifying the API key and secret key as part of the custom header in the following format:
X-goog-api-key = KEY_VALUE
X-Webhook-Access-Key = SECRET_VALUE
Replace the following:
SECRET_VALUE
: the secret key value that you generated using theGenerateSecret
API. You can also pass the secret as a query parameter in the endpoint URL(?secret=SECRET_VALUE)
. We recommend that you specify the secret as a header instead of specifying it in the URL.KEY_VALUE
: the API key value. You can also pass the API key as a query parameter in the endpoint URL(?key=KEY_VALUE)
. We recommend that you specify the API key as a header instead of specifying it in the URL.
Google Security Operations regional endpoints
When you construct the HTTPS endpoint URL to push feeds, use the following regional endpoints that Google Security Operations supports:
asia-northeast1-chronicle.googleapis.com
asia-south1-chronicle.googleapis.com
asia-southeast1-chronicle.googleapis.com
australia-southeast1-chronicle.googleapis.com
eu-chronicle.googleapis.com
europe-west2-chronicle.googleapis.com
europe-west3-chronicle.googleapis.com
europe-west6-chronicle.googleapis.com
europe-west12-chronicle.googleapis.com
govcloud-us-chronicle.googleapis.com
me-central1-chronicle.googleapis.com
me-central2-chronicle.googleapis.com
me-west1-chronicle.googleapis.com
northamerica-northeast2-chronicle.googleapis.com
us-chronicle.googleapis.com
Configuration by log type
The following table lists the log types that Google Security Operations supports
for the API
feed source type (that is, ingesting data from third-party APIs).
If a log type has Google Security Operations parser support, the ingested data is stored in Google Security Operations UDM format as well as raw log data.
Click a Data Source name for detailed reference information, prerequisites, and API examples for the log type.
To learn about prerequisites for other log types and feed source types, see Prerequisites.
Data Source | Log Type | Google Security Operations Parser Support |
---|---|---|
Anomali ThreatStream | ANOMALI_IOC |
Yes |
CrowdStrike Detection Monitoring | CS_DETECTS |
Yes |
Duo Authentication Logs | DUO_AUTH |
Yes |
Duo Users | DUO_USER_CONTEXT |
Yes |
Fidelis Cloud Passage Events | CLOUD_PASSAGE |
Yes |
Fox-IT | FOX_IT_STIX |
No |
Google Cloud Identity Devices | GCP_CLOUDIDENTITY_DEVICES |
Yes |
Google Cloud Identity Device Users | GCP_CLOUDIDENTITY_DEVICEUSERS |
Yes |
Google Workspace Activity | WORKSPACE_ACTIVITY |
Yes |
Google Workspace Alerts | WORKSPACE_ALERTS |
Yes |
Google Workspace Chrome | WORKSPACE_CHROMEOS |
Yes |
Google Workspace Groups | WORKSPACE_GROUPS |
Yes |
Google Workspace Mobile | WORKSPACE_MOBILE |
Yes |
Google Workspace Privileges | WORKSPACE_PRIVILEGES |
Yes |
Google Workspace Users | WORKSPACE_USERS |
Yes |
Imperva | IMPERVA_WAF |
Yes |
Microsoft Azure AD Directory Audit | AZURE_AD_AUDIT | Yes |
Microsoft Azure AD Context | AZURE_AD_CONTEXT |
Yes |
Microsoft Azure AD Sign-Ins | AZURE_AD |
Yes |
Microsoft Azure MDM Intune Audit Events | AZURE_MDM_INTUNE |
Yes |
Microsoft Graph Security API | MICROSOFT_GRAPH_ALERT |
Yes |
Microsoft 365 Management Activity | OFFICE_365 |
Yes |
Mimecast Secure Email Gateway | MIMECAST_MAIL |
Yes |
Netskope Alerts | NETSKOPE_ALERT |
Yes |
Netskope Alerts V2 | NETSKOPE_ALERT_V2 |
Yes |
Okta System Log | OKTA |
Yes |
Okta Users | OKTA_USER_CONTEXT |
Yes |
Palo Alto Networks Autofocus | PAN_IOC |
Yes |
Palo Alto Networks Cortex XDR | CORTEX_XDR |
Yes |
Palo Alto Networks Prisma Cloud Audit Logs | PAN_PRISMA_CLOUD |
Yes |
Proofpoint on Demand | PROOFPOINT_ON_DEMAND |
Yes |
Proofpoint TAP | PROOFPOINT_MAIL |
Yes |
Qualys VM | QUALYS_VM |
Yes |
Qualys Scan | QUALYS_SCAN |
No |
Rapid7 InsightVM | RAPID7_INSIGHT |
Yes |
Recorded Future | RECORDED_FUTURE_IOC |
Yes |
RH-ISAC | RH_ISAC_IOC |
Yes |
Salesforce | SALESFORCE |
Yes |
SentinelOne Alert | SENTINELONE_ALERT |
Yes |
ServiceNow CMDB | SERVICENOW_CMDB |
Yes |
Symantec Event Export | SYMANTEC_EVENT_EXPORT |
Yes |
Thinkst Canary | THINKST_CANARY |
Yes |
ThreatConnect | THREATCONNECT_IOC |
Yes |
Workday | WORKDAY |
Yes |
Workday Audit Logs | WORKDAY_AUDIT |
No |
AWS EC2 Hosts | AWS_EC2_HOSTS |
Yes |
AWS EC2 Instances | AWS_EC2_INSTANCES |
Yes |
AWS EC2 VPCs | AWS_EC2_VPCS |
Yes |
AWS Identity and Access Management | AWS_IAM |
Yes |
Anomali ThreatStream
This section provides API reference details for the ANOMALI_IOC
log type. For details about the data source, see the Anomali ThreatStream documentation.
Data source | Ingest schedule | details.feedSourceType |
details.logType |
---|---|---|---|
api.threatstream.com/api/v2/intelligence | Every minute | API |
ANOMALI_IOC |
Prerequisites
- Get the values for all required request fields.
- Get the following required permissions:
- None
Type-specific request fields
Field | Required | Description |
---|---|---|
details.anomaliSettings.authentication.user |
Yes | Username |
details.anomaliSettings.authentication.secret |
Yes | API key |
Sample Create Feed request
{
"details": {
"feedSourceType": "API",
"logType": "ANOMALI_IOC",
"anomaliSettings": {
"authentication": {
"user": "USERNAME",
"secret": "APIKEY"
},
}
}
}
AWS EC2 Hosts
This section provides API reference details for the AWS_EC2_HOSTS
log type.
Prerequisites
- Get the values for all required request fields.
- Get the following required permissions:
-
The user whose credentials are used to authenticate must have the
AmazonEC2ReadOnlyAccess
permission.
-
The user whose credentials are used to authenticate must have the
Type-specific request fields
The following table lists the field values required when creating a feed to collect
data for the AWS_EC2_HOSTS
log type.
Field | Required | Description |
---|---|---|
details.awsEc2HostsSettings.authentication.user |
Yes | This is the 20 character ID associated with your Amazon IAM account. |
details.awsEc2HostsSettings.authentication.secret |
Yes | This is the 40 character access key associated with your Amazon IAM account. |
Sample Create Feed request
{
"details": {
"awsEc2HostsSettings": {
"authentication": {
"user": "AccessKeyID",
"secret": "SecretAccessKey"
}
},
"feedSourceType": "API",
"logType": "AWS_EC2_HOSTS"
}
}
AWS EC2 Instances
This section provides API reference details for the AWS_EC2_INSTANCES
log type.
Prerequisites
- Get the values for all required request fields.
- Get the following required permissions:
-
The user whose credentials are used to authenticate must have the
AmazonEC2ReadOnlyAccess
permission.
-
The user whose credentials are used to authenticate must have the
Type-specific request fields
The following table lists the field values required when creating a feed to collect
data for the AWS_EC2_INSTANCES
log type.
Field | Required | Description |
---|---|---|
details.awsEc2InstancesSettings.authentication.user |
Yes | This is the 20 character ID associated with your Amazon IAM account. |
details.awsEc2InstancesSettings.authentication.secret |
Yes | This is the 40 character access key associated with your Amazon IAM account. |
Sample Create Feed request
{
"details": {
"awsEc2InstancesSettings": {
"authentication": {
"user": "AccessKeyID",
"secret": "SecretAccessKey"
}
},
"feedSourceType": "API",
"logType": "AWS_EC2_INSTANCES"
}
}
AWS EC2 VPCs
This section provides API reference details for the AWS_EC2_VPCS
log type.
Prerequisites
- Get the values for all required request fields.
- Get the following required permissions:
-
The user whose credentials are used to authenticate must have the
AmazonEC2ReadOnlyAccess
permission.
-
The user whose credentials are used to authenticate must have the
Type-specific request fields
The following table lists the field values required when creating a feed to collect
data for the AWS_EC2_VPCS
log type.
Field | Required | Description |
---|---|---|
details.awsEc2VpcsSettings.authentication.user |
Yes | This is the 20 character ID associated with your Amazon IAM account. |
details.awsEc2VpcsSettings.authentication.secret |
Yes | This is the 40 character access key associated with your Amazon IAM account. |
Sample Create Feed request
{
"details": {
"awsEc2VpcsSettings": {
"authentication": {
"user": "AccessKeyID",
"secret": "SecretAccessKey"
}
},
"feedSourceType": "API",
"logType": "AWS_EC2_VPCS"
}
}
AWS Identity and Access Management
This section provides API reference details for the AWS_IAM
log type.
Prerequisites
- Get the values for all required request fields.
- Get the following required permissions:
- The user whose credentials are used to authenticate must have the
IAMReadOnlyAccess
permission.
- The user whose credentials are used to authenticate must have the
Type-specific request fields
The following table lists the field values required when creating a feed to collect
data for the AWS_IAM
log type.
Field | Required | Description |
---|---|---|
details.awsIamSettings.authentication.user |
Yes | This is the 20 character ID associated with your Amazon IAM account. |
details.awsIamSettings.authentication.secret |
Yes | This is the 40 character access key associated with your Amazon IAM account. |
details.awsIamSettings.apiType |
Yes | API which needs to be called (Users/Roles/Groups). |
Sample Create Feed request
{
"details": {
"awsIamSettings": {
"authentication": {
"user": "AccessKeyID",
"secret": "SecretAccessKey"
},
"apiType": "USERS"
},
"feedSourceType": "API",
"logType": "AWS_IAM"
}
}
CrowdStrike Detection Monitoring
This section provides API reference details for the CS_DETECTS
log type. For details about the data source, see the CrowdStrike Detection Monitoring documentation.
Data source | Ingest schedule | details.feedSourceType |
details.logType |
---|---|---|---|
|
Every minute | API |
CS_DETECTS |
Prerequisites
- Get the values for all required request fields.
- Get the following required permissions:
- None
Type-specific request fields
Field | Required | Description |
---|---|---|
details.crowdstrikeDetectsSettings.authentication.clientId |
Yes | Application ID |
details.crowdstrikeDetectsSettings.authentication.clientSecret |
Yes | Client Secret |
details.crowdstrikeDetectsSettings.authentication.tokenEndpoint |
Yes | Authentication URL |
details.crowdstrikeDetectsSettings.hostname |
Yes | API Endpoint URL |
Sample Create Feed request
{
"details": {
"feedSourceType": "API",
"logType": "CS_DETECTS",
"crowdstrikeDetectsSettings": {
"authentication": {
"clientId": "CLIENT ID",
"clientSecret": "CLIENT SECRET",
"tokenEndpoint": "https://api.us-2.crowdstrike.com/oauth2/token"
},
"hostname": "api.crowdstrike.com"
}
}
}
Steps to enable Crowdstrike Feed
Create a CrowdStrike API Client
Within the CrowdStrike application, create an API client by navigating to Support and resources > API clients and keys.
Create a new API Client with API scopes granting permission to Read Detections.
Record the values for: Base URL + Client ID + Client Secret. You will need these to set up the Feed in Google Security Operations.
Setup the Google Security Operations feed
- Launch Google Security Operations, then select the Settings menu.
- Select Feeds in the left pane and click Add New.
- Select the Third Party API Source Type and Crowdstrike Detection Monitoring log type.
Fill in the requested parameters gathered earlier from CrowdStrike and click Submit.
The feed will begin to retrieve all detections from the CrowdStrike instance in chronological order. Detections older than 6 months will be dropped. After the backfill is complete, which can take some time depending on the number of detections in CrowdStrike, the feed checks for new detections every 5 minutes.
Duo Authentication Logs
This section provides API reference details for the DUO_AUTH
log type. For details about the data source, see the Duo Authentication Logs documentation.
Data source | Ingest schedule | details.feedSourceType |
---|---|---|
API_HOSTNAME/admin/v2/logs/authentication
Replace |
Every 30 minutes | API |
Prerequisites
- Get the values for all authentication fields.
- Get the following required permissions:
- None
Type-specific request fields
Field | Required | Description |
---|---|---|
details.duoAuthSettings.authentication.user |
Yes | The username to authenticate to Duo. |
details.duoAuthSettings.authentication.secret |
Yes | The secret to authenticate to Duo. |
details.duoAuthSettings.hostname |
Yes | The fully qualified domain name for your instance of the API, such as api-myinstance.duosecurity.com . |
Test the endpoint
The Duo Admin API provides programmatic access to the administrative functionality of Duo Security's two-factor authentication platform.
To query your Duo account's authentication logs, you need to send a request to
the /admin/v2/logs/authentication
endpoint.
For details on how to use the API, see the Authentication Logs section in the Duo documentation.
Sample Create Feed request
{
"details": {
"feedSourceType": "API",
"logType": "DUO_AUTH",
"duoAuthSettings": {
"authentication": {
"user": "USERNAME",
"secret": "SECRET"
},
"hostname": "api-mytenant.duosecurity.com"
}
}
}
Duo Users
This section provides API reference details for the DUO_USER_CONTEXT
log type. For details about the data source, see the Duo Users documentation.
Data source | Ingest schedule | details.feedSourceType |
---|---|---|
API_HOSTNAME/admin/v1/users
Replace |
Every 24 hours | API |
Prerequisites
- Get the values for all authentication fields.
- Get the following required permissions:
- None
Type-specific request fields
Field | Required | Description |
---|---|---|
details.duoUserContextSettings.authentication.user |
Yes | The username to authenticate to Duo. |
details.duoUserContextSettings.authentication.secret |
Yes | The secret to authenticate to Duo. |
details.duoUserContextSettings.hostname |
Yes | The fully qualified domain name for your instance of the API, such as api-myinstance.duosecurity.com . |
Sample Create Feed request
{
"details": {
"feedSourceType": "API",
"logType": "DUO_USER_CONTEXT",
"duoUserContextSettings": {
"authentication": {
"user": "USERNAME",
"secret": "SECRET"
},
"hostname": "api-mytenant.duosecurity.com"
}
}
}
Fidelis Cloud Passage Events
This section provides API reference details for the CLOUD_PASSAGE
log type. For details about the data source, see the Cloud Passage Events documentation.
Data source | Ingest schedule | details.feedSourceType |
---|---|---|
api.cloudpassage.com/events?event_types |
Every minute | API |
Prerequisites
- Get the values for all authentication fields.
- Get the following required permissions:
- None
Type-specific request fields
Field | Required | Description |
---|---|---|
details.cloudPassageSettings.authentication.user |
Yes | The username that is used for authentication. |
details.cloudPassageSettings.authentication.secret |
Yes | The secret that is for authentication. |
details.cloudPassageSettings.eventTypes |
No | The type of events to include in the response. If you don't specify any event types, then the following event types are fetched: fim_target_integrity_changed , lids_rule_failed , and sca_rule_failed . |
Sample Create Feed request
{
"details": {
"feedSourceType": "API",
"logType": "CLOUD_PASSAGE",
"cloudPassageSettings": {
"authentication": {
"user": "api_key_id",
"secret": "api_key_secret",
}
"eventTypes": [
"fim_target_integrity_changed",
"lids_rule_failed",
"sca_rule_failed"
],
}
}
}
Fox-IT
This section provides API reference details for the FOX_IT_STIX
log type. For details about the data source, see the Fox-IT documentation.
Prerequisites
- Get the values for all authentication and SSL fields.
- Get the following required permissions:
- None
Sample Create Feed request
{
"details": {
"feedSourceType": "API",
"logType": "FOX_IT_STIX",
"foxItStixSettings": {
"authentication": {
"user": "USERNAME",
"secret": "SECRET"
},
"ssl": {
"sslCertificate": "<cert>",
"encodedPrivateKey": "key"
}
"pollServiceURI": "https://stix.fox-it.com/services/poll",
"collection": "mycollection"
}
}
}
Google Cloud Identity Devices
This section provides API reference details for the GCP_CLOUDIDENTITY_DEVICES
log type. For details about the data source, see the Google Cloud Identity Devices documentation.
Data source | Ingest schedule | details.feedSourceType |
---|---|---|
cloudidentity.googleapis.com/v1/devices
|
Every 24 hours | API |
Prerequisites
- Get the values for all authentication fields.
- Get the following required permissions:
- None
Type-specific request fields
Field | Required | Description |
---|---|---|
details.googleCloudIdentityDevicesSettings.authentication.tokenEndpoint |
Yes | The endpoint to retrieve the OAuth JSON web token. |
details.googleCloudIdentityDevicesSettings.authentication.claims.issuer |
Yes | The JWT claims issuer, which is usually a client ID. |
details.googleCloudIdentityDevicesSettings.authentication.claims.subject |
Yes | The JWT claims subject, which is usually an email ID. |
details.googleCloudIdentityDevicesSettings.authentication.claims.auidence |
Yes | The JWT claims audience. |
details.googleCloudIdentityDevicesSettings.authentication.rsCredentials.privateKey |
Yes | An RSA private key in PEM format. |
details.googleCloudIdentityDevicesSettings.apiVersion |
No | The API version to use to fetch device information. The value must be either v1 , v1beta1 , or vx . If no version is specified, v1 version is used. |
Sample Create Feed request
{
"details": {
"feedSourceType": "API",
"logType": "GCP_CLOUDIDENTITY_DEVICES",
"googleCloudIdentityDevicesSettings": {
"authentication": {
"tokenEndPoint": "jwt_token_uri",
"claims": {
"issuer": "jwt_client_email",
"subject": "user_email",
"audience": "jwt_token_uri"
},
"rsCredentials": {
"private_key": "privatekey"
}
},
"apiVersion": "v1",
}
}
}
Google Cloud Identity Device Users
This section provides API reference details for the GCP_CLOUDIDENTITY_DEVICEUSERS
log type. For details about the data source, see the Google Cloud Identity Device Users documentation.
Data source | Ingest schedule | details.feedSourceType |
---|---|---|
cloudidentity.googleapis.com/v1/devices/-/deviceUsers
|
Every 24 hours | API |
Prerequisites
- Get the values for all authentication fields.
- Get the following required permissions:
- None
Type-specific request fields
Field | Required | Description |
---|---|---|
details.googleCloudIdentityDeviceUsersSettings.authentication.tokenEndpoint |
Yes | The endpoint to retrieve the OAuth JSON web token. |
details.googleCloudIdentityDeviceUsersSettings.authentication.claims.issuer |
Yes | The JWT claims issuer, which is usually a client ID. |
details.googleCloudIdentityDeviceUsersSettings.authentication.claims.subject |
Yes | The JWT claims subject, which is usually an email ID. |
details.googleCloudIdentityDeviceUsersSettings.authentication.claims.auidence |
Yes | The JWT claims audience. |
details.googleCloudIdentityDeviceUsersSettings.authentication.rsCredentials.privateKey |
Yes | An RSA private key in PEM format. |
details.googleCloudIdentityDeviceUsersSettings.apiVersion |
No | The API version to use to fetch device information. The value must be either v1 , v1beta1 , or vx . If no version is specified, v1 version is used. |
Sample Create Feed request
{
"details": {
"feedSourceType": "API",
"logType": "GCP_CLOUDIDENTITY_DEVICEUSERS",
"googleCloudIdentityDeviceUsersSettings": {
"authentication": {
"tokenEndPoint": "jwt_token_uri",
"claims": {
"issuer": "jwt_client_email",
"subject": "user_email",
"audience": "jwt_token_uri"
},
"rsCredentials": {
"private_key": "privatekey"
}
},
}
}
}
Google Workspace Activities
This section provides API reference details for the WORKSPACE_ACTIVITY
log type. For details about the data source, see the Google Workspace Activities documentation.
Data source | Ingest schedule | details.feedSourceType |
details.logType |
---|---|---|---|
admin.googleapis.com | Every hour | API |
WORKSPACE_ACTIVITY |
Prerequisites
In order for Google Security Operations to ingest Google Workspace activities, you must do the following:
- Enable the Admin SDK API on your Google Cloud project.
- Create a Service Account which will be used to authenticate against the Admin API.
- Generate a JSON key for the Service Account.
- Create a domain-wide delegation for the Service Account with the following OAuth scope:
- Create a Google Workspace user and assign it an admin role which includes the Reports admin privilege, or create a custom role which includes that privilege.
- Locate your Google Workspace customer ID.
To learn more about how to configure a feed to ingest Google Workspace logs, see Collect Google Workspace logs.
Type-specific request fields
Field | Required | Description |
---|---|---|
details.workspaceActivitySettings.authentication.tokenEndpoint |
Yes | The value of the token_uri field in the JSON key for the service account created to access the admin API. |
details.workspaceActivitySettings.authentication.claims.issuer |
Yes | The value of the client_email field in the JSON key for the service account created to access the admin API. |
details.workspaceActivitySettings.authentication.claims.subject |
Yes | The email address of the Google Workspace admin user with Reports privilege. |
details.workspaceActivitySettings.authentication.claims.audience |
Yes | The value of the token_uri field in the JSON key for the service account created to access the admin API. |
details.workspaceActivitySettings.authentication.rsCredentials.privateKey |
Yes | The value of the private_key field in the JSON key for the service account created to access the admin API. Note that literal newline characters (\n ) should be replaced with carriage returns. Also note that the field name is rsCredentials , and not rsaCredentials . |
details.workspaceActivitySettings.workspaceCustomerId |
Yes | The Google Workspace customer ID. The customer ID must have a leading 'C' character. The customer ID may appear differently depending on where in the Google admin console it is found. If the customer ID you have does not have a leading 'C', then prepend what you have with a 'C'. |
details.workspaceActivitySettings.applications |
Yes | The Google Workspace applications to gather activities for. See the following table for valid values. |
Google Workspace applications
Activities are associated with one or more applications. The applications that Google Security Operations supports includes the following.
details.workspaceActivitySettings.applications |
Description |
---|---|
access_transparency |
Access Transparency log events |
admin |
Admin log events |
calendar |
Calendar log events |
chat |
Chat log events |
drive |
Drive log events |
gcp |
Google Cloud activity events |
gplus |
Currents log events |
groups |
Groups log events |
groups_enterprise |
Groups Enterprise log events |
jamboard |
Jamboard log events |
login |
User log events |
meet |
Meet log events |
mobile |
Device log events |
rules |
Rule log events (beta) |
saml |
SAML log events |
token |
OAuth log events |
user_accounts |
User log events |
context_aware_access |
Context-Aware Access log events |
chrome |
Chrome log events |
data_studio |
Looker Studio log events |
keep |
Keep log events |
Sample Create Feed request
{
"details": {
"feedSourceType": "API",
"logType": "WORKSPACE_ACTIVITY",
"workspaceActivitySettings": {
"authentication": {
"tokenEndpoint": "https://oauth2.googleapis.com/token",
"claims": {
"issuer": "service-account@project.iam.gserviceaccount.com",
"subject": "user@domain.com",
"audience": "https://oauth2.googleapis.com/token"
},
"rsCredentials": {
"privateKey": "-----BEGIN PRIVATE KEY-----
ABCDeFGHIJKLMnopqrsT0u1VWXY...z/abCdefgHIJK+lMN2o345P=
-----END PRIVATE KEY-----"
},
},
"workspaceCustomerId": "C1e2x3ample",
"applications": [
"admin",
"groups",
"mobile"
],
}
}
}
Google Workspace Alerts
This section provides API reference details for the WORKSPACE_ALERTS
log type. For details about the data source, see the Google Workspace Alerts documentation.
Data source | Ingest schedule | details.feedSourceType |
details.logType |
---|---|---|---|
alertcenter.googleapis.com | Every hour | API |
WORKSPACE_ALERTS |
Prerequisites
For Google Security Operations to ingest Google Workspace alerts, complete the following steps:
- Enable the Alert Center API on your Google Cloud project.
- Create a Service Account which will be used to authenticate against the Alert Center API.
- Generate a JSON key for the Service Account.
- Create a domain-wide delegation for the Service Account with the following OAuth scope:
- Create a Google Workspace user and assign it an admin role which includes Alert Center view access, or create a custom role which includes that privilege.
- Locate your Google Workspace customer ID.
To learn more about how to configure a feed to ingest Google Workspace logs, see Collect Google Workspace logs.
Type-specific request fields
Field | Required | Description |
---|---|---|
details.workspaceAlertsSettings.authentication.tokenEndpoint |
Yes | The value of the token_uri field in the JSON key for the service account created to access the admin API. |
details.workspaceAlertsSettings.authentication.claims.issuer |
Yes | The value of the client_email field in the JSON key for the service account created to access the admin API. |
details.workspaceAlertsSettings.authentication.claims.subject |
Yes | The email address of the Google Workspace admin user with Alert Center view access. |
details.workspaceAlertsSettings.authentication.claims.audience |
Yes | The value of the token_uri field in the JSON key for the service account created to access the admin API. |
details.workspaceAlertsSettings.authentication.rsCredentials.privateKey |
Yes | The value of the private_key field in the JSON key for the service account created to access the Alert Center API. Note that literal newline characters (\n ) should be replaced with carriage returns. Also note that the field name is rsCredentials , and not rsaCredentials . |
details.workspaceAlertsSettings.workspaceCustomerId |
Yes | The Google Workspace customer ID. Note that the customer ID must not have a leading 'C' character. The customer ID may appear differently depending on where in the Google admin console it is found. If the customer ID you have has a leading 'C', then remove it before including in your request. |
Sample Create Feed request
{
"details": {
"feedSourceType": "API",
"logType": "WORKSPACE_ALERTS",
"workspaceAlertsSettings": {
"authentication": {
"tokenEndpoint": "https://oauth2.googleapis.com/token",
"claims": {
"issuer": "service-account@project.iam.gserviceaccount.com",
"subject": "user@domain.com",
"audience": "https://oauth2.googleapis.com/token"
},
"rsCredentials": {
"privateKey": "-----BEGIN PRIVATE KEY-----
ABCDeFGHIJKLMnopqrsT0u1VWXY...z/abCdefgHIJK+lMN2o345P=
-----END PRIVATE KEY-----"
},
},
"workspaceCustomerId": "1e2x3ample",
}
}
}
Google Workspace ChromeOS Devices
This section provides API reference details for the WORKSPACE_CHROMEOS
log type. For details about the data source, see the Google Workspace ChromeOS Devices documentation.
Data source | Ingest schedule | details.feedSourceType |
details.logType |
---|---|---|---|
admin.googleapis.com | Every 24 hours | API |
WORKSPACE_CHROMEOS |
Prerequisites
For Google Security Operations to ingest Google Workspace ChromeOS devices, complete the following steps:
- Enable the Admin SDK API on your Google Cloud project.
- Create a Service Account which will be used to authenticate against the Admin API.
- Generate a JSON key for the Service Account.
- Create a domain-wide delegation for the Service Account with the following OAuth scope:
- Create a Google Workspace user and assign it an admin role which includes Chrome Management Settings access, or create a custom role which includes that privilege.
- Locate your Google Workspace customer ID.
To learn more about how to configure a feed to ingest Google Workspace logs, see Collect Google Workspace logs.
Type-specific request fields
Field | Required | Description |
---|---|---|
details.workspaceChromeOsSettings.authentication.tokenEndpoint |
Yes | The value of the token_uri field in the JSON key for the service account created to access the admin API. |
details.workspaceChromeOsSettings.authentication.claims.issuer |
Yes | The value of the client_email field in the JSON key for the service account created to access the admin API. |
details.workspaceChromeOsSettings.authentication.claims.subject |
Yes | The email address of the Google Workspace admin user with Reports privilege. |
details.workspaceChromeOsSettings.authentication.claims.audience |
Yes | The value of the token_uri field in the JSON key for the service account created to access the admin API. |
details.workspaceChromeOsSettings.authentication.rsCredentials.privateKey |
Yes | The value of the private_key field in the JSON key for the service account created to access the admin API. Replace the literal newline characters (\n ) with carriage returns. The field name is rsCredentials , and not rsaCredentials . |
details.workspaceChromeOsSettings.workspaceCustomerId |
Yes | The Google Workspace customer ID. Note that the customer ID must have a leading 'C' character. The customer ID may appear differently depending on where in the Google admin console it is found. If the customer ID you have does not have a leading 'C', then prepend what you have with a 'C'. |
Sample Create Feed request
{
"details": {
"feedSourceType": "API",
"logType": "WORKSPACE_CHROMEOS",
"workspaceChromeOsSettings": {
"authentication": {
"tokenEndpoint": "https://oauth2.googleapis.com/token",
"claims": {
"issuer": "service-account@project.iam.gserviceaccount.com",
"subject": "user@domain.com",
"audience": "https://oauth2.googleapis.com/token"
},
"rsCredentials": {
"privateKey": "-----BEGIN PRIVATE KEY-----
ABCDeFGHIJKLMnopqrsT0u1VWXY...z/abCdefgHIJK+lMN2o345P=
-----END PRIVATE KEY-----"
},
},
"workspaceCustomerId": "C1e2x3ample",
}
}
}
Google Workspace Groups
This section provides API reference details for the WORKSPACE_GROUPS
log type. For details about the data source, see the Google Workspace Groups documentation.
Data source | Ingest schedule | details.feedSourceType |
details.logType |
---|---|---|---|
admin.googleapis.com | Every 24 hours | API |
WORKSPACE_GROUPS |
Prerequisites
For Google Security Operations to ingest Google Workspace ChromeOS devices, complete the following steps:
- Enable the Admin SDK API on your Google Cloud project.
- Create a Service Account which will be used to authenticate against the Admin API.
- Generate a JSON key for the Service Account.
- Create a domain-wide delegation for the Service Account with the following OAuth scope:
- Create a Google Workspace user and assign it an admin role which includes Admin API Group read privileges, or create a custom role which includes that privilege.
- Locate your Google Workspace customer ID.
To learn more about how to configure a feed to ingest Google Workspace logs, see Collect Google Workspace logs.
Type-specific request fields
Field | Required | Description |
---|---|---|
details.workspaceGroupsSettings.authentication.tokenEndpoint |
Yes | The value of the token_uri field in the JSON key for the service account created to access the admin API. |
details.workspaceGroupsSettings.authentication.claims.issuer |
Yes | The value of the client_email field in the JSON key for the service account created to access the admin API. |
details.workspaceGroupsSettings.authentication.claims.subject |
Yes | The email address of the Google Workspace admin user with the Admin API Group read privilege. |
details.workspaceGroupsSettings.authentication.claims.audience |
Yes | The value of the token_uri field in the JSON key for the service account created to access the admin API. |
details.workspaceGroupsSettings.authentication.rsCredentials.privateKey |
Yes | The value of the private_key field in the JSON key for the service account created to access the admin API. Note that literal newline characters (\n ) should be replaced with carriage returns. Note that the field name is rsCredentials , and not rsaCredentials . |
details.workspaceGroupsSettings.workspaceCustomerId |
Yes | The Google Workspace customer ID. Note that the customer ID must have a leading 'C' character. The customer ID may appear differently depending on where in the Google admin console it is found. If the customer ID you have does not have a leading 'C' then prepend what you have with a 'C'. |
Sample Create Feed request
{
"details": {
"feedSourceType": "API",
"logType": "WORKSPACE_GROUPS",
"workspaceGroupsSettings": {
"authentication": {
"tokenEndpoint": "https://oauth2.googleapis.com/token",
"claims": {
"issuer": "service-account@project.iam.gserviceaccount.com",
"subject": "user@domain.com",
"audience": "https://oauth2.googleapis.com/token"
},
"rsCredentials": {
"privateKey": "-----BEGIN PRIVATE KEY-----
ABCDeFGHIJKLMnopqrsT0u1VWXY...z/abCdefgHIJK+lMN2o345P=
-----END PRIVATE KEY-----"
},
},
"workspaceCustomerId": "C1e2x3ample",
}
}
}
Google Workspace Mobile Devices
This section provides API reference details for the WORKSPACE_MOBILE
log type. For details about the data source, see the Google Workspace Mobile Devices documentation.
Data source | Ingest schedule | details.feedSourceType |
details.logType |
---|---|---|---|
admin.googleapis.com | Every 24 hours | API |
WORKSPACE_GROUPS |
Prerequisites
In order for Google Security Operations to ingest Google Workspace ChromeOS devices, complete the following steps:
- Enable the Admin SDK API on your Google Cloud project.
- Create a Service Account which will be used to authenticate against the Admin API.
- Generate a JSON key for the Service Account.
- Create a domain-wide delegation for the Service Account with the following OAuth scope:
- Create a Google Workspace user and assign it an admin role which includes Admin API Group read privileges, or create a custom role which includes that privilege.
- Locate your Google Workspace customer ID.
To learn more about how to configure a feed to ingest Google Workspace logs, see Collect Google Workspace logs.
Type-specific request fields
Field | Required | Description |
---|---|---|
details.workspaceGroupsSettings.authentication.tokenEndpoint |
Yes | The value of the token_uri field in the JSON key for the service account created to access the admin API. |
details.workspaceGroupsSettings.authentication.claims.issuer |
Yes | The value of the client_email field in the JSON key for the service account created to access the admin API. |
details.workspaceGroupsSettings.authentication.claims.subject |
Yes | The email address of the Google Workspace admin user with the Admin API Group read privilege. |
details.workspaceGroupsSettings.authentication.claims.audience |
Yes | The value of the token_uri field in the JSON key for the service account created to access the admin API. |
details.workspaceGroupsSettings.authentication.rsCredentials.privateKey |
Yes | The value of the private_key field in the JSON key for the service account created to access the admin API. Note that literal newline characters (\n ) should be replaced with carriage returns. Note that the field name is rsCredentials , and not rsaCredentials . |
details.workspaceGroupsSettings.workspaceCustomerId |
Yes | The Google Workspace customer ID. Note that the customer ID must have a leading 'C' character. The customer ID may appear differently depending on where in the Google admin console it is found. If the customer ID you have does not have a leading 'C' then prepend what you have with a 'C'. |
Sample Create Feed request
{
"details": {
"feedSourceType": "API",
"logType": "WORKSPACE_MOBILE",
"workspaceMobileSettings": {
"authentication": {
"tokenEndpoint": "https://oauth2.googleapis.com/token",
"claims": {
"issuer": "service-account@project.iam.gserviceaccount.com",
"subject": "user@domain.com",
"audience": "https://oauth2.googleapis.com/token"
},
"rsCredentials": {
"privateKey": "-----BEGIN PRIVATE KEY-----
ABCDeFGHIJKLMnopqrsT0u1VWXY...z/abCdefgHIJK+lMN2o345P=
-----END PRIVATE KEY-----"
},
},
"workspaceCustomerId": "C1e2x3ample",
}
}
}
Google Workspace Privileges
This section provides API reference details for the WORKSPACE_PRIVILEGES
log type. For details about the data source, see the Google Workspace Privileges documentation.
Data source | Ingest schedule | details.feedSourceType |
details.logType |
---|---|---|---|
admin.googleapis.com | Every 24 hours | API |
WORKSPACE_PRIVILEGES |
Prerequisites
In order for Google Security Operations to ingest Google Workspace privileges, complete the following steps:
- Enable the Admin SDK API on your Google Cloud project.
- Create a Service Account which will be used to authenticate against the Admin API.
- Generate a JSON key for the Service Account.
- Create a domain-wide delegation for the Service Account with the following OAuth scope:
- Create a Google Workspace user and assign it a super admin role.
- Locate your Google Workspace customer ID.
To learn more about how to configure a feed to ingest Google Workspace logs, see Collect Google Workspace logs.
Type-specific request fields
Field | Required | Description |
---|---|---|
details.workspacePrivilegesSettings.authentication.tokenEndpoint |
Yes | The value of the token_uri field in the JSON key for the service account created to access the admin API. |
details.workspacePrivilegesSettings.authentication.claims.issuer |
Yes | The value of the client_email field in the JSON key for the service account created to access the admin API. |
details.workspacePrivilegesSettings.authentication.claims.subject |
Yes | The email address of the Google Workspace admin user. |
details.workspacePrivilegesSettings.authentication.claims.audience |
Yes | The value of the token_uri field in the JSON key for the service account created to access the admin API. |
details.workspacePrivilegesSettings.authentication.rsCredentials.privateKey |
Yes | The value of the private_key field in the JSON key for the service account created to access the admin API. Note that literal newline characters (\n ) should be replaced with carriage returns. Note that the field name is rsCredentials , and not rsaCredentials . |
details.workspacePrivilegesSettings.workspaceCustomerId |
Yes | The Google Workspace customer ID. Note that the customer ID must have a leading 'C' character. The customer ID may appear differently depending on where in the Google admin console it is found. If the customer ID you have does not have a leading 'C' then prepend what you have with a 'C'. |
Sample Create Feed request
{
"details": {
"feedSourceType": "API",
"logType": "WORKSPACE_PRIVILEGES",
"workspacePrivilegesSettings": {
"authentication": {
"tokenEndpoint": "https://oauth2.googleapis.com/token",
"claims": {
"issuer": "service-account@project.iam.gserviceaccount.com",
"subject": "user@domain.com",
"audience": "https://oauth2.googleapis.com/token"
},
"rsCredentials": {
"privateKey": "-----BEGIN PRIVATE KEY-----
ABCDeFGHIJKLMnopqrsT0u1VWXY...z/abCdefgHIJK+lMN2o345P=
-----END PRIVATE KEY-----"
},
},
"workspaceCustomerId": "C1e2x3ample",
}
}
}
Google Workspace Users
This section provides API reference details for the WORKSPACE_USERS
log type. For details about the data source, see the Google Workspace Users documentation.
Data source | Ingest schedule | details.feedSourceType |
details.logType |
---|---|---|---|
admin.googleapis.com | Every 24 hours | API |
WORKSPACE_USERS |
Prerequisites
For Google Security Operations to ingest Google Workspace ChromeOS devices, complete the following steps:
- Enable the Admin SDK API on your Google Cloud project.
- Create a Service Account which will be used to authenticate against the Admin API.
- Generate a JSON key for the Service Account.
- Create a domain-wide delegation for the Service Account with the following OAuth scope:
- Create a Google Workspace user and assign it an admin role which includes Admin API User read privileges, or create a custom role which includes that privilege.
- Locate your Google Workspace customer ID.
To learn more about how to configure a feed to ingest Google Workspace logs, see Collect Google Workspace logs.
Type-specific request fields
Field | Required | Description |
---|---|---|
details.workspaceUserSettings.authentication.tokenEndpoint |
Yes | The value of the token_uri field in the JSON key for the service account created to access the admin API. |
details.workspaceUserSettings.authentication.claims.issuer |
Yes | The value of the client_email field in the JSON key for the service account created to access the admin API. |
details.workspaceUserSettings.authentication.claims.subject |
Yes | The email address of the Google Workspace admin user with the Admin API User read privilege. |
details.workspaceUserSettings.authentication.claims.audience |
Yes | The value of the token_uri field in the JSON key for the service account created to access the admin API. |
details.workspaceUserSettings.authentication.rsCredentials.privateKey |
Yes | The value of the private_key field in the JSON key for the service account created to access the admin API. Note that literal newline characters (\n ) should be replaced with carriage returns. Note that the field name is rsCredentials , and not rsaCredentials . |
details.workspaceUserSettings.workspaceCustomerId |
Yes | The Google Workspace customer ID. Note that the customer ID must have a leading 'C' character. The customer ID may appear differently depending on where in the Google admin console it is found. If the customer ID you have does not have a leading 'C' then prepend what you have with a 'C'. |
Sample Create Feed request
{
"details": {
"feedSourceType": "API",
"logType": "WORKSPACE_USERS",
"workspaceUserSettings": {
"authentication": {
"tokenEndpoint": "https://oauth2.googleapis.com/token",
"claims": {
"issuer": "service-account@project.iam.gserviceaccount.com",
"subject": "user@domain.com",
"audience": "https://oauth2.googleapis.com/token"
},
"rsCredentials": {
"privateKey": "-----BEGIN PRIVATE KEY-----
ABCDeFGHIJKLMnopqrsT0u1VWXY...z/abCdefgHIJK+lMN2o345P=
-----END PRIVATE KEY-----"
},
},
"workspaceCustomerId": "C1e2x3ample",
}
}
}
Imperva
This section provides API reference details for the IMPERVA_WAF
log type. For details about the data source, see the Imperva documentation.
Data source | Ingest schedule | details.feedSourceType |
---|---|---|
api.imperva.com/audit-trail/v2/events
|
Every 24 hours | API |
Prerequisites
- Get the values for all authentication fields.
- Get the following required permissions:
- None
Type-specific request fields
Field | Required | Description |
---|---|---|
details.impervaWafSettings.authentication.headerKeyValues |
Yes | The HTTP header used to authenticate api.imperva.com in key-value format. |
Optional fields
initialStartTime
Sample Create Feed request
{
"details": {
"feedSourceType": "API",
"logType": "IMPERVA_WAF",
"impervaWafSettings": {
"authentication": {
"headerKeyValues": [{
"key": "key"
"value": "value"
}],
}
}
}
}
Microsoft Azure Active Directory Audit
This section provides API reference details for the AZURE_AD_AUDIT
log type. For details about the data source, see the Azure Active Directory Audit documentation.
Data source | Ingest schedule | details.feedSourceType |
details.logType |
---|---|---|---|
graph.microsoft.com | Every minute | API |
AZURE_AD_AUDIT |
Prerequisites
- Get an Azure AD Premium P1 or P2 license. For more information, see What licenses do I need.
- Get the values for all required request fields. Note that the token endpoint for OAuth 2.0 is: https://login.microsoftonline.com/{tenantId}/oauth2/token
- Get the following required permissions:
- The user whose credentials are used to authenticate against the Microsoft Graph
API to access
directory audits
must have the
permissions
AuditLog.Read.All
andDirectory.Read.All
.
- The user whose credentials are used to authenticate against the Microsoft Graph
API to access
directory audits
must have the
permissions
Type-specific request fields
Field | Required | Description |
---|---|---|
details.azureAdAuditSettings.authentication.clientId |
Yes | Application ID (a UUID) |
details.azureAdAuditSettings.authentication.clientSecret |
Yes | Client Secret |
details.azureAdAuditSettings.tenantId |
Yes | Tenant ID (a UUID) |
details.azureAdAuditSettings.hostname |
No | API Full Path, default value : "graph.microsoft.com/v1.0/auditLogs/directoryAudits" |
Test the API endpoint by using curl
Before you create the feed, you can test the Microsoft Graph API endpoint by using curl.
-
Request an OAuth token to authenticate your request to the API resource.
curl 'https://login.microsoftonline.com/TENANT_ID/oauth2/token' \ --data-urlencode 'grant_type=client_credentials' \ --data-urlencode 'client_id=CLIENT_ID' \ --data-urlencode 'client_secret=CLIENT_SECRET' \ --data-urlencode 'resource=https://graph.microsoft.com'
Replace the following:
- CLIENT_ID: Application ID
- CLIENT_SECRET: Client secret
- TENANT_ID: Tenant ID
The result of the curl request is a JSON response that contains the OAuth access token.
-
Send a request to the Microsoft Graph API endpoint using the OAuth token.
curl 'https://graph.microsoft.com/v1.0/auditLogs/signIns' \ --header 'Accept: application/json' \ --header 'Authorization: Bearer ACCESS_TOKEN'
Replace ACCESS_TOKEN with the value of the OAuth access token that you obtained from the previous step.
Sample Create Feed request
{
"details": {
"feedSourceType": "API",
"logType": "AZURE_AD_AUDIT",
"azureAdAuditSettings": {
"authentication": {
"clientId": "0ab12c34-d5ef-678g-9012-hi34j56k78l9",
"clientSecret": "clientSecret",
}
"tenantId": "0ab123c4-de56-78fg-90h1-ijk2l3456789",
"hostname": "graph.microsoft.com/v1.0/auditLogs/directoryAudits",
}
}
}
Microsoft Azure Active Directory Organizational Context
This section provides API reference details for the AZURE_AD_CONTEXT
log type. For details about the data source, see the Microsoft Graph API List users endpoint, which this feed uses to retrieve device and group data.
Data source | Ingest schedule | details.feedSourceType |
details.logType |
---|---|---|---|
graph.microsoft.com | Every 24 hours | API |
AZURE_AD_CONTEXT |
Prerequisites
- Get the values for all required request fields. The token endpoint for OAuth 2.0 is
https://login.microsoftonline.com/{tenantId}/oauth2/token
- Get the following required permissions:
- The user whose credentials are used to authenticate against Microsoft Graph
API to access organizational context must have permissions
Directory.Read.All
.
- The user whose credentials are used to authenticate against Microsoft Graph
API to access organizational context must have permissions
Type-specific request fields
Field | Required | Description |
---|---|---|
details.azureAdContextSettings.authentication.clientId |
Yes | Application ID (a UUID) |
details.azureAdContextSettings.authentication.clientSecret |
Yes | Client secret |
details.azureAdContextSettings.tenantId |
Yes | Tenant ID (a UUID) |
details.azureAdContextSettings.retrieveDevices |
No | Whether to retrieve device information |
details.azureAdContextSettings.retrieveGroups |
No | Whether to retrieve user group information |
details.azureAdContextSettings.hostname |
No | API Full Path, default value : graph.microsoft.com/beta |
Sample Create Feed request
{
"details": {
"feedSourceType": "API",
"logType": "AZURE_AD_CONTEXT",
"azureAdContextSettings": {
"authentication": {
"clientId": "0ab12c34-d5ef-678g-9012-hi34j56k78l9",
"clientSecret": "clientSecret",
}
"tenantId": "0ab123c4-de56-78fg-90h1-ijk2l3456789",
"retrieveDevices": false,
"retrieveGroups": false,
"hostname": "graph.microsoft.com/beta",
}
}
}
Microsoft Azure Active Directory Sign-ins
This section provides API reference details for the AZURE_AD
log type. For details about the data source, see the Azure Active Directory Sign-ins documentation.
Data source | Ingest schedule | details.feedSourceType |
details.logType |
---|---|---|---|
graph.microsoft.com | Every minute | API |
AZURE_AD |
Prerequisites
- Get an Azure AD Premium P1 or P2 license. For more information, see What licenses do I need.
- Get the values for all required request fields. The token endpoint for OAuth 2.0 is
https://login.microsoftonline.com/{tenantId}/oauth2/token
- Get the following required permissions:
- The user whose credentials are used to authenticate against Microsoft Graph
API to access sign-ins
must have permissions
AuditLog.Read.All
andDirectory.Read.All
.
- The user whose credentials are used to authenticate against Microsoft Graph
API to access sign-ins
must have permissions
Type-specific request fields
Field | Required | Description |
---|---|---|
details.azureAdSettings.authentication.clientId |
Yes | Application ID (a UUID) |
details.azureAdSettings.authentication.clientSecret |
Yes | Client Secret |
details.azureAdSettings.tenantId |
Yes | Tenant ID (a UUID) |
details.azureAdSettings.hostname |
No | API Full Path, default value : graph.microsoft.com/v1.0/auditLogs/signIns |
Sample Create Feed request
{
"details": {
"feedSourceType": "API",
"logType": "AZURE_AD",
"azureAdSettings": {
"authentication": {
"clientId": "0ab12c34-d5ef-678g-9012-hi34j56k78l9",
"clientSecret": "clientSecret",
}
"tenantId": "0ab123c4-de56-78fg-90h1-ijk2l3456789",
"hostname": "graph.microsoft.com/v1.0/auditLogs/signIns",
}
}
}
Microsoft Azure Microsoft Device Management Intune Audit Events
This section provides API reference details for the AZURE_MDM_INTUNE
log type. For details about the data source, see the Azure Microsoft Device Management Intune Audit Events documentation.
Data source | Ingest schedule | details.feedSourceType |
---|---|---|
The Microsoft Graph REST API endpoint URL. The default value is graph.microsoft.com/beta/deviceManagement/auditEvents
|
Every minute | API |
Prerequisites
- Get an active Intune license.
- Get the values for all authentication fields. The token endpoint for OAuth 2.0 is
https://login.microsoftonline.com/{tenantId}/oauth2/token
- Get the following required permission:
- The provisioned OAuth client must have permission
DeviceManagementApps.Read.All
orDeviceManagementApps.ReadWrite.All
.
- The provisioned OAuth client must have permission
Type-specific request fields
Field | Required | Description |
---|---|---|
details.azureMdmIntuneSettings.authentication.clientId |
Yes | The application ID. |
details.azureMdmIntuneSettings.authentication.clientSecret |
Yes | The client secret. |
details.azureMdmIntuneSettings.tenantId |
Yes | The tenant ID, which is a UUID. |
details.azureMdmIntuneSettings.hostname |
No | The Microsoft Graph REST API endpoint URL. The following is the default value: graph.microsoft.com/beta/deviceManagement/auditEvents . |
Sample Create Feed request
{
"details": {
"feedSourceType": "API",
"logType": "AZURE_MDM_INTUNE",
"azureMdmIntuneSettings": {
"authentication": {
"clientId": "0ab12c34-d5ef-678g-9012-hi34j56k78l9",
"clientSecret": "clientSecret",
}
"tenantId": "0ab123c4-de56-78fg-90h1-ijk2l3456789",
"hostname": "graph.microsoft.com/beta/deviceManagement/auditEvents",
}
}
}
Microsoft Graph Security API Alerts
This section provides API reference details for the MICROSOFT_GRAPH_ALERT
log type.
For details about the data source, see Microsoft Graph Security Legacy List alerts
and List alerts_v2.
Data source | Ingest schedule | details.feedSourceType |
---|---|---|
|
Every minute | API |
Prerequisites
- Get the values for authentication fields. The token endpoint for OAuth 2.0 is
https://login.microsoftonline.com/{tenantId}/oauth2/token
- Get the following required permissions; the API supports two data sources:
graph.microsoft.com/v1.0/security/alerts
requiresSecurityEvents.Read.All
permissionsgraph.microsoft.com/beta/security/alerts_v2
orgraph.microsoft.com/v1.0/security/alerts_v2
requiresSecurityAlert.Read.All
permissions- The user whose credentials are used must have
permissions
SecurityEvents.Read.All
.
Type-specific request fields
Field | Required | Description |
---|---|---|
details.microsoftGraphAlertSettings.authentication.clientId |
Yes | Application ID (a UUID) |
details.microsoftGraphAlertSettings.authentication.clientSecret |
Yes | Client secret |
details.microsoftGraphAlertSettings.tenantId |
Yes | Tenant ID (a UUID) |
details.microsoftGraphAlertSettings.authEndpoint |
Yes | The Microsoft Active Directory authentication endpoint. The default value is login.microsoftonline.com . |
details.microsoftGraphAlertSettings.hostname |
No | The API full path. The default value is graph.microsoft.com/v1.0/security/alerts |
Sample Create Feed request
{
"details": {
"feedSourceType": "API",
"logType": "MICROSOFT_GRAPH_ALERT",
"microsoftGraphAlertSettings": {
"authentication": {
"clientId": "0ab12c34-d5ef-678g-9012-hi34j56k78l9",
"clientSecret": "clientSecret",
}
"tenantId": "0ab123c4-de56-78fg-90h1-ijk2l3456789",
"hostname": "graph.microsoft.com/v1.0/security/alerts",
"authEndpoint": "login.microsoftonline.com",
}
}
}
Microsoft Office 365 Management Activity
This section provides API reference details for the OFFICE_365
log type. For details about the data source, see the Microsoft Office 365 Management Activity documentation.
Data source | Ingest schedule | details.feedSourceType |
details.logType |
---|---|---|---|
|
Every minute | API |
OFFICE_365 |
Prerequisites
- Get the values for all required request fields. The token endpoint for OAuth 2.0 is
https://login.microsoftonline.com/{tenantId}/oauth2/token
- Get the following required permissions:
- The user whose credentials are used to authenticate against the API must have permissions
ActivityFeed.Read
. If ingesting DLP data then the permissionActivityFeed.ReadDlp
must be specified.
- The user whose credentials are used to authenticate against the API must have permissions
To learn more about how to configure a feed to ingest Microsoft Office 365 logs, see Collect Microsoft 365 logs.
Type-specific request fields
Field | Required | Description |
---|---|---|
details.office365Settings.authentication.clientId |
Yes | Application ID (a UUID) |
details.office365Settings.authentication.clientSecret |
Yes | Client secret |
details.office365Settings.tenantId |
Yes | Tenant ID (a UUID) |
details.office365Settings.contentType |
Yes | The type of logs to fetch. See below to see the valid values for contentType . |
details.office365Settings.hostname |
No | API Full Path, default value: manage.office.com/api/v1.0 |
Office 365 Content Type
This section provides API reference details for the OFFICE_365
log type. For details about the data source, see the Office 365 Content Type documentation.
details.office365Settings.contentType |
Description |
---|---|
AUDIT_AZURE_ACTIVE_DIRECTORY |
Azure active directory audit logs. |
AUDIT_EXCHANGE |
Azure exchange audit logs. |
AUDIT_SHARE_POINT |
Azure share point audit logs. |
AUDIT_GENERAL
|
All other workloads not included in other Audit content types. |
DLP_ALL |
DLP events only for all workloads. |
Test the API endpoint by using curl
Before you create the feed, you can test the Office 365 Management Activity API endpoint by using curl.
-
Request an OAuth token to authenticate your request to the API resource.
curl 'https://login.microsoftonline.com/TENANT_ID/oauth2/token' \ --data-urlencode 'grant_type=client_credentials' \ --data-urlencode 'client_id=CLIENT_ID' \ --data-urlencode 'client_secret=CLIENT_SECRET' \ --data-urlencode 'resource=https://manage.office.com'
Replace the following:
- CLIENT_ID: Application ID
- CLIENT_SECRET: Client secret
- TENANT_ID: Tenant ID
The result of the curl request is a JSON response that contains the OAuth access token.
-
Send a request to the Office 365 Management Activity API using the OAuth token.
curl 'https://manage.office.com/api/v1.0/TENANT_ID/activity/feed/subscriptions/content?contentType=Audit.AzureActiveDirectory' \ --header 'Authorization: Bearer ACCESS_TOKEN'
Replace ACCESS_TOKEN with the value of the OAuth access token that you obtained from the previous step.
Sample Create Feed request
{
"details": {
"feedSourceType": "API",
"logType": "OFFICE_365",
"office365Settings": {
"authentication": {
"clientId": "0ab12c34-d5ef-678g-9012-hi34j56k78l9",
"clientSecret", "clientSecret",
},
"tenantId": "0ab123c4-de56-78fg-90h1-ijk2l3456789"",
"contentType": "AUDIT_AZURE_ACTIVE_DIRECTORY",
"hostname": "manage.office.com/api/v1.0",
}
}
}
Mimecast
This section provides API reference details for the MIMECAST_MAIL
log type. For details about the data source, see the Mimecast documentation.
Data source | Ingest schedule | details.feedSourceType |
---|---|---|
The fully qualified domain name of your Mimecast API endpoint, such as us-api.mimecast.com .
|
Every minute | API |
Prerequisites
- Get the values for all authentication fields.
- Get the following required permissions:
- None
Type-specific request fields
Field | Required | Description |
---|---|---|
details.mimecastMailSettings.authentication.headerKeyValues |
Yes | The configuration in the key-value format that is used to construct the authentication header. |
details.mimecastMailSettings.hostname |
Yes | The fully qualified domain name of your Mimecast API endpoint, such as us-api.mimecast.com . |
Test the endpoint
The API endpoint used to download Mimecast MTA logs is
/api/audit/get-siem-logs
. To use this endpoint, send a POST request to
/api/audit/get-siem-logs
.
For details on how to use the API, see the sample code in the Mimecast documentation.
Sample Create Feed request
{
"details": {
"feedSourceType": "API",
"logType": "MIMECAST_MAIL",
"mimecastMailSettings": {
"authentication": {
"headerKeyValues": [
{
"key": "access_key",
"value": "ACCESS_KEY"
},
{
"key": "app_id",
"value": "APP_ID"
},
{
"key": "app_key",
"value": "APP_KEY"
},
{
"key": "secret_key",
"value": "SECRET_KEY"
}
]
},
"hostname": "xx-api.mimecast.com"
}
}
}
Netskope Alerts V1
This section provides API reference details for the NETSKOPE_ALERT
log type. For details about the data source, see the Netskope Alerts documentation. Netskope REST API v1 data is supported.
Data source | Ingest schedule | details.feedSourceType |
---|---|---|
API_HOSTNAME/api/v1/alerts
Replace |
Every 10 minutes | API |
Prerequisites
- Get the values for all authentication fields. Use auth tokens for the Netskope REST API v1.
- Get the following required permissions:
- None
Type-specific request fields
Field | Required | Description |
---|---|---|
details.netskopeAlertSettings.authentication.headerKeyValues |
Yes | The HTTP header used to authenticate Netskope in key-value format. |
details.netskopeAlertSettings.hostname |
Yes | The fully qualified domain name of your Netskope REST API endpoint. |
details.netskopeAlertSettings.feedname |
Yes | The REST endpoint to connect to. This can be alerts or events . |
details.netskopeAlertSettings.contentType |
Yes | The value of the type query parameter that determines which type of event or alert is acquired. |
Optional fields
initialStartTime
Test the API endpoint
Before you create the feed, you can test the Netskope alerts API endpoint by
sending a POST request to https://TENANT_URL/api/v1/alerts
. This endpoint returns alerts generated by Netskope.
The following is an example request using curl:
curl -X POST 'https://TENANT_URL/api/v1/alerts?' \ -H 'Content-Type: application/json' \ -d 'timeperiod=86400' \ -d 'type=Security%20Assessment' \ -d 'limit=1' \ -d 'stimeperiod=2592000' \ -d 'query=%28compliance_standards.standard%20eq%20%27CSA-CCM-3.0.1%27%29' \ -d 'token=ACCESS_TOKEN'
Replace the following:
- TENANT_URL: URL of your tenant
- ACCESS_TOKEN: OAuth access token
To learn more about the different query parameters that can be used as a part of the request, see the Get Alerts Data page in the Netskope documentation.
Sample Create Feed request
{
"details": {
"feedSourceType": "API",
"logType": "NETSKOPE_ALERT",
"netskopeAlertSettings": {
"authentication": {
"headerKeyValues": [{
"key": "token",
"value": secret
}]
},
"hostname": hostname,
"feedname": feedname,
"contentType": contenttype
}
},
"display_name": displayname
}
Netskope Alerts V2
This section provides API reference details for the NETSKOPE_ALERT_V2
log type. For details about the data source, see the Netskope Alerts V2 documentation. Netskope REST API v2 data is supported.
Data source | Content Type | Content Category |
---|---|---|
API_HOSTNAME /api/v2/events/dataexport/alerts/uba |
uba | alerts |
API_HOSTNAME /api/v2/events/dataexport/alerts/securityassessment |
securityassessment | alerts |
API_HOSTNAME /api/v2/events/dataexport/alerts/quarantine |
quarantine | alerts |
API_HOSTNAME /api/v2/events/dataexport/alerts/remediation |
remediation | alerts |
API_HOSTNAME /api/v2/events/dataexport/alerts/policy |
policy | alerts |
API_HOSTNAME /api/v2/events/dataexport/alerts/malware |
malware | alerts |
API_HOSTNAME /api/v2/events/dataexport/alerts/malsite |
malsite | alerts |
API_HOSTNAME /api/v2/events/dataexport/alerts/compromisedcredential |
compromisedcredential | alerts |
API_HOSTNAME /api/v2/events/dataexport/alerts/ctep |
ctep | alerts |
API_HOSTNAME /api/v2/events/dataexport/alerts/dlp |
dlp | alerts |
API_HOSTNAME /api/v2/events/dataexport/alerts/watchlist |
watchlist | alerts |
API_HOSTNAME /api/v2/events/dataexport/events/application |
application | events |
API_HOSTNAME /api/v2/events/dataexport/events/audit |
audit | events |
API_HOSTNAME /api/v2/events/dataexport/events/connection |
connection | events |
API_HOSTNAME /api/v2/events/dataexport/events/incident |
incident | events |
API_HOSTNAME /api/v2/events/dataexport/events/infrastructure |
infrastructure | events |
API_HOSTNAME /api/v2/events/dataexport/events/network |
network | events |
API_HOSTNAME /api/v2/events/dataexport/events/page |
page | events |
- Ingest schedule = Every 10 mins
details.feedSourceType
= API
Replace API_HOSTNAME
with the fully qualified
domain name of your Netskope REST API v2 endpoint, such as myinstance.goskope.com
.
Prerequisites
- Get the values for all authentication fields. Use auth tokens for the Netskope REST API v2.
- Create a Netskope access token following the steps on the REST API v2 Overview page. Note, when creating the Netskope token make sure to select all the relevant endpoint privileges.
Type-specific request fields
Field | Required | Description |
---|---|---|
details.netskopeAlertV2Settings.authentication.headerKeyValues |
Yes | The HTTP header used to authenticate Netskope in key-value format. |
details.netskopeAlertV2Settings.hostname |
Yes | The fully qualified domain name of your Netskope REST API endpoint. |
details.netskopeAlertV2Settings.contentCategory |
Yes | The REST endpoint to connect to. This can be alerts or events . |
details.netskopeAlertV2Settings.contentTypes |
Yes | The type of event or alert. Allowed values for alerts are uba, securityassessment, quarantine, remediation, policy, malware, malsite, compromisedcredential, ctep, dlp and watchlist. Allowed values for events are application, audit, connection, incident, infrastructure, network and page. |
Test the API endpoint
Before you create the feed, you can test the Netskope alerts V2 API endpoint by
sending a GET request to https://TENANT_URL
. This endpoint returns alerts generated by Netskope.
The following is an example request using curl:
curl -X 'GET' \ 'https://TENANT_URL' \ -H 'accept: application/json' \ -H 'Netskope-Api-Token: ACCESS_TOKEN'
Replace the following:
- TENANT_URL: URL of one of the Data sources listed in the Data source table.
- ACCESS_TOKEN: OAuth access token (See Prerequisites for details of creating the token.)
To learn more about the different query parameters that can be used as a part of the request, see the Get Alerts Data page in the Netskope documentation.
Sample Create Feed request
{
"details": {
"feedSourceType": "API",
"logType": "NETSKOPE_ALERT_V2",
"netskopeAlertV2Settings": {
"authentication": {
"headerKeyValues": [{
"key": "Netskope-Api-Token",
"value": "token_value"
}]
},
"contentTypes": [
"uba",
"securityassessment"
],
"hostname": "myinstance.goskope.com",
"contentCategory": "alerts"
}
}
}
Okta System Log
This section provides API reference details for the OKTA
log type. For details about the data source, see the Okta System Log documentation.
Data source | Ingest schedule | details.feedSourceType |
---|---|---|
API_HOSTNAME/api/v1/logs
Replace |
Every minute | API |
Prerequisites
- Get the values for all authentication fields.
- Get the following required permissions:
- None
Type-specific request fields
Field | Required | Description |
---|---|---|
details.oktaUserContextSettings.authentication.headerKeyValues |
Yes | The HTTP header used to authenticate Okta in key-value format. |
details.oktaUserContextSettings.hostname |
Yes | The fully qualified domain name of your Okta instance. |
Test the API endpoint
Before you create the feed, you can test the Okta System Log API endpoint by
sending a GET request to OKTA_URL/api/v1/logs
. This endpoint returns system log events that can be ingested into a SIEM platform.
The following is an example request to obtain system log events from a particular point of time in the past:
curl -v -X GET \ -H "Accept: application/json" \ -H "Content-Type: application/json" \ -H "Authorization: SSWS API_TOKEN" \ "https://OKTA_URL/api/v1/logs?since=DATETIME"
Replace the following:
- API_TOKEN: OAuth access token
- OKTA_URL: fully qualified domain name of your Okta instance, such
as
example.okta.com
- DATETIME: timestamp in UTC format according to ISO 8601, separating date and time
with a
T
. For example:2024-01-31T00:00:00Z
. The API will fetch the logs recorded after the specified timestamp.
Sample Create Feed request
{
"details": {
"feedSourceType": "API",
"logType": "OKTA",
"oktaSettings": {
"authentication": {
"headerKeyValues": [{
"key": "Authorization",
"value": "APITOKEN"
}]
},
"hostname": "hostname"
}
}
}
Okta Users
This section provides API reference details for the OKTA_USER_CONTEXT
log type. For details about the data source, see the Okta Users documentation.
Data source | Ingest schedule | details.feedSourceType |
---|---|---|
API_HOSTNAME/api/v1/logs
Replace |
Every minute | API |
Prerequisites
- Get the values for
hostname
and all authentication fields. - Get the following required permissions:
- None
Type-specific request fields
Field | Required | Description |
---|---|---|
details.oktaUserContextSettings.authentication.headerKeyValues |
Yes | The HTTP header used to authenticate Okta in key-value format. |
details.oktaUserContextSettings.hostname |
Yes | The fully qualified domain name of your Okta instance. |
details.oktaUserContextSettings.managerIdReferenceField |
No | This ID is required when you use a non Okta ID to reference managers. |
Sample Create Feed request
managerIdReferenceField
is required when you use a non-Okta ID to reference
managers. It should be a JSON field path pointing to the field that contains
the manager ID in the result of a call to the "users" Okta API.
{
"details": {
"feedSourceType": "API",
"logType": "OKTA_USER_CONTEXT",
"oktaSettings": {
"authentication": {
"headerKeyValues": [{
"key": "Authorization",
"value": "APITOKEN"
}]
},
"hostname": "hostname",
"managerIdReferenceField": "fooId"
}
}
}
Palo Alto Networks AutoFocus
This section provides API reference details for the PAN_IOC
log type. For details about the data source, see the Palo Alto Networks AutoFocus documentation.
Data source | Ingest schedule | details.feedSourceType |
---|---|---|
autofocus.paloaltonetworks.com/api/v1.0/IOCFeed/FEED_ID/FEED_NAME
Replace |
Every five minutes | API |
Prerequisites
- Get the values for
feedId
,feed
, and all authentication fields. - Get the following required permissions:
- None
Test the endpoint
To get the results for a custom threat indicator feed, you need to send a
request to the custom feed resource of the AutoFocus API. The custom feed endpoint is as follows: /IOCFeed/OUTPUT_FEED_ID/OUTPUT_FEED_NAME
.
The following is an example request to retrieve threat intelligence:
curl -X GET \ -H "apiKey:API_KEY" \ https://autofocus.paloaltonetworks.com/api/v1.0/IOCFeed/OUTPUT_FEED_ID/OUTPUT_FEED_NAME?limit=MAX_ENTRIES
Replace the following:
API_KEY: API key tied to your license
OUTPUT_FEED_ID: custom threat feed ID number
OUTPUT_FEED_NAME: name of the custom feed
MAX_ENTRIES: maximum number of indicator entries displayed in the output
For details on how to use the Palo Alto AutoFoucs API, see the Get Custom Threat Indicator Feed documentation.
Sample Create Feed request
{
"details": {
"feedSourceType": "API",
"logType": "PAN_IOC",
"panIocSettings": {
"authentication": {
"headerKeyValues": [{
"key": "key"
"value": "value"
}],
}
"feedId": "ID",
"feed": "feed"
}
}
}
Palo Alto Networks Cortex XDR
This section provides API reference details for the CORTEX_XDR
log type. For details about the data source, see the Palo Alto Networks Cortex XDR documentation.
Data source | Ingest schedule | details.feedSourceType |
---|---|---|
API_HOSTNAME/public_api/v1/incidents/get_incidents
Replace |
Every five minutes | API |
Prerequisites
- Get the values for all authentication fields.
- Make sure the API key is an advanced key, not a standard key.
- Get the following required permissions:
- None
Type-specific request fields
Field | Required | Description |
---|---|---|
details.cortexXdrSettings.authentication.headerKeyValues |
Yes | The HTTP header used to authenticate Cortex XDR API in key-value format. |
details.cortexXdrSettings.hostname |
Yes | The fully qualified domain name of your Cortex XDR instance. |
details.cortexXdrSettings.endpoint |
No | The API endpoint to connect to retrieve logs, which include incidents or alerts . |
Optional fields
initialStartTime
Sample Create Feed request
{
"details": {
"feedSourceType": "API",
"logType": "CORTEX_XDR",
"cortexXdrSettings": {
"authentication": {
"headerKeyValues": [{
"key": "Authorization"
"value": "api_key"
},
{
"key": "x-xdr-auth-id"
"value": "api_key_id"
}
],
},
"hostname": "api-abcd.xdr.ab.paloaltonetworks.com",
"endpoint": "incidents"
}
}
}
Palo Alto Networks Prisma Cloud Audit Logs
This section provides API reference details for the PAN_PRISMA_CLOUD
log type. For details about the data source, see the Palo Alto Networks Prisma Cloud Audit Logs documentation.
Data source | Ingest schedule | details.feedSourceType |
---|---|---|
api.prismacloud.io/audit/redlock
|
Every five minutes | API |
Prerequisites
- Get the values for all authentication fields.
- Get the following required permissions:
- None
Type-specific request fields
Field | Required | Description |
---|---|---|
details.panPrismaCloudSettings.authentication.user |
Yes | The Prisma Cloud username. |
details.panPrismaCloudSettings.authentication.password |
Yes | The Prisma Cloud password. |
details.panPrismaCloudSettings.hostname |
Yes | The Palo Alto Prisma Cloud API hostname. |
Test the endpoints by using curl
Before you create the feed, you can test the API endpoints by using curl.
Send a GET request to https://api.prismacloud.io/audit/redlock
The following example returns audit logs for events that took place on the Prisma Cloud platform:
curl -L 'https://api.prismacloud.io/audit/redlock' \ -H 'Accept: application/json; charset=UTF-8' \ -H 'x-redlock-auth: API_KEY_VALUE'
Replace the following:
- API_KEY_VALUE: The Prisma Cloud authentication value is a JSON Web Token (JWT).
Optional fields
timeType
, timeAmount
, timeUnit
For details about the data source, see the Palo Alto Networks Prisma Cloud Audit Logs documentation.
Sample Create Feed request
{
"details": {
"feedSourceType": "API",
"logType": "PAN_PRISMA_CLOUD",
"panPrismaCloudSettings": {
"authentication": {
"user": "user",
"password": "password"
},
"hostname": "api2.prismacloud.io"
}
}
}
Proofpoint on Demand
This section provides API reference details for the PROOFPOINT_ON_DEMAND
log type. For details about the data source, see the Proofpoint on Demand documentation.
Data source | Ingest schedule | details.feedSourceType |
---|---|---|
The default data endpoint is used. | Every hour | API |
Prerequisites
- Get the values for all authentication fields.
- Make sure that the token is not used in any other instance or connection, whether inside or outside Google SecOps, as Proofpoint limits tokens to one active session.
- Get the following required permissions:
- None
Type-specific request fields
Field | Required | Description |
---|---|---|
details.proofpointOnDemandSettings.authentication.headerKeyValues |
Yes | The HTTP header used to authenticate logstream.proofpoint.com in the key-value format. |
details.proofpointOnDemandSettings.clusterId |
Yes | The cluster ID, which is a user group string. |
Other fields
proofpointOnDemandSourceDetails
Optional fields
initialStartTime
Test the endpoint
Before you create a real-time email processing log feed, you can test connectivity between your system and the Proofpoint on Demand (PoD) Log API.
The following is an example request to receive uncompressed data:
curl -i --no-buffer \ -H "Connection: Upgrade" \ -H "Upgrade: websocket" \ -H "Host: logstream.proofpoint.com:443" \ -H "Authorization: Bearer ACCESS_TOKEN " \ -H "Sec-WebSocket-Key: KEY" \ -H "SecWebSocket-Version: 13" \ "https://logstream.proofpoint.com:443/v1/stream?cid=CLUSTER_ID&type=message&sinceTime=DATE_TIME"
Replace the following:
ACCESS_TOKEN: a token provided by Proofpoint for a customer cluster to authenticate with the service.
KEY: a base64-encoded key used in the WebSocket opening handshake.
CLUSTER_ID: the cluster ID assigned by Proofpoint.
DATE_TIME: start time to begin streaming log data, in ISO 8601 format, which includes timezone information. For example:
2018-08-31T00:00:00-0800
. The API fetches the logs recorded after the specified timestamp.
Sample Create Feed request
{
"details": {
"feedSourceType": "API",
"logType": "PROOFPOINT_ON_DEMAND",
"proofpointOnDemandSettings": {
"authentication": {
"user": "user",
"secret": "secret"
},
"clusterId": "ID"
}
}
}
Proofpoint TAP
This section provides API reference details for the PROOFPOINT_MAIL
log type. For details about the data source, see the Proofpoint SIEM API documentation.
Data source | Ingest schedule | details.feedSourceType |
---|---|---|
The default data endpoint is used. | Every 24 hours | API |
Prerequisites
- Get the values for all authentication fields.
- Get the following required permissions:
- None
Type-specific request fields
Field | Required | Description |
---|---|---|
details.proofpointMailSettings.authentication.user |
Yes | The user account required for authentication. |
details.proofpointMailSettings.authentication.secret |
Yes | The secret required for authentication. |
Test the endpoint
Before you create the feed, you can test the Proofpoint TAP SIEM API endpoint by
sending a GET request to /v2/siem/all
.
To fetch events for all clicks and messages relating to known threats within the specified time period, use a GET request as follows:
curl \ "https://tap-api-v2.proofpoint.com/v2/siem/all?format=syslog&sinceSeconds=SECONDS" \ --user "PRINCIPAL:SECRET" \ -s
Replace the following:
SECONDS: an integer representing a time window in seconds from the current API server time. For example,
3600
.PRINCIPAL: Proofpoint service principal to authenticate to the SIEM API.
SECRET: Proofpoint API secret to authenticate to the SIEM API.
To learn more about the different query parameters that can be used as a part of the request, see the Proopoint TAP SIEM API documentation.
Sample Create Feed request
{
"details": {
"feedSourceType": "API",
"logType": "PROOFPOINT_MAIL",
"proofpointMailSettings": {
"authentication": {
"user": "user",
"secret": "secret"
}
}
}
}
Qualys VM
This section provides API reference details for the QUALYS_VM
log type. For
details about the data source, see the
Qualys VM documentation (PDF).
Data source | Ingest schedule | details.feedSourceType |
---|---|---|
The domain and full path of the resource, such as qualysapi.qualys.com/api/2.0/fo/asset/host/?action=list .
|
Every minute | API |
Prerequisites
- Get the values for all authentication fields.
- Get the following required permissions:
- None
Type-specific request fields
Field | Required | Description |
---|---|---|
details.qualysVmSettings.authentication.user |
Yes | The user account required for authentication. |
details.qualysVmSettings.authentication.secret |
Yes | The secret required for authentication. |
details.qualysVmSettings.hostname |
Yes | The domain and full path of the resource, such as qualysapi.qualys.com/api/2.0/fo/asset/host/?action=list . |
Test the endpoints by using curl
Before you create the feed, you can test the API endpoints by using curl.
-
To test the endpoint for the Qualys VM Host List API, use
the following
curl
command:curl -H "X-Requested-With: Curl Sample" -u "USERNAME:PASSWORD" \ "https://qualysapi.qg3.apps.qualys.com/api/2.0/fo/asset/host/?action=list"
-
To test the endpoint for the Qualys VM Host List Detection API, use
the following
curl
command:curl -H "X-Requested-With: Curl Sample" -u "USERNAME:PASSWORD" \ "https://qualysapi.qg3.apps.qualys.com/api/2.0/fo/asset/host/vm/detection/?action=list"
Replace the following:
- USERNAME: username of your Qualys account
- PASSWORD: password of your Qualys account
Sample Create Feed request for Qualys VM Host List API
{
"details": {
"feedSourceType": "API",
"logType": "QUALYS_VM",
"qualysVmSettings": {
"authentication": {c
"user": "USERNAME",
"secret": "PASSWORD"
},
"hostname": "qualysapi.qualys.com/api/2.0/fo/asset/host/?action=list"
}
}
}
Sample Create Feed request for Qualys VM Host List Detection API
{
"details": {
"feedSourceType": "API",
"logType": "QUALYS_VM",
"qualysVmSettings": {
"authentication": {
"user": "USERNAME",
"secret": "PASSWORD"
},
"hostname": "qualysapi.qualys.com/api/2.0/fo/asset/host/vm/detection/?action=list"
}
}
}
Qualys Scan
This section provides API reference details for the QUALYS_SCAN
log type. For
details about the data source, see the
Qualys VM documentation (PDF).
Data source | Ingest schedule | details.feedSourceType
|
details.logType
|
---|---|---|---|
qualysapi.qualys.com | Every Day | API
|
QUALYS_SCAN
|
Prerequisites
- Get the values for all authentication fields.
- Get the following required permissions:
- Ensure API access is enabled for the user.
Scan APIs
The Qualys Scan APIs that Google Security Operations supports include the following.
details.qualysScanSettings.api_type |
Description |
---|---|
SCAN_SUMMARY_OUTPUT |
Scan Summaries API to identify which hosts were scanned or not scanned and why. |
SCAN_COMPLIANCE_OUTPUT |
Scan Compliance API to list out the compliance scans in your Qualys account. |
SCAN_COMPLIANCE_CONTROL_OUTPUT |
Compliance Control API to view a list of compliance controls which are visible to the user. |
Test the endpoints by using curl
Before you create the feed, you can test the API endpoints by using curl.
-
To test the endpoint for the API type
SCAN_SUMMARY_OUTPUT
, use the followingcurl
command:curl -H "X-Requested-With: Curl Sample" -u "USERNAME:PASSWORD" \ "https://qualysapi.qg3.apps.qualys.com/api/2.0/fo/scan/vm/summary/?action=list&scan_datetime_since=DATETIME"
-
To test the endpoint for the API type
SCAN_COMPLIANCE_OUTPUT
, use the followingcurl
command:curl -H "X-Requested-With: Curl Sample" -u "USERNAME:PASSWORD" \ "https://qualysapi.qg3.apps.qualys.com/api/2.0/fo/scan/compliance/?action=list&launched_after_datetime=DATETIME"
-
To test the endpoint for the API type
SCAN_COMPLIANCE_CONTROL_OUTPUT
, use the followingcurl
command:curl -H "X-Requested-With: Curl Sample" -u "USERNAME:PASSWORD" \ "https://qualysapi.qg3.apps.qualys.com/api/2.0/fo/compliance/control/?action=list&updated_after_datetime=DATETIME"
Replace the following:
- USERNAME: username of your Qualys account
- PASSWORD: password of your Qualys account
- DATETIME: timestamp in UTC format according to ISO 8601, separating date and time
with a
T
. For example:2024-01-31T18:00:42Z
. The API will fetch the logs recorded after the specified timestamp.
Sample Create Feed request for Qualys Scan API
{
"details": {
"feedSourceType": "API",
"logType": "QUALYS_SCAN",
"qualysScanSettings": {
"authentication": {
"user": "USERNAME",
"secret": "PASSWORD"
},
"hostname": "qualysapi.qualys.com",
"api_type": "SCAN_SUMMARY_OUTPUT"
}
}
}
Rapid7 InsightVM
This section provides API reference details for the RAPID7_INSIGHT
log type. For details about the data source, see the Rapid7 InsightVM documentation.
Data source | Ingest schedule | details.feedSourceType |
---|---|---|
The REST endpoint, which must be either vulnerabilities or assets .
|
Every minute | API |
Prerequisites
- Get the values for all authentication fields.
- Get the following required permissions:
- None
Type-specific request fields
Field | Required | Description |
---|---|---|
details.rapid7InsightSettings.authentication.headerKeyValues |
Yes | The HTTP header used to authenticate us.api.insight.rapid7.com in key-value format. |
details.rapid7InsightSettings.endpoint |
Yes | The REST endpoint to connect to. The endpoint value must be either vulnerabilities or assets . |
details.rapid7InsightSettings.hostname |
No | The fully qualified domain name of the Rapid7 endpoint, such as us.api.insight.rapid7.com . |
Optional fields
initialStartTime
Sample Create Feed request
{
"details": {
"feedSourceType": "API",
"logType": "RAPID7_INSIGHT",
"rapid7InsightSettings": {
"authentication": {
"headerKeyValues": [{
"key": "X-Api-Key",
"value": ApiToken "API_TOKEN"
}],
},
"endpoint": "assets"
"hostname": "us.api.insight.rapid7.com"
}
}
}
Replace API_TOKEN with your API token.
Recorded Future
This section provides API reference details for the RECORDED_FUTURE_IOC
log type. For details about the data source, see the Recorded Future documentation.
Data source | Ingest schedule | details.feedSourceType |
---|---|---|
api.recordedfuture.com/v2/fusion/files .
|
Every two hours | API |
Prerequisites
- Get the values for all authentication fields.
- Get the following required permissions:
- None
Type-specific request fields
Field | Required | Description |
---|---|---|
details.recordedFutureIocSettings.authentication.headerKeyValues |
Yes | The HTTP header used to authenticate to api.recordedfuture.com in key-value format. |
Sample Create Feed request
{
"details": {
"feedSourceType": "API",
"logType": "RECORDED_FUTURE_IOC",
"recordedFutureIocSettings": {
"authentication": {
"user": "user",
"secret": "secret"
},
}
}
}
RH-ISAC
This section provides API reference details for the RH_ISAC_IOC
log type. For details about the data source, see the RH-ISAC documentation.
Data source | Ingest schedule | details.feedSourceType |
---|---|---|
api.trustar.co/api/1.3/indicators/search . |
Every 24 hours | API |
Prerequisites
- Get the values for all authentication fields.
- Get the following required permissions:
- None
Type-specific request fields
Field | Required | Description |
---|---|---|
details.rhIsacIocSettings.authentication.tokenEndpoint |
Yes | The endpoint to retrieve the OAuth token. |
details.rhIsacIocSettings.authentication.clientId |
Yes | The application ID. |
details.rhIsacIocSettings.authentication.clientSecret |
Yes | The client secret. |
Other fields
tags
, queueDelay
Optional fields
initialStartTime
Sample Create Feed request
{
"details": {
"feedSourceType": "API",
"logType": "RH_ISAC_IOC",
"rhIsacIocSettings": {
"authentication": {
"tokenEndPoint": "endpoint",
"clientId": "clientId",
"clientSecret": "clientSecret"
}
}
}
}
Salesforce
This section provides API reference details for the SALESFORCE
log type. For details about the data source, see the Salesforce documentation.
Data source | Ingest schedule | details.feedSourceType |
---|---|---|
API_HOSTNAME/services/data/v50.0/query
Replace |
Every minute | API |
Prerequisites
- Salesforce Shield is required.
- Get the values for all authentication fields as described in OAuth 2.0 Username-Password Flow for Special Scenarios.
- Get the following required permissions:
- None
Type-specific request fields
Field | Required | Description |
---|---|---|
details.salesforceSettings.hostname |
Yes | The fully qualified domain name of your Salesforce REST API endpoint, such as myinstance.salesforce.com . |
details.salesforceSettings.oauthPasswordGrantAuth.tokenEndpoint |
No | The endpoint to retrieve the OAuth token. This field must be specified in the following format: https://SF_INSTANCE.my.salesforce.com/services/oauth2/token?grant_type=password . Replace SF_INSTANCE with your Salesforce instance name. This field is required only for OAuth password grant. |
details.salesforceSettings.oauthPasswordGrantAuth.clientId |
No | The application ID. This field is required only for OAuth password grant. |
details.salesforceSettings.oauthPasswordGrantAuth.clientSecret |
No | The client secret. This field is required only for OAuth password grant. |
details.salesforceSettings.oauthPasswordGrantAuth.user |
No | The username used for authentication. This field is required only for OAuth password grant. |
details.salesforceSettings.oauthPasswordGrantAuth.password |
No | The password used for authentication. This field is required only for OAuth password grant. |
details.salesforceSettings.oauthJwtCredentials.tokenEndpoint |
No | The endpoint to retrieve the OAuth JSON web token. This field is required only for OAuth JWT grant. |
details.salesforceSettings.oauthJwtCredentials.claims.issuer |
No | The JWT claims issuer, which is usually a client ID. This field is required only for OAuth JWT grant. |
details.salesforceSettings.oauthJwtCredentials.claims.subject |
No | The JWT claims subject, which is usually an email ID. This field is required only for OAuth JWT grant. |
details.salesforceSettings.oauthJwtCredentials.claims.audience |
No | The JWT claims audience. This field is required only for OAuth JWT grant. |
details.salesforceSettings.oauthJwtCredentials.rsCredentials.privateKey |
No | An RSA private key in PEM format. This field is required only for OAuth JWT grant. |
Optional fields
initialStartTime
Test the endpoint
Before you create the feed, you can test the REST API endpoint by
sending a GET request to /services/data/vAPI_VERSION/query
.
The Query
resource is used to retrieve field
values from a record.
To query event monitoring records based on fields, such as LogDate
and EventType
, use a GET request as follows:
curl https://SUBDOMAIN.my.salesforce.com/services/data/vAPI_VERSION/query \ -X GET \ -H "Authorization: Bearer AUTH_TOKEN" \ -G \ --data-urlencode "q=SELECT Id, EventType, LogFile, LogDate, LogFileLength FROM EventLogFile WHERE LogDate > Yesterday AND EventType = 'API'"
Replace the following:
- SUBDOMAIN: the subdomain name relevant to the Salesforce instance being accessed.
- API_VERSION: version number of the API endpoint. For example,
60.0
. - AUTH_TOKEN: OAuth access token.
Sample Create Feed request using OAuth password grant
{
"details": {
"feedSourceType": "API",
"logType": "SALESFORCE",
"salesforceSettings": {
"authentication": {
"tokenEndpoint": "endpoint",
"clientId": "clientId",
"clientSecret": "clientSecret",
"user": "user",
"password": "password"
},
"hostname": "hostname"
}
}
}
Sample Create Feed request using OAuth JWT grant
{
"details": {
"feedSourceType": "API",
"logType": "SALESFORCE",
"salesforceSettings": {
"authentication": {
"tokenEndpoint": "endpoint",
"issuer": "clientId",
"subject": "emailID",
"audience": "audience",
"privateKey": "RSAKey"
},
"hostname": "hostname"
}
}
}
SentinelOne Alert
This section provides API reference details for the SENTINELONE_ALERT
log type. For details about the data source, see the SentinelOne Alert documentation.
Data source | Ingest schedule | details.feedSourceType |
---|---|---|
API_HOSTNAME/web/api/v2.1/cloud-detection/alerts
Replace |
Every five minutes | API |
Prerequisites
- Get the values for all authentication fields.
- Get the following required permissions:
- None
Type-specific request fields
Field | Required | Description |
---|---|---|
details.sentineloneAlertSettings.authentication.headerKeyValues |
Yes | The HTTP headers to authenticate the SenitnelOne alerts, threats, and static-indicator API in key-value format. |
details.sentineloneAlertSettings.hostname |
Yes | The fully qualified domain name of the SenitnelOne API. |
details.sentineloneAlertSettings.initialStartTime |
No | The time when the alerts must be fetched. |
details.sentineloneAlertSettings.isAlertApiSubscribed |
No | Indicates whether the alerts API is subscribed. |
Sample Create Feed request
{
"details": {
"feedSourceType": "API",
"logType": "SENTINELONE_ALERT",
"sentineloneAlertSettings": {
"authentication": {
"headerKeyValues": [{
"key": "Authorization",
"value": "ApiToken"
}]
},
"hostname": "hostname",
"isAlertApiSubscribed": false
}
}
}
ServiceNow CMDB
This section provides API reference details for the SERVICENOW_CMDB
log type. For details about the data source, see the ServiceNow CMDB documentation.
Data source | Ingest schedule | details.feedSourceType |
---|---|---|
The fully qualified domain name of your ServiceNow REST API endpoint, such as myinstance.servicenow.com .
|
Every 24 hours | API |
Prerequisites
- Get the values for all the required fields.
- Get the following required permissions:
- None
Type-specific request fields
Field | Required | Description |
---|---|---|
details.serviceNowCmdbSettings.authentication.user |
Yes | The username required for authentication. |
details.serviceNowCmdbSettings.authentication.secret |
Yes | The secret required for authentication. |
details.serviceNowCmdbSettings.hostname |
Yes | The fully qualified domain name of your ServiceNow REST API endpoint, such as myinstance.servicenow.com . |
details.serviceNowCmdbSettings.feedname |
Yes | The ServiceNow table, which corresponds to a collection of records. |
Sample Create Feed request
{
"details": {
"feedSourceType": "API",
"logType": "SERVICENOW_CMDB",
"servicenowCmdbSettings": {
"authentication": {
"user": "user",
"secret": "secret"
},
"hostname": "hostname",
"feedname": "feedname"
}
}
}
Symantec Event Export
This section provides API reference details for the SYMANTEC_EVENT_EXPORT
log type. For details about the data source, see the Symantec Event Export documentation.
Data source | Ingest schedule | details.feedSourceType |
---|---|---|
The Symantec Event Export API. | Every hour | API |
Prerequisites
- Get the values for all authentication fields.
- Get the following required permissions:
- None
Type-specific request fields
Field | Required | Description |
---|---|---|
details.symantecEventExportSettings.authentication.tokenEndpoint |
Yes | The endpoint to retrieve the OAuth token. |
details.symantecEventExportSettings.authentication.clientId |
Yes | The application ID. |
details.symantecEventExportSettings.authentication.clientSecret |
Yes | The client secret. |
details.symantecEventExportSettings.authentication.refreshToken |
Yes | An OAuth 2.0 token used to refresh access tokens when they expire. |
Sample Create Feed request
{
"details": {
"feedSourceType": "API",
"logType": "SYMANTEC_EVENT_EXPORT",
"symantecEventExportSettings ": {
"authentication": {
"tokenEndPoint": "REFRESH TOKEN URI",
"clientId": "CLIENT ID",
"clientSecret": "CLIENT SECRET",
"refreshToken": "REFRESH TOKEN",
}
}
}
}
Thinkst Canary
This section provides API reference details for the THINKST_CANARY
log type. For details about the data source, see the Thinkst Canary documentation.
Data source | Ingest schedule | details.feedSourceType |
---|---|---|
API_HOSTNAME/api/v1/incidents/all
Replace |
Every 30 minutes | API |
Prerequisites
- Get the values for all the required fields.
- Get the following required permissions:
- None
Type-specific request fields
Field | Required | Description |
---|---|---|
details.thinkstCanarySettings.authentication.headerKeyValues |
Yes | The HTTP headers in key-value format. |
details.thinkstCanarySettings.hostname |
Yes | The fully qualified domain name of the Thinkst Canary REST API endpoint, such as myinstance.canary.tools . |
Test the API endpoint
Before you create the feed, you can test the Canary API endpoint by
sending a GET request to DOMAIN/api/v1/incidents/all
.
The following is an example request to get all incidents:
curl https://DOMAIN.canary.tools/api/v1/incidents/all \ -d auth_token=AUTH_TOKEN \ -d limit=1 \ -G
Replace the following:
- DOMAIN: unique hash identifying your Canary Console
- AUTH_TOKEN: OAuth access token
To learn more about the different query parameters that can be used as a part of the request, see the Canary API documentation.
Sample Create Feed request
{
"details": {
"feedSourceType": "API",
"logType": "THINKST_CANARY",
"thinkstCanarySettings": {
"authentication": {
"user": "user",
"secret": "secret"
},
"hostname": "hostname"
}
}
}
ThreatConnect
This section provides API reference details for the THREATCONNECT_IOC
log type. For details about the data source, see the ThreatConnect documentation.
Data source | Ingest schedule | details.feedSourceType |
---|---|---|
The fully qualified domain name of the ThreatConnect REST API endpoint, such as myinstance.threatconnect.com .
|
Every five minutes | API |
Prerequisites
- Get the values for all the required fields.
- Get the following required permissions:
- None
Type-specific request fields
Field | Required | Description |
---|---|---|
details.threatConnectIocSettings.authentication.user |
Yes | The username required for authentication. |
details.threatConnectIocSettings.authentication.password |
Yes | The password required for authentication |
details.threatConnectIocSettings.hostname |
Yes | The fully qualified domain name of the ThreatConnect REST API endpoint, such as myinstance.threatconnect.com . |
details.threatConnectIocSettings.owners |
Yes | All the owner names, where an owner identifies a collection of IoCs. |
Other fields
queueDelay
Optional fields
initialStartTime
Sample Create Feed request
{
"details": {
"feedSourceType": "API",
"logType": "THREATCONNECT_IOC",
"threatConnectIocSettings": {
"authentication": {
"user": "user",
"secret": "secret"
},
"hostname": "hostname",
"owners": [{
"owner"
}]
}
}
}
Workday
This section provides API reference details for the WORKDAY
log type. For details about the data source, see the Workday Administrator Guide (Integrations > Workday REST API).
Data source | Ingest schedule | details.feedSourceType |
details.logType |
---|---|---|---|
|
Every 24 hours | API |
WORKDAY |
Prerequisites
- In the Workday documentation for configuring OAuth 2.0 for your REST API client, follow the steps in Register API Clients.
- Ensure that the Workday administrator provides you the
Get
andView
permissions for the required security domain policies and provides access to the Workday API endpoints.
Type-specific request fields
Field | Required | Description |
---|---|---|
details.workdaySettings.authentication.secret |
Yes | The access token generated by Workday after completing the steps to register
OAuth 2.0 clients with Workday.
To set up authentication, you must either specify the access token or all of the following OAuth fields: token endpoint, client ID, client secret, and refresh token. |
details.workdaySettings.authentication.tokenEndpoint" |
Yes | The endpoint from which to get the access token. |
details.workdaySettings.authentication.user |
Yes | The client ID generated by Workday after completing the steps to register OAuth 2.0 clients with Workday. |
details.workdaySettings.authentication.secret |
Yes | The client secret generated by Workday after completing the steps to register OAuth 2.0 clients with Workday. |
details.workdaySettings.authentication.refreshToken |
Yes | The refresh token generated by Workday after completing the steps to register OAuth 2.0 clients with Workday. |
details.workdaySettings.hostname |
Yes | The hostname of the Workday REST web service. Example: services1.workday.com . |
details.workdaySettings.tenantId |
Yes | The name of the tenant. |
Sample Create Feed request
The following sample uses a token endpoint, client ID, client secret, and refresh token:
{
"details": {
"feedSourceType": "API",
"logType": "WORKDAY",
"workdaySettings": {
"authentication": {
"tokenEndpoint": "TokenEndpoint",
"user": "ClientID",
"clientSecret": "ClientSecret"
"refreshToken": "RefreshToken"
},
"hostname": "hostname",
"tenantId": "ID"
}
}
}
The following sample uses an access token:
{
"details": {
"feedSourceType": "API",
"logType": "WORKDAY",
"workdaySettings": {
"authentication": {
"secret": "AccessToken"
},
"hostname": "hostname",
"tenantId": "ID"
}
}
}