Use ingestion scripts deployed as Cloud Functions
Chronicle has provided a set of ingestion scripts, written in Python, that are intended to be deployed as Cloud Functions. These scripts enable you to ingest data from the following log sources, listed by name and log type.
- Citrix audit logs (CITRIX_MONITOR)
- Duo Admin (DUO_ADMIN)
- One Login User Context (ONELOGIN_USER_CONTEXT)
- MISP (MISP_IOC)
- Citrix session metadata (CITRIX_SESSION_METADATA)
- Slack Audit (SLACK_AUDIT)
- Box (BOX)
- OneLogin (ONELOGIN_SSO)
- Google Cloud Pub/Sub
- STIX/TAXII threat intelligence (STIX)
These scripts are located in the Chronicle GitHub repository.
Known limitation: When these scripts are used in a stateless environment such as Cloud Functions, they may not send all logs to Chronicle because they lack checkpoint functionality. Chronicle has tested the scripts with the Python 3.9 runtime.
Before you begin
Read the following resources that provide context and background information that enable you to use the Chronicle ingestion scripts effectively.
- Deploying Cloud Functions for information about how to deploy Cloud Functions from your local machine.
- Creating and accessing secrets explains how to use Secret Manager. You will need this to store and access the Chronicle service account JSON file.
- Install the Google Cloud CLI. You will use this to deploy the Cloud Function.
- Google Cloud Pub/Sub documentation if you plan to ingest data from Pub/Sub.
Assemble the files for a single log type
Each sub-directory in Chronicle GitHub contains files that ingest data for a single Chronicle log type. The script connects to a single source device and then sends raw logs to Chronicle using the Ingestion API. We recommend that you deploy each log type as a separate Cloud Function. Access the scripts in the Chronicle GitHub repository. Each sub-directory in GitHub contains the following files specific to the log type it ingests.
main.py
is the ingestion script specific to the log type. It connects to the source device and ingests data to Chronicle..env.yml
stores configuration required by the Python script and is specific to the deployment. You modify this file to set configuration parameters required by the ingestion script.README.md
provides information about configuration parameters.Requirements.txt
defines the dependencies required by the ingestion script. In addition, thecommon
folder contains utility functions that all ingestion scripts depend on.
Perform the following steps to assemble the files that ingest data for a single log type:
- Create a deployment directory to store the files for the Cloud Function. This will contain all files needed for the deployment.
- Copy all files from the GitHub sub-directory of the desired log type, for example OneLogin User Context, to this deployment directory.
- Copy the
common
folder and all contents to the deployment directory. The contents of the directory will look similar to the following:
one_login_user ├─common │ ├─__init__.py │ ├─auth.py │ ├─env_constants.py │ ├─ingest.py │ ├─status.py │ └─utils.py ├─env.yml ├─main.py └─requirements.txt
Configure the scripts
- Launch a Cloud Shell session.
- Connect with SSH to a Google Cloud Linux VM. See Connect to Linux VMs using Google tools.
Upload the ingestion scripts by clicking > Upload or Download to move your files or folders to or from Cloud Shell.
MoreFiles and folders can only be uploaded to and downloaded from your home directory. For more options to transfer files between Cloud Shell and your local workstation, see Upload and download files and folders from Cloud Shell.
Edit the
.env.yml
file for the function and populate the required environment variables. The following table lists runtime environment variables common to all ingestion scripts.Variable name Description Required Default Secret CHRONICLE_CUSTOMER_ID
Chronicle customer ID. Yes None No CHRONICLE_REGION
Chronicle region. Yes us
Other valid values areasia-southeast1
,europe
, andeurope-west2
.No CHRONICLE_SERVICE_ACCOUNT
Contents of the Chronicle service account JSON file. Yes None Yes CHRONICLE_NAMESPACE
The namespace that the Chronicle logs are labeled with. For information about Chronicle namespaces, see Work with Asset namespaces. No None No Each script requires environment variables specific to the script. See Configuration parameters by log type for details about the environment variables required by each log type.
Environment variables marked as Secret = Yes
must be configured as secrets
in Secret Manager. See
Secret Manager pricing
for information on the cost of using Secret Manager.
See Creating and accessing secrets for detailed instructions.
After the secrets are created in Secret Manager, use
the secret resource name as the value for environment variables. For example:
projects/{project_id}/secrets/{secret_id}/versions/{version_id}
, where {project_id}
, {secret_id}
,
and {version_id}
are specific to your environment.
Set up a scheduler or trigger
All scripts, except Pub/Sub, are implemented to collect the data at periodic intervals from a source device. You must set up a trigger using Cloud Scheduler to fetch data over time. The ingestion script for Pub/Sub continuously monitors the Pub/Sub subscription. For more information, see Running services on a schedule and Using Pub/Sub to trigger a Cloud Function.
Deploy the Cloud Function
- Launch a Cloud Shell session.
- Connect via SSH to a Google Cloud Linux VM. See Connect to Linux VMs using Google tools.
- Change to the directory where you copied ingestion scripts.
Execute the following command to deploy the Cloud Function.
gcloud functions deploy <FUNCTION NAME> --entry-point main --trigger-http --runtime python39 --env-vars-file .env.yml
Where
FUNCTION_NAME
is the name you define for the Cloud Function.Note: If you do not change directory to the location of the files, make sure to use the
--source
option to specify the location of the deployment scripts.
View runtime logs
The ingestion scripts print runtime messages to stdout. Cloud Functions provides a mechanism to view log messages. For more information, see the Cloud Functions information about Viewing runtime logs.
Configuration parameters by log type
Box
This script gets details about events that happen within Box and ingests them
to Chronicle. The data provide insights to CRUD operations on objects in the Box
environment. For information about Box events, see the
Box events API.
Define the following environment variables in the .env.yml
file. For more
information about the Box Client ID, Client Secret, and Subject ID, see
Client Credentials Grant.
Variable name | Description | Default Value | Secret |
---|---|---|---|
CHRONICLE_CUSTOMER_ID |
Chronicle platform customer ID. | None | No |
POLL_INTERVAL |
Frequency interval at which the function executes to get additional log data (in minutes). This duration must be the same as the Cloud Scheduler job interval. | 5 | No |
CHRONICLE_REGION |
Chronicle platform region. | us Other valid values are: asia-southeast1 , europe , and europe-west2 . |
No |
CHRONICLE_SERVICE_ACCOUNT |
Path to the secret in Secret Manager that stores the Chronicle service account JSON file. | None | Yes |
BOX_CLIENT_ID |
Client ID of Box platform, available in Box developer console. | None | No |
BOX_CLIENT_SECRET |
Path to the secret in Secret Manager that stores the client secret of Box platform used for authentication. | None | Yes |
BOX_SUBJECT_ID |
Box User ID or Enterprise ID. | None | No |
CHRONICLE_NAMESPACE |
The namespace that the Chronicle logs are labeled with. For information about Chronicle namespaces, see Work with Asset Namespaces. | None | No |
Citrix Session Metadata
This script collects Citrix session metadata from Citrix environments, such as
user login details, logon duration, session creation time, session ending time
and other metadata related to session. For more information, see the
Citrix Sessions API.
Define the following environment variables in the .env.yml
file. For information
about Citrix Client IDs and Client Secrets, see
Getting started with Citrix APIs.
Variable name | Description | Default Value | Secret |
---|---|---|---|
CHRONICLE_CUSTOMER_ID |
Chronicle platform customer ID. | None | No |
CHRONICLE_REGION |
Chronicle platform region. | us Other valid values are: asia-southeast1 , europe , and europe-west2 . |
No |
CHRONICLE_SERVICE_ACCOUNT |
Path to the secret in Secret Manager that stores the Chronicle service account JSON file. | None | Yes |
URL_DOMAIN |
Citrix URL domain. | None | No |
CITRIX_CLIENT_ID | Citrix Client ID. | None | No |
CITRIX_CLIENT_SECRET | Path to the secret in Secret Manager that stores the Citrix Client Secret used for authentication. | None | Yes |
CITRIX_CUSTOMER_ID | Citrix customer ID. | None | No |
POLL_INTERVAL |
Frequency interval at which the function executes to get additional log data (in minutes). This duration must be the same as the Cloud Scheduler job interval. | 30 | No |
CHRONICLE_NAMESPACE |
The namespace that the Chronicle logs are labeled with. For information about Chronicle namespaces, see Work with Asset Namespaces. | None | No |
Citrix Audit logs
This script collects Citrix Audit logs which helps identify activities
performed in the Citrix Cloud environment providing information like what
changed, who changed, when it was changed, etc. For more information, see the
Citrix Cloud SystemLog API.
Define the following environment variables in the .env.yml
file. For information
about Citrix Client IDs and Client Secrets, see
Getting started with Citrix APIs.
Variable name | Description | Default Value | Secret |
---|---|---|---|
CHRONICLE_CUSTOMER_ID |
Chronicle platform customer ID. | None | No |
CHRONICLE_REGION |
Chronicle platform region. | us Other valid values are: asia-southeast1 , europe , and europe-west2 . |
No |
CHRONICLE_SERVICE_ACCOUNT |
Path to the secret in Secret Manager that stores the Chronicle service account JSON file. | None | Yes |
CITRIX_CLIENT_ID | Citrix API Client ID. | None | No |
CITRIX_CLIENT_SECRET | Path to the secret in Secret Manager that stores the Citrix API Client Secret used for authentication. | None | Yes |
CITRIX_CUSTOMER_ID | Citrix CustomerID. | None | No |
POLL_INTERVAL |
Frequency interval at which additional log data is collected (in minutes). This duration must be the same as the Cloud Scheduler job interval. | 30 | No |
URL_DOMAIN |
Citrix Cloud Endpoint. | None | No |
CHRONICLE_NAMESPACE |
The namespace that the Chronicle logs are labeled with. For information about Chronicle namespaces, see Work with Asset Namespaces. | None | No |
MISP
This script fetches threat relation information from MISP, an open source
threat intelligence and sharing platform, and ingests it in Chronicle. For more
information, see the
MISP Events API.
Define the following environment variables in the .env.yml
file.
Variable | Description | Default Value | Secret |
---|---|---|---|
CHRONICLE_CUSTOMER_ID |
Chronicle platform customer ID. | None | No |
POLL_INTERVAL |
Frequency interval at which the function executes to get additional log data (in minutes). This duration must be the same as the Cloud Scheduler job interval. | 5 | No |
CHRONICLE_REGION |
Chronicle platform region. | us
Other valid values are: asia-southeast1 , europe , and europe-west2 . |
No |
CHRONICLE_SERVICE_ACCOUNT |
Path to the secret in Secret Manager that stores the Chronicle service account JSON file. | None | Yes |
ORG_NAME |
Organization name for filtering events. | None | No |
API_KEY |
Path to the secret in Secret Manager that stores the API key for used authentication. | None | Yes |
TARGET_SERVER |
The IP address of the MISP instance that you created. | None | No |
CHRONICLE_NAMESPACE |
The namespace that the Chronicle logs are labeled with. For information about Chronicle namespaces, see Work with Asset Namespaces. | None | No |
OneLogin Events
This script gets events from a OneLogin environment and ingests them into
Chronicle. These events provide information such as operations on user accounts. For more information, see the
OneLogin Events API.
Define the following environment variables in the .env.yml
file. For information
about OneLogin Client IDs and Client Secrets, see
Working with API Credentials.
Variable name | Description | Default Value | Secret |
---|---|---|---|
CHRONICLE_CUSTOMER_ID |
Chronicle platform customer ID. | None | No |
POLL_INTERVAL |
Frequency interval at which the function executes to get additional log data (in minutes). This duration must be the same as the Cloud Scheduler job interval. | 5 | No |
CHRONICLE_REGION |
Chronicle platform region. | us Other valid values are: asia-southeast1 , europe , and europe-west2 . |
No |
CHRONICLE_SERVICE_ACCOUNT |
Path to the secret in Secret Manager that stores the Chronicle service account JSON file. | None | Yes |
CLIENT_ID |
Client ID of OneLogin platform. | None | No |
CLIENT_SECRET |
Path to the secret in Secret Manager that stores the client secret of OneLogin platform used for authentication. | None | Yes |
TOKEN_ENDPOINT |
The URL to request an Access Token. | https://api.us.onelogin.com/auth/oauth2/v2/token' | No |
CHRONICLE_NAMESPACE |
The namespace that the Chronicle logs are labeled with. For information about Chronicle namespaces, see Work with Asset Namespaces. | None | No |
One Login User Context
This script gets data related to user accounts from a OneLogin environment and
ingests it into Chronicle. For more information, see
the OneLogin User API.
Define the following environment variables in the .env.yml
file. For information
about OneLogin Client IDs and Client Secrets, see
Working with API Credentials.
Variable name | Description | Default Value | Secret |
---|---|---|---|
CHRONICLE_CUSTOMER_ID |
Chronicle platform customer ID. | None | No |
POLL_INTERVAL |
Frequency interval at which the function executes to get additional log data (in minutes). This duration must be the same as the Cloud Scheduler job interval. | 30 | No |
CHRONICLE_REGION |
Chronicle platform region. | us Other valid values are: asia-southeast1 , europe , and europe-west2 . |
No |
CHRONICLE_SERVICE_ACCOUNT |
Path to the secret in Secret Manager that stores the Chronicle service account JSON file. | None | Yes |
CLIENT_ID |
Client ID of OneLogin platform. | None | No |
CLIENT_SECRET |
Path to the secret in Secret Manager that stores the client secret of OneLogin platform used for authentication. | None | Yes |
TOKEN_ENDPOINT |
The URL to request an Access Token. | https://api.us.onelogin.com/auth/oauth2/v2/token |
No |
CHRONICLE_NAMESPACE |
The namespace that the Chronicle logs are labeled with. For information about Chronicle namespaces, see Work with Asset Namespaces. | None | No |
Google Cloud Pub/Sub
This script collects messages from Pub/Sub subscriptions and ingests the data to Chronicle. It continuously monitors the subscription gateway and ingests newer messages when they appear. For more information, see the following documents:
This ingestion script requires that you set variables in both the .env.yml
file and the Cloud Scheduler job.
Define the following environment variables in the
.env.yml
file.Variable name Description Default Value Secret CHRONICLE_CUSTOMER_ID
Chronicle platform customer ID. None No CHRONICLE_REGION
Chronicle platform region. us
Other valid values are:asia-southeast1
,europe
, andeurope-west2
.No CHRONICLE_SERVICE_ACCOUNT
Path to the secret in Secret Manager that stores the Chronicle service account JSON file. None Yes CHRONICLE_NAMESPACE
The namespace that the Chronicle logs are labeled with. For information about Chronicle namespaces, see Work with Asset Namespaces. None No Set the following variables in the Cloud Scheduler Message body field as a JSON formatted string. See creating the Cloud Scheduler for more information about the Message body field.
Variable name Description Default Value Secret PROJECT_ID
Pub/Sub project ID. See creating and managing projects for information about the project Id. None No SUBSCRIPTION_ID
Pub/Sub Subscription ID. None No CHRONICLE_DATA_TYPE
Ingestion label for the log type provided while pushing data to Chronicle. See Supported default parsers for a list of supported log types. None No Here is an example JSON formatted string for the Message body field.
{ "PROJECT_ID":"projectid-0000","SUBSCRIPTION_ID":"subscription-id","CHRONICLE_DATA_TYPE":"SQUID_PROXY"}
Duo Admin
The script gets events from Duo Admin related to CRUD operations performed on
various objects such as user account, security, etc. For more information, see
the
Duo Admin API.
Define the following environment variables in the .env.yml
file.
Variable name | Description | Default Value | Secret |
---|---|---|---|
CHRONICLE_CUSTOMER_ID |
Chronicle platform customer ID. | None | No |
CHRONICLE_REGION |
Chronicle platform region. | us Other valid values are: asia-southeast1 , europe , and europe-west2 . |
No |
CHRONICLE_SERVICE_ACCOUNT |
Path to the secret in Secret Manager that stores the Chronicle service account JSON file. | None | Yes |
POLL_INTERVAL |
Frequency interval at which the function executes to get additional log data (in minutes). This duration must be the same as the Cloud Scheduler job interval. | None | No |
DUO_API_DETAILS |
Path to the secret in Secret Manager that stores the DUO account
JSON file. This contains the Duo Admin API integration key, Duo Admin
API secret key, and the Duo Admin API hostname. For example:
{
"ikey": "abcd123",
"skey": "def345",
"api_host": "abc-123"
}
See Duo Admin documentation for instructions about downloading the JSON file. |
None | Yes |
CHRONICLE_NAMESPACE |
The namespace that the Chronicle logs are labeled with. For information about Chronicle namespaces, see Work with Asset Namespaces. | None | No |
Slack
This script gets audit logs from a Slack Enterprise Grid organization and
ingest it to Chronicle. For more information, see
Slack Audit Logs API.
Define the following environment variables in the .env.yml
file.
Variable name | Description | Default Value | Secret |
---|---|---|---|
CHRONICLE_CUSTOMER_ID |
Chronicle platform customer ID. | None | No |
CHRONICLE_REGION |
Chronicle platform region. | us Other valid values are: asia-southeast1 , europe , and europe-west2 . |
No |
CHRONICLE_SERVICE_ACCOUNT |
Path to the secret in Secret Manager that stores the Chronicle service account JSON file. | None | Yes |
POLL_INTERVAL |
Frequency interval at which the function executes to get additional log data (in minutes). This duration must be the same as the Cloud Scheduler job interval. | 5 | No |
SLACK_ADMIN_TOKEN | Path to the secret in Secret Manager that stores the Slack
Authentication token. |
None |
Yes |
CHRONICLE_NAMESPACE |
The namespace that the Chronicle logs are labeled with. For information about Chronicle namespaces, see Work with Asset Namespaces. | None | No |
STIX/TAXII
This script pulls indicators from STIX/TAXII server and ingests them into
Chronicle. For more information refer to the
STIX/TAXII API documentation.
Define the following environment variables in .env.yml
file.
Variable Name | Description | Default | Secret |
---|---|---|---|
CHRONICLE_CUSTOMER_ID |
Chronicle platform customer ID. | None | No |
CHRONICLE_REGION |
Chronicle platform region. | us Other valid values are: asia-southeast1 , europe , and europe-west2 . |
No |
CHRONICLE_SERVICE_ACCOUNT |
Path to the secret in Secret Manager that stores the Chronicle service account JSON file. | None | Yes |
POLL_INTERVAL |
Frequency interval (in minutes) at which the function executes. This duration must be same as the Cloud Scheduler job. | 60 | No |
TAXII_VERSION |
The STIX/TAXII version to use. Possible options are 1.1, 2.0, 2.1 | None | No |
TAXII_DISCOVERY_URL |
Discovery URL of TAXII server. | None | No |
TAXII_COLLECTION_NAMES |
Collections (CSV) from which to fetch the data. Leave empty to fetch data from all of the collections. | None | No |
TAXII_USERNAME |
Username required for authentication, if any. | None | No |
TAXII_PASSWORD_SECRET_PATH |
Password required for authentication, if any. | None | Yes |