Chronicle Ingestion API, version 1

The Chronicle Ingestion API enables you to forward logs directly to Chronicle, eliminating the need for additional hardware or software (for example, forwarders) in your environment.

The Chronicle Ingestion API is a RESTful API with a JSON payload. Managed Security Service Providers (MSSPs) and Technology Partners can develop systems to call the Chronicle Ingestion API directly and forward logs to Chronicle.

The Ingestion API supports:

  • UDM events
  • Unstructured logs
  • Log types retrieval

You can forward your data to Chronicle using either of the following types of Chronicle Ingestion API endpoints:

  • Unified Data Model (UDM) event
  • Unstructured log

If you have formatted your log data using Chronicle's UDM, you can forward UDM events to your Chronicle account using the UDM API endpoint. Since UDM events are standardized, Chronicle is better able to process and interpret the data, increasing Chronicle's ability to recognize security compromises and threats within your enterprise. See Format log data as UDM for information about converting raw logs to UDM.

You can also forward your data to Chronicle as unstructured logs using the unstructured log API endpoints. Your unstructured log data is normalized within the Chronicle infrastructure and made available to you through the Chronicle UI. However, some information may be difficult to extract from the unstructured log data and might only be searchable using Raw Log Scan.

Whenever possible, we recommend forwarding your data to Chronicle as UDM events.

How to Authenticate with the Ingestion API

API keys are used to authenticate calls. The API keys are linked to customers and are provided by your Chronicle representative.

Both the UDM and unstructured ingestion API endpoints require a Google Cloud-supplied security key for access. The key ensures that only you can forward data to your Chronicle account through the ingestion API.

Calling the Ingestion API

The following example illustrates how you could access the Ingestion API using a cURL command:

curl \
--header "Content-Type: application/json" \
--request POST \
--data '{"log_type":"BIND_DNS","entries":[
{"log_text":"26-Feb-2019 13:35:02.187 client 10.120.20.32#4238:
query: altostrat.com IN A + (203.90.80.102)"}]}' \
--url https://malachiteingestion-pa.googleapis.com/v1/unstructuredlogentries?key=ACzjZyDfwOthABCDwm0V2KW87cQF_zs5ox4Oh3Q

The first part of this statement includes the cURL command and the POST request:

curl --header "Content-Type: application/json" --request POST --data

The second part of this statement includes the fields for the API endpoint:

'{"log_type":"BIND_DNS","entries":[{"log_text":"26-Feb-2019 13:35:02.187 client
10.120.20.32#4238: query: altostrat.com IN A + (203.0.113.102)"}]}'

Lastly, the statement includes the URL with the API endpoint (in this case, unstructuredlogentries) and your Google Cloud-supplied API key:

https://malachiteingestion-pa.googleapis.com/v1/unstructuredlogentries?key=ACzjZyDfwOthHNMBwm0V2KW87cQF_zs5ox4Oh3Q

Regional Endpoints

Chronicle provides regional endpoints for each API. For example:

https://malachiteingestion-pa.googleapis.com/v1/udmevents
https://europe-malachiteingestion-pa.googleapis.com/v1/udmevents
https://asia-southeast1-malachiteingestion-pa.googleapis.com/v1/udmevents

Frequently asked questions

Does Google Cloud provide a certificate signed by a trusted certificate authority? Or do customers provide their own certificate?

No. Google Cloud provides Chronicle customers with API keys.

What is the recommended batch size (per HTTP request)?

1 megabyte.

Can a customer create an allowlist to include certain public IP addresses that are allowed to connect to the collector API endpoint? Or do you support other forms of client authorization?

Chronicle does not support allow listing specific IP addresses. Any connection is valid as long as it has the API key.

What should we do if an HTTP request fails? What are the error codes?

See the section Handling Errors in the Cloud API Design Guide, specifically the following:

Clients should retry on 500 and 503 errors with exponential backoff. The minimum delay should be 1s unless it is documented otherwise. For 429 errors, the client may retry with minimum 30s delay. For all other errors, retry may not be applicable.

Do you support standard HTTP GZIP compression (for example, Accept-Encoding: gzip header)?

Yes, Chronicle supports standard HTTP GZIP compression.

Ingestion API reference

udmevents

Use this method to forward UDM events to Chronicle in batches.

Request

POST https://malachiteingestion-pa.googleapis.com/v1/udmevents

Request body

The following example illustrates how you could format your log data using the udmevents API endpoint. It shows how to format your enterprise log data using UDM.

{
"events": [{
  "metadata": {
    "event_timestamp": "2019-10-22T12:00:00.000Z",
    "event_type": "USER_LOGIN",
    "product_name": "Acme SSO",
    "vendor_name": "Acme"
  },
  "principal": {
    "ip": [
      "10.1.2.3"
    ]
  },
  "target": {
    "application": "Acme Connect",
    "user": {
      "user_display_name": "Mary Jane",
      "userid": "mary@altostrat.com"
    }
  }
  },
  {
  "metadata": {
    "event_timestamp": "2019-10-23T04:00:00.000Z",
    "event_type": "NETWORK_HTTP",
    "product_name": "Acme Proxy",
    "vendor_name": "Acme"
  },
  "network": {
    "http": {
      "method": "GET",
      "response_code": 200
    }
  },
  "principal": {
    "hostname": "host0",
    "ip": "10.1.2.3",
    "port": 60000
  },
  "target": {
    "hostname": "www.altostrat.com",
    "ip": "198.51.100.68",
    "port": 443,
    "url": "www.altostrat.com/images/logo.png"
    }
  }
]}

Body parameters

Field Value Required Description
events[] array Yes Array of UDM events.

Response

You should not receive a response unless there is an error in the method syntax.

unstructuredlogentries

Use this method to forward unstructured log entries to Chronicle one batch at a time.

Each batch of log data has a maximum size limit of 1 MB. Do not send multiple types of timestamps (use either ts_epoch_microseconds or ts_rfc3339 but not both). If you do not provide a separate timestamp, be sure to include one within the log_text field.

Request

POST https://malachiteingestion-pa.googleapis.com/v1/unstructuredlogentries

Request body

The following example illustrates how you could format your log data using the unstructuredlogentries API endpoint. It shows how to format three BIND DNS logs.

{
  "customer_id": "abc123d4-e517-4d48-b6ed-84aa0f987654",
  "log_type": "BIND_DNS",
  "entries": [
    {
      "log_text": "26-Feb-2019 13:35:02.187 client 10.120.20.32#4238: query: altostrat.com IN A + (203.0.113.102)"
      "ts_epoch_microseconds": 1551188102187000
    },
    {
      "log_text": "26-Feb-2019 13:37:04.523 client 10.50.100.33#1116: query: examplepetstore.com IN A + (203.0.113.102)"
      "ts_rfc3339": "2019-26-02T13:37:04.523-08:00"
    },
    {
      "log_text": "26-Feb-2019 13:39:01.115 client 10.1.2.3#3333: query: www.example.com IN A + (203.0.113.102)"
    },
  ]
}

Body parameters

Field Value Required Description
customer_id string Yes Customer ID/UUID
log_type string Yes Identifies the log entries in the batch (for example, WINDOWS_DNS).
entries[] array Yes Array of objects containing the fields for the raw log and its timestamp.
entries[].log_text string Yes Text of the raw log entry. This should not contain any binary data and should only use UTF-8 strings.
entries[].ts_epoch_microseconds uint64 Optional UNIX timestamp in microseconds associated with the log entry.
entries[].ts_rfc3339 string Optional Timestamp associated with the log entry in RFC 3339 format.

Response

You should not receive a response unless there is an error in the method syntax.

createentities

Creates entities. You are limited to 1MB of data per request.

Request

POST https://malachiteingestion-pa.googleapis.com/v1/entities

Request body

{
  log_type: <log type goes here>
  entities: <array of Entities>
}

Body parameters

Field Value Required Description
log_type string Yes Any log_type value returned by the logtypes endpoint.
entities[] array Yes Array of Entities.

Sample request body

{
 "log_type": "AZURE_AD_CONTEXT",
 "entities": [{
   "metadata": {
      "collected_timestamp":"2021-11-14T15:30:18.142265Z",
      "entity_type": "USER",
      "vendor_name": "vendor",
      "product_name": "product"
   },
   "entity": {
      "user": {
        "userid": "johndoe",
        "product_object_id": "doejohn"
      }
   }
  },
  {
   "metadata": {
      "collected_timestamp":"2021-11-14T16:30:18.142265Z",
      "entity_type": "USER",
      "vendor_name": "vendor",
      "product_name": "product"
   },
   "entity": {
      "user": {
        "userid": "janedoe",
        "product_object_id": "doejane"
      }
   }
  }]
}

Response

Returns ane empty JSON with 200 OK, indicating the operation has completed successfully.

logtypes

Use this method to retrieve a list of supported log types. The log types are formatted using the following fields:

Request

GET https://malachiteingestion-pa.googleapis.com/v1/logtypes

Response

The following example illustrates the format of the information returned when calling the logtypes API endpoint.

{
  "logtypes": [
    {
     "log_type": "BIND_DNS",
     "description": "BIND DNS Server"
    },
    {
      "log_type": "WINDOWS_DNS",
      "description": "Windows DNS"
    },
    {
      "log_type": "WINDOWS_DHCP",
      "description": "Windows DHCP"
    },
    {
      "log_type": "WINEVTLOG", 
      "description": "Windows Event Log"
    }
  ]
}

Response fields

Field Value Description
logtypes[] array Returns an array of the currently supported log types.
logtypes[].log_type string Log type. Appears in the response only.
logtypes[].description string Human readable description of the log type. Appears in the response only.