Collect HashiCorp audit logs

Supported in:

Google SecOps SIEM

Note: This feature is covered by Pre-GA Offerings Terms of the Google Security Operations Service Specific Terms. Pre-GA features might have limited support, and changes to pre-GA features might not be compatible with other pre-GA versions. For more information, see the Google Security Operations Technical Support Service guidelines and the Google Security Operations Service Specific Terms.

This parser processes HashiCorp audit logs in JSON, Syslog, or combined formats. It extracts fields, performs Grok and KV parsing for both standard and "runner" type messages, handles JSON payloads, and maps the extracted data to the UDM. The parser also includes error handling and dropping malformed logs.

Before you begin

• Ensure that you have a Google Security Operations instance.
• Ensure that you have a Windows 2016 or later or a Linux host with systemd.
• If running behind a proxy, ensure firewall ports are open.
• Ensure that you have privileged access to HCP.

Get Google SecOps ingestion authentication file

1. Sign in to the Google SecOps console.
2. Go to SIEM Settings > Collection Agents.
3. Download the Ingestion Authentication File.

Get Google SecOps customer ID

1. Sign in to the Google SecOps console.
2. Go to SIEM Settings > Profile.
3. Copy and save the Customer ID from the Organization Details section.

Install BindPlane Agent

1. For Windows installation, run the following script:
   msiexec /i "https://github.com/observIQ/bindplane-agent/releases/latest/download/observiq-otel-collector.msi" /quiet
2. For Linux installation, run the following script:
   sudo sh -c "$(curl -fsSlL https://github.com/observiq/bindplane-agent/releases/latest/download/install_unix.sh)" install_unix.sh
3. Additional installation options can be found in this installation guide.

Configure BindPlane Agent to ingest Syslog and send to Google SecOps

1. Access the machine where BindPlane is installed.

2. Edit the config.yaml file as follows:

   receivers:
       udplog:
           # Replace the below port <54525> and IP <0.0.0.0> with your specific values
           listen_address: "0.0.0.0:54525" 
   
   exporters:
       chronicle/chronicle_w_labels:
           compression: gzip
           # Adjust the creds location below according the placement of the credentials file you downloaded
           creds: '{ json file for creds }'
           # Replace <customer_id> below with your actual ID that you copied
           customer_id: <customer_id>
           endpoint: malachiteingestion-pa.googleapis.com
           # You can apply ingestion labels below as preferred
           ingestion_labels:
           log_type: SYSLOG
           namespace: auditd
           raw_log_field: body
   service:
       pipelines:
           logs/source0__chronicle_w_labels-0:
               receivers:
                   - udplog
               exporters:
                   - chronicle/chronicle_w_labels
   

3. Restart the BindPlane Agent to apply the changes:

   sudo systemctl restart bindplane
   

Enable Syslog for HCP Vault

1. Sign in to the HCP Portal.
2. Go to Vault Clusters.
3. Select your Vault cluster from the list of deployed clusters.
4. In the Cluster Overview, locate and copy the Vault Address (for example, https://vault-cluster-name.hashicorpcloud.com:8200).
5. Go to the Access Details section, and copy the Root Token.

Install the Vault CLI

• For Linux:

  curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
  echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
  sudo apt update && sudo apt install vault
  

• For macOS (using Homebrew):

  brew tap hashicorp/tap
  brew install hashicorp/tap/vault
  

• For Windows:

  Download the executable file.
  Extract it and add the Vault binary to your system's PATH.
  

• Verify the Vault CLI installation by running:

  vault --version
  

Configure HCP Vault using CLI to send audit logs to BindPlane

1. Open the terminal or the command prompt.

2. Set the Vault server address using the environment variable:

   export VAULT_ADDR="https://vault-cluster-name.hashicorpcloud.com:8200"
   

3. Log in to Vault using the root token:

   vault login <root-token>
   

Configure the Syslog Path to an External Syslog Socket

1. Run the following command to enable syslog and send to BindPlane:

   vault audit enable socket address="udp://<bindplane-ip>:<bindplane-port>" socket_type="udp" tag="vault"
   

2. Confirm the new configuration:

   vault audit list
   

3. The output should display the new socket configuration.

4. Optional: Automate the setup using Terraform:

   • Create a Terraform configuration file (audit.tf):

   resource "vault_audit" "syslog" {
     type        = "syslog"
     description = "Syslog audit logs"
     options = {
       tag      = "vault"
       facility = "LOCAL0"
     }
   }
   
   resource "vault_audit" "socket" {
     type        = "socket"
     description = "Remote syslog socket"
     options = {
       address     = "udp://<syslog-server-ip>:514"
       socket_type = "udp"
       tag         = "vault"
     }
   }
   

   • Apply the configuration:

   terraform init
   terraform apply
   

Troubleshooting logs not received

• Verify the syslog server is reachable:

  ping <syslog-server-ip>
  

UDM Mapping Table

Log Field	UDM Mapping	Logic
auth.accessor	security_result.about.resource.attribute.labels.value	The value of auth.accessor from the raw log is mapped to a label with key "auth_accessor" under security_result.about.resource.attribute.labels in the UDM.
auth.client_token	security_result.about.resource.attribute.labels.value	The value of auth.client_token from the raw log is mapped to a label with key "auth_client_token" under security_result.about.resource.attribute.labels in the UDM.
auth.display_name	target.user.user_display_name	The value of auth.display_name from the raw log is mapped to target.user.user_display_name in the UDM.
auth.entity_id	target.resource.product_object_id	The value of auth.entity_id from the raw log is mapped to target.resource.product_object_id in the UDM.
auth.metadata.account_id	target.user.userid	The value of auth.metadata.account_id from the raw log is mapped to target.user.userid in the UDM.
auth.metadata.auth_type	security_result.about.resource.attribute.labels.value	The value of auth.metadata.auth_type from the raw log is mapped to a label with key "auth_type" under security_result.about.resource.attribute.labels in the UDM.
auth.metadata.role_id	security_result.about.resource.attribute.labels.value	The value of auth.metadata.role_id from the raw log is mapped to a label with key "role_id" under security_result.about.resource.attribute.labels in the UDM.
auth.metadata.role_name	target.resource.attribute.roles.name	The value of auth.metadata.role_name from the raw log is mapped to target.resource.attribute.roles.name in the UDM.
auth.token_ttl	security_result.about.resource.attribute.labels.value	The value of auth.token_ttl from the raw log is mapped to a label with key "auth_token_ttl" under security_result.about.resource.attribute.labels in the UDM.
auth.token_type	security_result.about.resource.attribute.labels.value	The value of auth.token_type from the raw log is mapped to a label with key "auth_token_type" under security_result.about.resource.attribute.labels in the UDM.
cluster	observer.resource.name	The value of cluster from the raw log is mapped to observer.resource.name in the UDM.
error	security_result.description	The value of error from the raw log is mapped to security_result.description in the UDM.
headers.accept	security_result.about.resource.attribute.labels.value	The value of headers.accept from the raw log is mapped to a label with key "httpHeaders accept" under security_result.about.resource.attribute.labels in the UDM.
headers.httpHeaders.cache-control	additional.fields.value.string_value	The value of headers.httpHeaders.cache-control from the raw log is mapped to a field with key "httpHeaders cache control" under additional.fields in the UDM.
headers.snyk-acting-org-public-id	principal.resource.attribute.labels.value	The value of headers.snyk-acting-org-public-id from the raw log is mapped to a label with key "snyk-acting-org-public-id" under principal.resource.attribute.labels in the UDM.
headers.snyk-flow-name	principal.resource.attribute.labels.value	The value of headers.snyk-flow-name from the raw log is mapped to a label with key "snyk-flow-name" under principal.resource.attribute.labels in the UDM.
headers.snyk-request-id	principal.resource.attribute.labels.value	The value of headers.snyk-request-id from the raw log is mapped to a label with key "snyk-request-id" under principal.resource.attribute.labels in the UDM.
headers.user-agent	network.http.parsed_user_agent	The value of headers.user-agent from the raw log is parsed as a user agent and mapped to network.http.parsed_user_agent in the UDM.
headers.x-forwarded-host	principal.hostname	The value of headers.x-forwarded-host from the raw log is mapped to principal.hostname in the UDM.
headers.x-forwarded-port	principal.port	The value of headers.x-forwarded-port from the raw log is mapped to principal.port in the UDM.
headers.x-real-ip	principal.ip	The value of headers.x-real-ip from the raw log is mapped to principal.ip in the UDM.
hostname	observer.hostname	The value of hostname from the raw log is mapped to observer.hostname in the UDM.
httpHeaders.cf-cache-status	target.resource.attribute.labels.value	The value of httpHeaders.cf-cache-status from the raw log is mapped to a label with key "cf-cache-status" under target.resource.attribute.labels in the UDM.
httpHeaders.cf-ray	target.resource.attribute.labels.value	The value of httpHeaders.cf-ray from the raw log is mapped to a label with key "cf-ray" under target.resource.attribute.labels in the UDM.
httpHeaders.content-length	security_result.about.resource.attribute.labels.value	The value of httpHeaders.content-length from the raw log is mapped to a label with key "httpHeaders Content-Length" under security_result.about.resource.attribute.labels in the UDM.
httpHeaders.content-type	security_result.about.resource.attribute.labels.value	The value of httpHeaders.content-type from the raw log is mapped to a label with key "httpHeaders Content-Type" under security_result.about.resource.attribute.labels in the UDM.
httpHeaders.gitlab-lb	target.resource.attribute.labels.value	The value of httpHeaders.gitlab-lb from the raw log is mapped to a label with key "gitlab-lb" under target.resource.attribute.labels in the UDM.
httpHeaders.gitlab-sv	target.resource.attribute.labels.value	The value of httpHeaders.gitlab-sv from the raw log is mapped to a label with key "gitlab-sv" under target.resource.attribute.labels in the UDM.
httpHeaders.ratelimit-limit	target.resource.attribute.labels.value	The value of httpHeaders.ratelimit-limit from the raw log is mapped to a label with key "ratelimit-limit" under target.resource.attribute.labels in the UDM.
httpHeaders.ratelimit-observed	target.resource.attribute.labels.value	The value of httpHeaders.ratelimit-observed from the raw log is mapped to a label with key "ratelimit-observed" under target.resource.attribute.labels in the UDM.
httpHeaders.ratelimit-remaining	target.resource.attribute.labels.value	The value of httpHeaders.ratelimit-remaining from the raw log is mapped to a label with key "ratelimit-remaining" under target.resource.attribute.labels in the UDM.
httpHeaders.ratelimit-reset	target.resource.attribute.labels.value	The value of httpHeaders.ratelimit-reset from the raw log is mapped to a label with key "ratelimit-reset" under target.resource.attribute.labels in the UDM.
httpHeaders.ratelimit-resettime	target.resource.attribute.labels.value	The value of httpHeaders.ratelimit-resettime from the raw log is mapped to a label with key "ratelimit-resettime" under target.resource.attribute.labels in the UDM.
httpHeaders.referrer-policy	target.resource.attribute.labels.value	The value of httpHeaders.referrer-policy from the raw log is mapped to a label with key "referrer-policy" under target.resource.attribute.labels in the UDM.
httpHeaders.server	target.resource.attribute.labels.value	The value of httpHeaders.server from the raw log is mapped to a label with key "server" under target.resource.attribute.labels in the UDM.
httpHeaders.x-content-type-options	target.resource.attribute.labels.value	The value of httpHeaders.x-content-type-options from the raw log is mapped to a label with key "x-content-type-options" under target.resource.attribute.labels in the UDM.
httpHeaders.x-frame-options	target.resource.attribute.labels.value	The value of httpHeaders.x-frame-options from the raw log is mapped to a label with key "x-frame-options" under target.resource.attribute.labels in the UDM.
httpHeaders.x-request-id	target.resource.attribute.labels.value	The value of httpHeaders.x-request-id from the raw log is mapped to a label with key "x-request-id" under target.resource.attribute.labels in the UDM.
httpStatus	network.http.response_code	The value of httpStatus from the raw log is mapped to network.http.response_code in the UDM.
httpUrl	target.url	The value of httpUrl from the raw log is mapped to target.url in the UDM.
insertId	metadata.product_log_id	The value of insertId from the raw log is mapped to metadata.product_log_id in the UDM.
job	additional.fields.value.string_value	The value of job from the raw log is mapped to a field with key "job id" under additional.fields in the UDM.
job_status	additional.fields.value.string_value	The value of job_status from the raw log is mapped to a field with key "job_status" under additional.fields in the UDM.
labels.compute.googleapis.com/resource_name	target.resource.name	The value of labels.compute.googleapis.com/resource_name from the raw log is mapped to target.resource.name in the UDM.
labels.k8s-pod/app_kubernetes_io/instance	target.resource.attribute.labels.value	The value of labels.k8s-pod/app_kubernetes_io/instance from the raw log is mapped to a label with key "Kubernetes IO Instance" under target.resource.attribute.labels in the UDM.
labels.k8s-pod/app_kubernetes_io/name	target.resource.attribute.labels.value	The value of labels.k8s-pod/app_kubernetes_io/name from the raw log is mapped to a label with key "Kubernetes IO Instance Name" under target.resource.attribute.labels in the UDM.
labels.k8s-pod/component	target.resource.attribute.labels.value	The value of labels.k8s-pod/component from the raw log is mapped to a label with key "component" under target.resource.attribute.labels in the UDM.
labels.k8s-pod/controller-revision-hash	target.resource.attribute.labels.value	The value of labels.k8s-pod/controller-revision-hash from the raw log is mapped to a label with key "Controller Revision Hash" under target.resource.attribute.labels in the UDM.
labels.k8s-pod/helm_sh/chart	target.resource.attribute.labels.value	The value of labels.k8s-pod/helm_sh/chart from the raw log is mapped to a label with key "Kubernetes IO Instance Manager SH" under target.resource.attribute.labels in the UDM.
labels.k8s-pod/vault-active	target.resource.attribute.labels.value	The value of labels.k8s-pod/vault-active from the raw log is mapped to a label with key "Vault active" under target.resource.attribute.labels in the UDM.
labels.k8s-pod/vault-initialized	target.resource.attribute.labels.value	The value of labels.k8s-pod/vault-initialized from the raw log is mapped to a label with key "Vault initialized" under target.resource.attribute.labels in the UDM.
labels.k8s-pod/vault-perf-standby	target.resource.attribute.labels.value	The value of labels.k8s-pod/vault-perf-standby from the raw log is mapped to a label with key "vault perf standby" under target.resource.attribute.labels in the UDM.
labels.k8s-pod/vault-sealed	target.resource.attribute.labels.value	The value of labels.k8s-pod/vault-sealed from the raw log is mapped to a label with key "Vault sealed" under target.resource.attribute.labels in the UDM.
labels.k8s-pod/vault-version	target.resource.attribute.labels.value	The value of labels.k8s-pod/vault-version from the raw log is mapped to a label with key "Vault version" under target.resource.attribute.labels in the UDM.
maskedToken	security_result.about.resource.attribute.labels.value	The value of maskedToken from the raw log is mapped to a label with key "maskedToken" under security_result.about.resource.attribute.labels in the UDM.
method	network.http.method, operation	The value of method from the raw log is mapped to operation. If operation is not empty, then network.application_protocol is set to "HTTP". Based on the value of operation, network.http.method is derived.
msg	metadata.description	The value of msg from the raw log is mapped to metadata.description in the UDM.
pid	target.process.pid	The value of pid from the raw log is mapped to target.process.pid in the UDM.
request.client_token	target.resource.attribute.labels.value	The value of request.client_token from the raw log is mapped to a label with key "request_client_token" under target.resource.attribute.labels in the UDM.
request.client_token_accessor	target.resource.attribute.labels.value	The value of request.client_token_accessor from the raw log is mapped to a label with key "request_client_token_accessor" under target.resource.attribute.labels in the UDM.
request.data.role_id	target.resource.attribute.labels.value	The value of request.data.role_id from the raw log is mapped to a label with key "request_data_role_id" under target.resource.attribute.labels in the UDM.
request.data.secret_id	target.resource.attribute.labels.value	The value of request.data.secret_id from the raw log is mapped to a label with key "request_data_secret_id" under target.resource.attribute.labels in the UDM.
request.id	network.session_id	The value of request.id from the raw log is mapped to network.session_id in the UDM.
request.mount_accessor	target.resource.attribute.labels.value	The value of request.mount_accessor from the raw log is mapped to a label with key "request_mount_accessor" under target.resource.attribute.labels in the UDM.
request.mount_type	target.resource.attribute.labels.value	The value of request.mount_type from the raw log is mapped to a label with key "request_mount_type" under target.resource.attribute.labels in the UDM.
request.namespace.id	target.namespace	The value of request.namespace.id from the raw log is mapped to target.namespace in the UDM.
request.operation	target.resource.attribute.labels.value, network.http.method, operation	The value of request.operation from the raw log is mapped to operation. If operation is not empty, then network.application_protocol is set to "HTTP". Based on the value of operation, network.http.method is derived. The value of request.operation is also mapped to a label with key "capabilities" under target.resource.attribute.labels in the UDM.
request.path	target.url	The value of request.path from the raw log is mapped to target.url in the UDM.
request.remote_address	principal.ip	The value of request.remote_address from the raw log is mapped to principal.ip in the UDM.
request.remote_port	principal.port	The value of request.remote_port from the raw log is mapped to principal.port in the UDM.
request.wrap_ttl	target.resource.attribute.labels.value	The value of request.wrap_ttl from the raw log is mapped to a label with key "request_wrap_ttl" under target.resource.attribute.labels in the UDM.
resource.labels.container_name	additional.fields.value.string_value	The value of resource.labels.container_name from the raw log is mapped to a field with key "container name" under additional.fields in the UDM.
resource.labels.location	target.location.name	The value of resource.labels.location from the raw log is mapped to target.location.name in the UDM.
resource.labels.namespace_name	target.namespace	The value of resource.labels.namespace_name from the raw log is mapped to target.namespace in the UDM.
resource.labels.pod_name	additional.fields.value.string_value	The value of resource.labels.pod_name from the raw log is mapped to a field with key "pod_name" under additional.fields in the UDM.
resource.labels.project_id	target.cloud.project.name	The value of resource.labels.project_id from the raw log is mapped to target.cloud.project.name in the UDM.
response.data.num_uses	target.resource.attribute.labels.value	The value of response.data.num_uses from the raw log is mapped to a label with key "response_data_num_uses" under target.resource.attribute.labels in the UDM.
response.data.orphan	target.resource.attribute.labels.value	The value of response.data.orphan from the raw log is mapped to a label with key "response_data_orphan" under target.resource.attribute.labels in the UDM.
response.data.renewable	target.resource.attribute.labels.value	The value of response.data.renewable from the raw log is mapped to a label with key "response_data_renewable" under target.resource.attribute.labels in the UDM.
response.data.ttl	target.resource.attribute.labels.value	The value of response.data.ttl from the raw log is mapped to a label with key "response_data_ttl" under target.resource.attribute.labels in the UDM.
response.wrap_info.accessor	target.resource.attribute.labels.value	The value of response.wrap_info.accessor from the raw log is mapped to a label with key "response_wrap_info_accessor" under target.resource.attribute.labels in the UDM.
response.wrap_info.token	target.resource.attribute.labels.value	The value of response.wrap_info.token from the raw log is mapped to a label with key "response_wrap_info_token" under target.resource.attribute.labels in the UDM.
response.wrap_info.ttl	target.resource.attribute.labels.value	The value of response.wrap_info.ttl from the raw log is mapped to a label with key "response_wrap_info_ttl" under target.resource.attribute.labels in the UDM.
response.wrap_info.wrapped_accessor	target.resource.attribute.labels.value	The value of response.wrap_info.wrapped_accessor from the raw log is mapped to a label with key "response_wrap_info_wrapped_accessor" under target.resource.attribute.labels in the UDM.
runner	principal.user.userid	The value of runner from the raw log is mapped to principal.user.userid in the UDM.
status	network.http.response_code	The value of status from the raw log is mapped to network.http.response_code in the UDM.
streamingID	target.resource.attribute.labels.value	The value of streamingID from the raw log is mapped to a label with key "streamingID" under target.resource.attribute.labels in the UDM.
time	metadata.event_timestamp.seconds, metadata.event_timestamp.nanos	The value of time from the raw log is parsed and used to populate the metadata.event_timestamp field in the UDM.
type	metadata.product_event_type	The value of type from the raw log is mapped to metadata.product_event_type in the UDM.
url	principal.url	The value of url from the raw log is mapped to principal.url in the UDM. The value "MACHINE" is assigned to extensions.auth.type in the UDM. The value "USER_LOGIN" is assigned to metadata.event_type in the UDM. The value "HASHICORP" is assigned to metadata.log_type in the UDM. The value "HASHICORP" is assigned to metadata.product_name in the UDM. The value "HASHICORP" is assigned to metadata.vendor_name in the UDM. The value "SERVICE_ACCOUNT" is assigned to target.resource.attribute.roles.type in the UDM.

Changes

2023-10-26

• Added a Grok pattern to handle SYSLOG+JSON logs.

2023-09-22

• Modified mapping for "request.remote_port" from "target.port" to "principal.port".
• Modified mapping for "request.remote_address" from "target.ip" to "principal.ip".
• Mapped "error" to "security_result.description".
• Mapped "resource.labels.namespace_name" to "target.namespace".
• Mapped "resource.labels.pod_name", "resource.labels.container_name" to "additional.fields".
• Mapped "resource.labels.project_id" to "target.cloud.project.name".
• Mapped "resource.labels.location" to "target.location.name".
• Mapped "insertId" to "metadata.product_log_id".
• Mapped "labels.k8s-pod/app_kubernetes_io/instance", "labels.k8s-pod/app_kubernetes_io/name", "labels.k8s-pod/component", "labels.k8s-pod/helm_sh/chart", "labels.k8s-pod/controller-revision-hash", "labels.k8s-pod/vault-initialized", "labels.k8s-pod/vault-version", "labels.k8s-pod/vault-sealed", "labels.k8s-pod/vault-perf-standby", and "labels.k8s-pod/vault-active" to "target.resource.attribute.labels".
• Mapped "labels.compute.googleapis.com/resource_name" to "target.resource.name".

2023-04-26

• Added a Grok pattern to handle syslog logs.
• Mapped "status" to "network.http.response_code".
• Mapped "runner" to "principal.user.userid"
• Mapped "job_id", "job_status" to "additional.fields".

2023-03-24

• Mapped "host" to "observer.hostname".
• Mapped "cluster" to "observer.resource.name".
• If log contains cluster, then mapped "cluster" to "observer.resource.resource_type".
• Added JSON block to retrieve data from "_raw" field.
• "httpStatus" mapped to "network.http.response_code".
• "httpUrl" mapped to "target.url".
• "pid" mapped to "target.process.pid".
• "msg" mapped to "metadata.description".
• "url" mapped to "principal.url".
• "hostname" mapped to "observer.hostname".
• "streamingID", "requestId", "httpHeaders.cf-cache-status", "httpHeaders.cf-ray", "httpHeaders.gitlab-lb", "httpHeaders.gitlab-sv", "httpHeaders.x-request-id", "httpHeaders.x-content-type-options", "httpHeaders.x-frame-options", "httpHeaders.ratelimit-limit", "httpHeaders.ratelimit-observed", "httpHeaders.ratelimit-remaining", "httpHeaders.ratelimit-reset", "httpHeaders.ratelimit-resettime", "httpHeaders.server", "httpHeaders.referrer-policy" mapped to "target.resource.attribute.labels".
• "method" mapped to "network.application_protocol".
• "headers.user-agent" mapped to "network.http.parsed_user_agent".
• "httpHeaders.cache-control" mapped to "additional.fields".
• "httpHeaders.content-type", "httpHeaders.content-length", "maskedToken", "headers.accept" mapped to "security_result.about.resource.attribute.labels".
• "headers.x-real-ip" mapped to "principal.ip".
• "headers.x-forwarded-host" mapped to "principal.hostname".
• "headers.x-forwarded-port" mapped to "principal.port".
• "headers.snyk-acting-org-public-id", "headers.snyk-flow-name", "headers.snyk-request-id" mapped to "principal.resource.attribute.labels".

2023-02-09

• Newly created parser.