Collect Microsoft Windows AD data

Supported in:

Google SecOps SIEM

This document includes the following information:

Deployment architecture and installation steps, plus any required configuration that produce logs supported by the Google Security Operations parser for Microsoft Windows Active Directory events. For an overview of Google Security Operations data ingestion, see Data ingestion to Google Security Operations.
Information about how the parser maps fields in the original log to Google Security Operations Unified Data Model fields.

Based on your deployment architecture, configure the BindPlane agent or the NXLog agent to ingest Microsoft Windows Active Directory logs into Google Security Operations. We recommend that you use the BindPlane agent to forward the logs of the Windows Active Directory into Google Security Operations.

Information in this document applies to the parser with the WINDOWS_AD ingestion label. The ingestion label identifies which parser normalizes raw log data to structured UDM format.

Before you begin

Before you configure BindPlane agent or the NXLog agent, complete the following tasks:

Configure all systems to use UTC time zone.
Configure Microsoft Windows AD servers.
Review supported devices and versions.
Review supported log types.

Configure Microsoft Windows AD servers

On each Microsoft Windows Active Directory server, create and configure a PowerShell script to save the log data to an output file. The BindPlane agent or NXLog reads the output file.

# Set the location where the log file will be written
$OUTPUT_FILENAME="<Path_of_the_output_file>"

If (Test-Path -Path $OUTPUT_FILENAME) { Remove-Item -path $OUTPUT_FILENAME -ErrorAction SilentlyContinue}

# USER_CONTEXT: Gets all Active Directory users and their properties.
Get-ADUser -Filter * -properties samAccountName | % { Get-ADUser $_.SamAccountName -properties * | ConvertTo-JSON -compress | Out-File -encoding utf8 $OUTPUT_FILENAME -Append }

# ASSET_CONTEXT: Gets all Active Directory assets and their properties.
Get-ADComputer -Filter * -properties samAccountName | % { Get-ADComputer $_.SamAccountName -properties * | ConvertTo-JSON -compress | Out-File -encoding utf8 $OUTPUT_FILENAME -Append }

Replace the following :
- Replace the value of $OUTPUT_FILENAME with the location of the output file.
- Store the data in JSON format.
- Set the encoding to UTF-8.
- Use the -Filter parameter, rather than the -LDAPFilter parameter, when calling the Get-ADUser and Get-ADComputer cmdlets.
Create a recurring task that runs the script to fetch and write data to the output file.
1. Open the Task Scheduler application.
2. Click Create task.
3. Enter a name and description for the task.
4. Select the Run with highest privileges checkbox to make sure all data is retrieved.
5. In the Triggers tab, define when you want to repeat the task.
6. In the Action tab, add a new action and provide the path of the file where the script is stored.

Review supported devices and versions

Microsoft Windows Server is released with the following editions: Foundation, Essentials, Standard, and Datacenter. The event schema of logs generated by each edition does not differ.

The Google Security Operations Parser supports logs from the following Microsoft Windows server versions:

Microsoft Windows Server 2019
Microsoft Windows Server 2016
Microsoft Windows Server 2012

The Google Security Operations parser supports logs that are collected by NXLog Community Edition or Enterprise Edition.

Review supported log types

The Google Security Operations parser will parse and normalize data retrieved from the User Context and the Asset Context. It supports logs generated with English language text and doesn't support logs generated in non-English languages.

Configure the BindPlane agent

We recommend that you use the BindPlane agent to forward the logs of the Windows Active Directory into Google Security Operations.

After installation, the BindPlane agent service appears as the observIQ service in the list of Windows services.

Install the BindPlane agent on each Windows Active Directory server. For more information about installing the BindPlane agent, see the BindPlane Agent installation instructions.

Create a configuration file for the BindPlane agent with the following contents.

receivers:
  filelog:
    include: [ `FILE_PATH` ]
    operators:
      - type: json_parser
    start_at: beginning
  windowseventlog/activedirectoryservice:
    channel: Directory Service
    raw: true
processors:
  batch:

exporters:
  chronicle/activedirectory:
    endpoint: https://malachiteingestion-pa.googleapis.com
    creds: '{
    "type": "service_account",
    "project_id": "malachite-projectname",
    "private_key_id": `PRIVATE_KEY_ID`,
    "private_key": `PRIVATE_KEY`,
    "client_email":"`SERVICE_ACCOUNT_NAME`@malachite-`PROJECT_ID`.iam.gserviceaccount.com",
    "client_id": `CLIENT_ID`,
    "auth_uri": "https://accounts.google.com/o/oauth2/auth",
    "token_uri": "https://oauth2.googleapis.com/token",
    "auth_provider_x509_cert_url":"https://www.googleapis.com/oauth2/v1/certs",
    "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/`SERVICSERVICE_ACCOUNT_NAME`%40malachite-`PROJECT_ID`.iam.gserviceaccount.com",
    "universe_domain": "googleapis.com"
    }'
  log_type: 'WINDOWS_AD'
  override_log_type: false
  raw_log_field: body
  customer_id: `CUSTOMER_ID`

service:
  pipelines:
    logs/ads:
      receivers:
        - filelog
        - windowseventlog/activedirectoryservice
      processors: [batch]
      exporters: [chronicle/activedirectory]

Replace the following :
- FILE_PATH with the path of the file where the output of the PowerShell script mentioned in the Configure Microsoft Windows AD servers has been stored.
- PRIVATE_KEY_ID, PRIVATE_KEY SERVICSERVICE_ACCOUNT_NAME,PROJECT_ID, CLIENT_ID and CUSTOMER_ID with the respective values from the service account JSON file which you can download from Google Cloud. For more information about service account keys, see Create and delete service account keys.
To start the observIQ agent service, select Services > Extended > the observIQ Service > start.

Configure NXLog

The following diagram shows an architecture in which NXLog agents are installed to collect and send Microsoft Windows events to Google Security Operations. Compare this information with your environment to be sure these components are installed. Your deployment might be different from this architecture and might be more complex.

NXLog forwarder ingestion.

If you use the NXLog agent instead of the BindPlane agent, verify the following:

A PowerShell script is created and configured on each Microsoft Windows Server that runs Active Directory to collect USER_CONTEXT and ASSET_CONTEXT data. For more information, see the Configure Microsoft Windows AD servers.
NXLog is installed on each Microsoft Windows AD server to send data to the central Microsoft Windows Server or Linux server.
Google Security Operations forwarder is installed on the central Microsoft Windows Server or Linux server to forward log data to Google Security Operations.

Configure NXLog

Install the NXLog agent on each collector that runs on the Windows Active Directory server. This application forwards logs to the central Microsoft Windows or Linux server. For more information, see the NXLog documentation.

Create a configuration file for each NXLog instance. Use the NXLog im_file module to read from the file and parse the lines into fields. Use om_tcp to forward data to the central Microsoft Windows or Linux server.

The following is an example of the NXLog configuration. Replace <hostname> and <port> values with information about the destination central Microsoft Windows or Linux Server. In the <Input in_adcontext> section and File property, add the path of the output log file written by the PowerShell script. Always set DirCheckInterval and PollInterval. If these are not defined, NXLog polls for files every 1 second.

define ROOT C:\Program Files\nxlog
define ADCONTEXT_OUTPUT_DESTINATION_ADDRESS <hostname>
define ADCONTEXT_OUTPUT_DESTINATION_PORT <port>

Moduledir   %ROOT%\modules
CacheDir    %ROOT%\data
Pidfile     %ROOT%\data\nxlog.pid
SpoolDir    %ROOT%\data
LogFile     %ROOT%\data\nxlog.log

<Input in_adcontext>
    Module im_file
    File "<Path_of_the_output_file>"
    DirCheckInterval 3600
    PollInterval 3600
</Input>

<Output out_chronicle_adcontext>
    Module  om_tcp
    Host    %ADCONTEXT_OUTPUT_DESTINATION_ADDRESS%
    Port    %ADCONTEXT_OUTPUT_DESTINATION_PORT%
</Output>

<Route ad_context_to_chronicle>
    Path in_adcontext => out_chronicle_adcontext
</Route>

Start the NXLog service in each system.

Configure the forwarder on a central server

For information about installing and configuring the forwarder on Linux, see Installing and configuring the forwarder on Linux. For information about installing and configuring the forwarder on Microsoft Windows, see Installing and configuring the forwarder on Microsoft Windows.

Configure the system with the UTC time zone.
Install the Google Security Operations forwarder on the central Microsoft Windows or Linux server.

Configure the Google Security Operations forwarder to send logs to Google Security Operations. The following is an example forwarder configuration:

  - syslog:
      common:
        enabled: true
        data_type: WINDOWS_AD
        batch_n_seconds: 10
        batch_n_bytes: 1048576
      tcp_address: 0.0.0.0:10518
      connection_timeout_sec: 60

Field mapping reference: device log fields to UDM fields

This section describes how the parser maps original log fields to Unified Data Model fields.

User Context logs

NXLog field	UDM field
`GivenName`	`entity.entity.user.first_name`
`Surname`	`entity.entity.user.last_name`
`SamAccountName`	`entity.entity.user.userid`
`SID.Value`	`entity.entity.user.windows_sid`
`ObjectClass`	If the value is `user`, `entity.metadata.entity_type` is set to `USER`
`ObjectGuid`	If the `ObjectGuid` log field value is not empty, then the `ObjectGuid` log field is mapped to the `entity.entity.user.product_object_id` UDM field. Else, If the `objectGUID` log field value is not empty, then the `objectGUID` log field is mapped to the `entity.entity.user.product_object_id` UDM field. Else, If the `ObjectGUID` log field value is not empty, then the `ObjectGUID` log field is mapped to the `entity.entity.user.product_object_id` UDM field.
`objectGUID`	If the `ObjectGuid` log field value is not empty, then the `ObjectGuid` log field is mapped to the `entity.entity.user.product_object_id` UDM field. Else, If the `objectGUID` log field value is not empty, then the `objectGUID` log field is mapped to the `entity.entity.user.product_object_id` UDM field. Else, If the `ObjectGUID` log field value is not empty, then the `ObjectGUID` log field is mapped to the `entity.entity.user.product_object_id` UDM field.
`ObjectGUID`	If the `ObjectGuid` log field value is not empty, then the `ObjectGuid` log field is mapped to the `entity.entity.user.product_object_id` UDM field. Else, If the `objectGUID` log field value is not empty, then the `objectGUID` log field is mapped to the `entity.entity.user.product_object_id` UDM field. Else, If the `ObjectGUID` log field value is not empty, then the `ObjectGUID` log field is mapped to the `entity.entity.user.product_object_id` UDM field.
`AccountExpirationDate`	`entity.entity.user.account_expiration_time`
`badPwdCount`	`entity.entity.user.attribute.label.key` and `entity.entity.user.attribute.label.value`
`CanonicalName`	`entity.entity.administrative_domain`
`City`	`entity.entity.user.personal_address.city`
`Company`	`entity.entity.user.company_name`
`Country`	`entity.entity.user.personal_address.country_or_region`
`Department`	`entity.entity.user.department`
`Description`	`entity.metadata.description`
`DisplayName`	`entity.entity.user.user_display_name`
`EmailAddress`	`entity.entity.user.email_addresses`
`EmployeeID`	`entity.entity.user.employee_id`
`HomeDirectory`	`entity.entity.file.full_path`
`HomePage`	`entity.entity.url`
`HomePhone`	`entity.entity.user.phone_numbers`
`LastBadPasswordAttempt`	`entity.entity.user.last_bad_password_attempt_time`
`lastLogoff`	`entity.entity.user.attribute.label.key` and `entity.entity.user.attribute.label.value`
`lastLogon`	`entity.entity.user.attribute.label.key` and `entity.entity.user.attribute.label.value`
`LastLogonDate`	`entity.entity.user.last_login_time`
`Manager`	Values for `GUID`, `SAMAccountname`, `SID` all mapped to different UDM fields: - `SID` is stored in manager.windows_sid - `Distinguished` name (i.e. value in first CN) is stored in `manager.user_display_name` - `GUID`,`SamAccountName` is stored in `manager.userid`
`MemberOf`	The following fields in the first occurrence of `CN` are set: `entity.relations.entity.group.group_display_name` `entity.relations.entity_type` set to `GROUP` `entity.relations.relationship` set t0 `MEMBER` `entity.relations.direction` set to `UNIDIRECTIONAL`
`MobilePhone`	`entity.entity.user.phone_numbers`
`Office`	`entity.entity.user.office_address.name`
`PasswordExpired`	`entity.entity.user.attribute.label.key` and `entity.entity.user.attribute.label.value`
`PasswordLastSet`	`entity.entity.user.last_password_change_time`
`PasswordNeverExpires`	`entity.entity.user.attribute.label.key` and `entity.entity.user.attribute.label.value`
`PasswordNotRequired`	`entity.entity.user.attribute.label.key` and `entity.entity.user.attribute.label.value`
`PrimaryGroup`	Following fields are set: - `entity.relations.entity.group.group_display_name` - `entity.relations.entity_type` set to `GROUP` - `entity.relations.relationship` set to `MEMBER` - `entity.relations.direction` set to `UNIDIRECTIONAL`
`ServicePrincipalNames`	`entity.entity.user.attribute.label.key` and `entity.entity.user.attribute.label.value`
`State`	`entity.entity.user.personal_address.state`
`StreetAddress`	`entity.entity.user.personal_address.name`
`Title`	`entity.entity.user.title`
`whenCreated`	`entity.user.attribute.creation_time`
`AccountLockoutTime`	`entity.entity.user.account_lockout_time`

Asset Context logs

NXLog Field	UDM Field
`DNSHostName`	`entity.entity.asset.hostname`
`SamAccountName`	`entity.entity.asset.asset_id`
`SID.Value`	`entity.entity.user.windows_sid`
`ObjectClass`	If the value is `computer`, `entity.metadata.entity_type` set to `ASSET`
`ObjectGuid`	If the `ObjectGuid` log field value is not empty, then the `ObjectGuid` log field is mapped to the `entity.entity.asset.product_object_id` UDM field. Else, If the `ObjectGUID` log field value is not empty, then the `ObjectGUID` log field is mapped to the `entity.entity.asset.product_object_id` UDM field.
`ObjectGUID`	If the `ObjectGuid` log field value is not empty, then the `ObjectGuid` log field is mapped to the `entity.entity.asset.product_object_id` UDM field. Else, If the `ObjectGUID` log field value is not empty, then the `ObjectGUID` log field is mapped to the `entity.entity.asset.product_object_id` UDM field.
`AccountExpirationDate`	`entity.entity.user.account_expiration_time`
`badPwdCount`	`entity.entity.asset.attribute.label.key` and `entity.entity.asset.attribute.label.value`
`CanonicalName`	`entity.entity.administrative_domain`
`countryCode`	`entity.entity.asset.location.country_or_region`
`Description`	`entity.entity.metadata.description`
`HomePage`	`entity.entity.url`
`IPv4Address`	`entity.entity.asset.ip`
`IPv6Address`	`entity.entity.asset.ip`
`LastBadPasswordAttempt`	`entity.entity.user.last_bad_password_attempt_time`
`lastLogoff`	`entity.entity.asset.attribute.label.key` and `entity.entity.asset.attribute.label.value`
`lastLogon`	`entity.entity.asset.attribute.label.key` and `entity.entity.asset.attribute.label.value`
`LastLogonDate`	`entity.entity.user.last_login_time`
`Location`	`entity.entity.asset.location.name`
`ManagedBy`	The following fields are set: `entity.entity.user.user_display_name` `entity.relations.entity_type` set to `USER` `entity.relations.relationship` set to `ADMINISTERS` `entity.relations.direction` set to `UNIDIRECTIONAL`
`ObjectCategory`	`entity.entity.asset.category`
`OperatingSystem`	If the `name` contains `Windows`, `entity.entity.asset.platform_software.platform` field is set to `WINDOWS`.
`OperatingSystemServicePack`	`entity.entity.asset.platform_software.platform_patch_level`
`OperatingSystemVersion`	The field `entity.entity.asset.platform_software.platform_version` is set to `%{OperatingSystem} - %{OperatingSystemVersion}`
`PasswordExpired`	`entity.entity.asset.attribute.label.key` and `entity.entity.asset.attribute.label.value`
`PasswordLastSet`	`entity.entity.user.last_password_change_time`
`PasswordNeverExpires`	`entity.entity.asset.attribute.label.key` and `entity.entity.asset.attribute.label.value`
`PasswordNotRequired`	`entity.entity.asset.attribute.label.key` and `entity.entity.asset.attribute.label.value`
`PrimaryGroup`	The following fields are set: - `entity.relations.entity.group.group_display_name` - `entity.relations.entity_type` set to `GROUP` - `entity.relations.relationship` set to `MEMBER` - `entity.relations.direction` set to `UNIDIRECTIONAL`
`ServicePrincipalNames`	`entity.entity.asset.attribute.label.key` and `entity.entity.asset.attribute.label.value`
`whenChanged`	`entity.entity.asset.attribute.last_update_time`
`whenCreated`	`entity.entity.asset.attribute.creation_time`
`AccountLockoutTime`	`entity.entity.user.account_lockout_time`