Manage forwarder configurations through the Google Security Operations UI

Note: Forwarder configuration files not created with the Google Security Operations UI or the Forwarder Management API cannot be managed with the Google Security Operations UI. You need to re-create those configurations using the Google Security Operations UI or the Forwarder Management API to take advantage of the new forwarder management features.

This page describes how to create, manage, and download forwarder configurations using the Google Security Operations user interface (UI). You can also do these tasks programmatically using the Forwarder Management API.

Naming conventions

This document uses these naming conventions:

• Google Security Operations Forwarder: The deployed software component.
• forwarder: The short name for a forwarder configuration when it's stored in your Google Security Operations instance.
• collector: The short name for a collector configuration when it's stored in your Google Security Operations instance.

Add forwarders

Adding a forwarder is the first step of configuring a Google Security Operations Forwarder. Adding a forwarder enables you to do the following:

• Name a forwarder configuration.
• Specify forwarder configuration values.

Adding a new forwarder creates a partially complete forwarder configuration. To complete the forwarder configuration, you need to add a collector. After you add at least one collector, you can download the forwarder configuration and deploy it on a machine or device where Google Security Operations Forwarder is installed.

Instead of adding a new forwarder, you can clone one or more existing forwarders. For details, see Clone forwarders.

To add a new forwarder, follow these steps:

1. In the navigation bar, click Settings.
2. Under Settings, click Forwarders.
3. Click Add new forwarder.
4. In the Forwarder name field, type a name.

5. Optional: Expand the Configuration values section and specify any of the following:

   • Upload compression: Select Yes to compress log data before it's uploaded to Google Security Operations. The default is No. For details about data compression, see Upload compression.
   • Asset namespace: Type a namespace that identifies the logs that are collected by this forwarder. This namespace will be applied for all of the collectors that are added to this forwarder, unless you specify a namespace for a collector at the collector level. If you specify a namespace at both the forwarder level and the collector level, the collector's namespace is used instead of the forwarder's namespace for the logs from that collector. For more information about asset namespaces, see Asset namespaces.
   • Label key and Label value: Type a key and a value. If desired, you can also click Add new label to add one or more additional label key:value pairs. This is a global setting that applies to the forwarder and the forwarder's collectors, unless it is overridden at the collector level. For details, see Labels.
   • Filter description, Regular expression, and Filter behavior: Add filters that filter logs based on regular expression (RE2 syntax) matches against each incoming line of the raw log. The Filter Behavior determines whether to allow or block the incoming line upon a match. By default, including when the filter behavior is unspecified, the behavior on a match is to block the incoming line and then continue evaluating the next line for a match. For more information, see Regular expression filters.

6. (Syslog collection only) Optional: Toggle Server settings to configure the forwarder's built-in HTTP server, which can be used to configure load balancing and high availability options for syslog collection on Linux. For details about these settings, see HTTP server settings for syslog collection.

7. Click Submit.

   The forwarder is added and Add collector configuration window appears.

8. In the Collector name field, type a name.

9. Click the Log type field to view a list of log types, and do one of the following:

   • If you don't see the log type you want, start typing its name in the box to view more suggestions. For a complete list of supported log types, see Supported data sets.
   • Select a log type from the list.

10. Optional: Expand the Configuration values section and specify any of the following:

    • Asset namespace: Type a namespace that identifies the logs that are collected by this collector. If a namespace is specified for a collector, the collector's namespace is used instead of the forwarder's namespace for the logs from that collector. For more information about asset namespaces, see Asset namespaces.
    • Label key and label value: Type a key and a value. If desired, you can also click Add another to add one or more additional label key:value pairs. For this collector's logs, this setting overrides labels specified at the forwarder level. For details, see Labels.
    • Filter description, Regular expression, and Filter behavior: Add filters that filter logs based on regular expression (RE2 syntax) matches against each incoming line of the raw log. The filter behavior determines whether to allow or block the incoming line upon a match. By default, including when the filter behavior is unspecified, the behavior on a match is to block the incoming line and then continue evaluating the next line for a match. For more information, see Regular expression filters.

11. Optional: Expand the Advanced settings section and specify any of the following:

    • Max seconds per batch: The number of seconds between batches. The default is 10.
    • Max bytes per batch: The number of bytes queued before forwarder batch upload. The default is 1048576.

12. Optional: Disk buffer: Set the toggle to on to enable disk buffering for the collector. For details about disk buffering, see Disk buffering. When enabled, you can specify the following settings:

    • Directory path: The directory path for files written.
    • Max file buffer bytes: The maximum disk size used by the collector before backlogged messages are buffered to disk. The default is 1073741824. The maximum is 4294967296.

13. Click the Collector type field and select a collector type. Each collector type has its own settings that you can configure. For details about the collector types and their settings, see Collector type settings.

14. Click Submit.

Add collectors

You can add one or more collectors to an existing forwarder.

Adding a collector enables you to do the following:

• Name the collector.
• Specify the log type to collect, such as Pan Firewall, Cisco ASA Firewall, and others.
• Specify the collector type: File, Kafka, PCAP, Splunk, Syslog, or WebProxy.
• Specify collector configuration values.

After you add at least one collector to a forwarder, you can download the forwarder configuration and deploy it on a machine or device where Google Security Operations Forwarder is installed.

To add a new collector to a forwarder, follow these steps:

1. In the navigation bar, click Settings.
2. Under Settings, click Forwarders.
3. On the Forwarders page, find the forwarder you want. If the list of forwarders is long, use the Search field.
4. Hold the pointer over the forwarder for which you want to add a collector. The expand menu icon displays.
5. Click the expand menu icon.
6. Choose Add new collector.
7. In the Collector name field, type a name.

8. Click the Log type field to view a list of log types, and do one of the following:

   • If you don't see the log type you want, start typing its name in the box to view more suggestions. For a complete list of supported log types, see Supported data sets.
   • Select a log type from the list.

9. Optional: Expand the Configuration values section and specify any of the following:

   • Asset namespace: Type a namespace that identifies the logs that are collected by this collector. If a namespace is specified for a collector, the collector's namespace is used instead of the forwarder's namespace for the logs from that collector. For more information about asset namespaces, see Asset namespaces.
   • Label key and label value: Type a key and a value. If desired, you can also click Add another to add one or more additional label key:value pairs. For this collector's logs, this setting overrides labels specified at the forwarder level. For details, see Labels.
   • Filter description, Regular expression, and Filter behavior: Add filters that filter logs based on regular expression (RE2 syntax) matches against each incoming line of the raw log. The filter behavior determines whether to allow or block the incoming line upon a match. By default, including when the filter behavior is unspecified, the behavior on a match is to block the incoming line and then continue evaluating the next line for a match. For more information, see Regular expression filters.

10. Optional: Expand the Advanced settings section and specify any of the following:

    • Max seconds per batch: The number of seconds between batches. The default is 10.
    • Max bytes per batch: The number of bytes queued before forwarder batch upload. The default is 1048576.

11. Optional: Disk buffer: Set the toggle to on to enable disk buffering for the collector. For details about disk buffering, see Disk buffering. When enabled, you can specify the following settings:

    • Directory path: The directory path for files written.
    • Max file buffer bytes: The maximum disk size used by the collector before backlogged messages are buffered to disk. The default is 1073741824. The maximum is 4294967296.

12. Click the Collector type field and select a collector type. Each collector type has its own settings that you can configure. For details about the collector types and their settings, see Collector type settings.

13. Click Submit.

Manage forwarders

List the forwarders in a Google Security Operations instance

To list the forwarders in a Google Security Operations instance, follow these steps:

1. In the navigation bar, click Settings.
2. Under Settings, click Forwarders. The page displays the list of forwarders.
3. Optional: Sort the list by clicking the Name or Last updated column.

Optionally, use the search field to narrow the results in your list.

Clone forwarders

Cloning lets you create a copy of one or more forwarder configurations.

To clone forwarders, follow these steps:

1. On the Forwarders page, select the checkbox for each forwarder that you want to clone.

2. Click the expand menu icon.

3. Select Clone selected.

4. Click Clone. A copy of each forwarder is added.

Edit a forwarder configuration

To edit a forwarder configuration, follow these steps:

1. In the navigation bar, click Settings.
2. Under Settings, click Forwarders. The page displays the list of forwarders.

3. Hold the pointer over the forwarder for which you want to edit the configuration. The expand menu icon displays.

4. Click the expand menu icon.

5. Choose Edit forwarder configuration.

6. Make your changes to the configuration. For more information, see the configuration steps in the procedure for adding forwarders.

7. Click Submit.

Delete forwarders

To delete forwarders, follow these steps:

1. On the Forwarders page, select the checkbox for each forwarder that you want to delete.

2. Click the expand menu icon.

3. Select Delete selected.

4. Click Delete selected.

Manage collectors

List the collectors in a Google Security Operations instance

To list the collectors in a Google Security Operations instance, follow these steps:

1. In the navigation bar, click Settings.
2. Under Settings, click Forwarders. The page displays the list of forwarders.
3. Click the expander arrow next to the Name column heading. This expands all of the forwarders, displaying up to five collectors for each forwarder.
4. If a forwarder has more than five collectors, click the See all collectors link.

Edit a collector configuration

To edit a collector configuration, follow these steps:

1. In the navigation bar, click Settings.
2. Under Settings, click Forwarders. The page displays the list of forwarders.

3. Click the expander arrow of the forwarder for which you want to edit a collector.

4. If there are more than five collectors, click the See all collectors link.

5. Hold the pointer over the collector for which you want to edit the configuration. The Edit link displays.

6. Click Edit.

7. Make your changes to the configuration. For more information, see the configuration steps in the procedure for adding collectors.

8. Click Submit.

Delete a collector

To delete a collector, follow these steps:

1. In the navigation bar, click Settings.
2. Under Settings, click Forwarders. The page displays the list of forwarders.

3. Click the expander arrow of the forwarder for which you want to delete a collector.

4. If there are more than five collectors, click the See all collectors link.

5. Hold the pointer over the collector for which you want to edit the configuration. The Delete link displays.

6. Click the Delete link.

7. To confirm, click the Delete button.

Download configuration files

Downloading a forwarder requires at least one collector. If you try to download a forwarder without a collector, you get an error.

You can download the forwarder configuration (.conf) file, authentication (_auth.conf) file, or both, for any forwarder listed in your Google Security Operations instance as long as it has at least one collector. After downloading the files, you'll need to deploy them on the Windows or Linux system where the Google Security Operations Forwarder resides.

To download forwarder configuration files:

1. In the navigation bar, click Settings.
2. Under Settings, click Forwarders. The page displays the list of forwarders.

3. On the Forwarders page, find the forwarder you want. If the list of forwarders is long, use the Search field.

4. Hold the pointer over the forwarder for which you want to download configuration files. The expand menu icon displays.

5. Click the expand menu icon.

6. Choose Download.

7. In the Download forwarder configuration dialog, do one of the following:

   • To download the forwarder configuration file, click the download icon next to the .conf file type.
   • To download the forwarder authentication file, click the download icon next to the _auth.conf file type.
   • To download both files, click Download all.

Configuration settings reference

Forwarder configurations include one or more collectors.

You can configure the following settings at the forwarder level:

• Upload compression

You can configure the following settings at the forwarder level and the collector level. To understand the result of configuring the setting at both levels, see the section for the setting.

• Asset namespaces
• Labels
• Regular expression filters
• HTTP server settings for syslog collection

You can configure the following settings at the collector level:

• Log type
• Disk buffering
• Collector type settings

Upload compression

Default: Enabled

You can configure upload compression for a forwarder, but not for a collector. When enabled, this setting compresses logs before they are uploaded to Google Security Operations. This reduces network bandwidth consumption during transfer to Google Security Operations. However, compression can lead to increased CPU usage.

The tradeoff between bandwidth and CPU usage depends on many factors, including the type of log data, the compressibility of that data, the availability of CPU cycles on the host running the forwarder, and the need for reducing network bandwidth consumption. For example, text based logs compress well and can provide substantial bandwidth savings with low CPU usage. However, encrypted payloads of raw packets do not compress well and incur higher CPU usage.

Asset namespaces

Default: The field is empty if not specified.

You can configure an asset namespace for a forwarder, collector, or both. You can use a namespace to identify logs from distinct network segments and deconflict overlapping IP addresses. Any namespaces that you configure appear with the associated assets in the Google Security Operations user interface. You can also search for namespaces using the Google Security Operations Search feature.

You can specify a namespace for a forwarder and specify different namespaces for one or more collectors of the forwarder. If a namespace is specified for a collector, the collector's namespace is used instead of the forwarder's namespace for the logs from that collector.

For information about using namespaces, see Asset namespaces.

Labels

Default: The fields are empty if not specified.

You can configure labels for a forwarder, collector, or both. Labels are used to attach arbitrary metadata to logs by using key and value pairs. Labels can be configured for an entire forwarder or within a specific collector of a forwarder. If both are provided, the labels are merged with the collector's keys taking precedence over the forwarder's keys if the keys overlap.

Regular expression filters

Default: The fields are empty if not specified.

You can configure regular expression filters for a forwarder, collector, or both. Regular expression filters let you block or allow incoming lines of a raw log that match the expression.

The filters use the RE2 syntax.

The filters must include a regular expression and, optionally, define a behavior when there is a match. The default behavior on a match is block (you can also explicitly configure it as block).

Alternatively, you can specify filters with the allow behavior. If you specify any allow filters, the forwarder blocks any logs that do not match at least one allow filter.

It is possible to define an arbitrary number of filters. Block filters take precedence over allow filters.

When filters are defined, they must be assigned a name. The names of active filters are reported to Google Security Operations by using forwarder health metrics. Filters defined at the forwarder level are merged with filters defined at the collector level. Collector-level filters take precedence in cases of conflicting names. If no filters are defined either at the forwarder or collector level, the behavior is to allow all.

HTTP server settings for syslog collection

The Google Security Operations Forwarder can be deployed in an environment where a Layer 4 load balancer is installed between the data source and forwarder instances. This allows you to distribute the log collection across multiple forwarders or send logs to a different forwarder if one fails. This feature is supported only with the syslog collection type.

The forwarder includes a built-in HTTP server that responds to HTTP health checks from the load balancer. The HTTP server also helps ensure that logs are not lost during startup or shutdown of a forwarder.

The server settings in forwarder configurations support setting timeout durations and status codes returned in response to health checks received in container scheduler and orchestration-based deployments, and from traditional load balancers.

Use the following URL paths for health, readiness, and liveness checks. The <host:port> values are defined in the forwarder configuration.

• http://<host:port>/meta/available: liveness checks for container schedulers/orchestrators, such as Kubernetes.
• http://<host:port>/meta/ready: readiness checks and traditional load balancer health checks.

Setting	Description
Graceful timeout	The amount of time new connections are still accepted after the forwarder returns an unready status in response to a health check. This is also the time to wait between receiving a signal to stop and actually beginning the shutdown of the server itself. This allows the load balancer time to remove the forwarder from the pool. Valid values are in seconds. For example, to specify 10 seconds, type 10. Decimal values are not allowed. Default: 15 seconds
Drain timeout	The amount of time the forwarder waits for active connections to successfully close on their own before being closed by the server. For example, to specify 5 seconds, type 5. Decimal values are not allowed. Default: 10 seconds
Port	The port number that the HTTP server listens on for health checks from the load balancer. The value must be between 1024-65535. Default: 8080
IP address/hostname	The IP address, or a hostname that can be resolved to an IP address, that the server should listen to. Default: 0.0.0.0 (the local system)
Read timeout	Used to tune the HTTP server. Typically, does not need to be changed from the default setting. The maximum amount of time allowed to read the entire request, both the header and the body. You can set both the read timeout field and read header timeout field. Default: 3 seconds
Read header timeout	Used to tune the HTTP server. Typically, does not need to be changed from the default setting. The maximum amount of time allowed to read request headers. The connection's read deadline is reset after reading the header. Default: 3 seconds
Write timeout	Used to tune the HTTP server. Typically, does not need to be changed from the default setting. The maximum amount of time allowed to send a response. It is reset when a new request header is read. Default: 3 seconds
Idle timeout	Used to tune the HTTP server. Typically, does not need to be changed from the default setting. The maximum amount of time to wait for the next request when idle connections are enabled. If the idle timeout field is set to zero, the value of the read timeout field is used. If both are zero, the read header timeout field is used. Default: 3 seconds
Available status code	The status code the forwarder returns when a liveness check is received and the forwarder is available. Container schedulers and orchestrators, such as Kubernetes, often send liveness checks. Default: 204
Ready status code	The status code the forwarder returns when it is ready to accept traffic in either of the following situations: A readiness check is received from a container scheduler or orchestrator, such as Kubernetes. A health check is received from a traditional load balancer. Default: 204
Unready status code	The status code the forwarder returns when it is not ready to accept traffic. Default: 503

Log type

For a complete list of supported log types, see Supported data sets.

Disk buffering

Disk buffering enables you to buffer backlogged messages to disk as opposed to memory. The backlogged messages can be stored in case the forwarder crashes or the underlying host crashes. Be aware that enabling disk buffering can affect performance.

Note: If you receive the error message "Batcher ceiling (xxx) reached, freeing a batch from the backlog," increase the max_file_buffer_bytes to 1 GB or more.

If disk buffering is disabled, the collector uses 1 GB of memory (RAM) for the logs it collects. You can specify the maximum using the Max file buffer bytes setting in the collector configuration. This determines the maximum RAM size used by the collector before backlogged messages are buffered to disk. The default is 1073741824. The maximum is 4294967296.

If you are running the forwarder using Docker, Google recommends mounting a volume separate from your configuration volume for isolation purposes. Also, each input should be isolated with its own directory or volume to avoid conflicts.

Collector type settings

Each collector configuration must specify a collector type. This section describes the collector types and their settings.

• File
• Kafka
• Pcap
• Splunk
• Syslog
• WebProxy

File

Use the file collector type to upload logs from a single log file.

Field	Required or optional field for this type	Description
File path	Required	The directory path and filename. For example: /opt/chronicle/edr/output/sample.txt

Kafka

Use the kafka collector type to ingest data from Kafka topics. Kafka consumer groups are leveraged to enable you to deploy up to three Google Security Operations Forwarders to pull data from the same Kafka topic. For more information, see Kafka. For more information about Kafka consumer groups, see Kafka Consumer.

Field	Required or optional for this type	Description
Username	Required	The username of an identity used for authentication.
Password	Required	The account password associated with the username.
Topic	Required	The Kafka topic from which to ingest data.
Group ID	Required	A group ID.
Timeout	Required	The maximum number of seconds a dial will wait for a connect to complete. Default: 60
Brokers	Optional	Enter a broker in the text box. For example: broker-1:9092 Click Add another to add another broker. Note: All values are replaced during an update operation. Therefore, to update a list of brokers to add a new broker, specify all existing brokers and the new broker.
TLS certificate	Required	The path and certificate filename. For example: /path/to/cert.pem
TLS certificate key	Required	The path and certificate key filename. For example: /path/to/cert.key
Minimum TLS version	Required	The minimum TLS version. Example: TLSv1_3
TLS insecure skip verification	Required	Enables SSL certification verification. Default: disabled

Pcap

This section covers the following topics:

• Using pcap on Windows
• Using pcap on Linux
• Collector settings for pcap

Using pcap on Windows

The Google Security Operations forwarder can capture packets directly from a network interface using Npcap on Windows systems.

Note: Using WinPcap is deprecated. We recommend using Npcap instead.

Contact Google Security Operations Support to update your Google Security Operations forwarder configuration file to support packet capture.

To run a Packet Capture (PCAP) forwarder, you need the following:

• Install Npcap on the Microsoft Windows host.
• On the Windows host, grant Google Security Operations forwarder root or administrator privileges to monitor the network interface.
• No command-line options are needed.
• On the Npcap installation, enable WinPcap compatibility mode.

Using pcap on Linux

Use the pcap collector type to capture packets directly from a network interface using libcap on Linux. For more information on libcap, refer to libcap—Linux manual page.

Packets are captured and sent to Google Security Operations instead of log entries. Packet capture is handled from a local interface only.

Google Security Operations configures Google Security Operations forwarder with the Berkeley Packet Filter (BPF) expression used when capturing packets (for example, port 53 and not localhost). For more information, see Berkeley packet filters.

Collector settings for pcap

The same collector configuration settings apply for a Linux or a Windows host.

Field	Required or optional for this type	Description
Network interface	Required	The interface to listen to for PCAP data. Note: For a Windows host, this is the GUID for the interface used to capture packets. To get this value, run getmac.exe on the machine where Google Security Operations Forwarder is installed (either the server or the machine listening on the span port). The getmac.exe output starts with \Device\Tcpip_. Replace this with \Device\NPF_. Example: \Device\NPF_{2E0E9440-ABFF-4E5B-B43C-E188FCAD1234}
Berkeley packet filter	Required	The Berkeley Packet Filter (BPF) for pcap. Example: udp port 53

Splunk

Use the splunk collector type to collect Splunk data.

Field	Required or optional for this type	Description
Username	Required	The username of an identity used for authentication.
Password	Required	The password of the account identified by the username.
Host	Required	The host or IP address for the Splunk REST API. Example: https://10.0.113.15
Port	Required	The port of the Splunk REST API.
Minimum window size	Required	The minimum time range in seconds passed to the Splunk query. This parameter is used for tuning if the requirement is to change how frequently the Splunk server is queried when the forwarder is in steady state. Also, when there is a lag, the Splunk API call can be made several times. Default: 10
Maximum window size	Required	The maximum time range in seconds passed to the Splunk query. This parameter is used for tuning in cases when there is a lag or if more data is required per query. Change this parameter (equal to or greater than) when you change the minimum parameter. Lag cases can occur if a Splunk query call is taking longer than the maximum window size. Note: The time ranges never overlap when the Splunk server is queried. The time range queried is always between the minimum and maximum window parameters. Default: 30
Query string	Required	The query used to filter records within Splunk. Example: search index=* sourcetype=dns
Query mode	Required	The query mode for Splunk. Example: realtime
Cert ignored	Optional	If enabled, the certificate is ignored. Default: disabled

Syslog

Use the syslog collector type to collect syslog data. You can configure any appliance or server that supports sending syslog data over a TCP or UDP connection to forward its data to Google Security Operations Forwarder. You can control the exact data the appliance or server sends to Google Security Operations Forwarder. Google Security Operations Forwarder can then forward the data to Google Security Operations.

Field	Required or optional for this type	Description
Protocol	Required	The connection protocol the collector will use to listen for syslog data. Valid values are: TCP UDP
Address	Required	The target IP address or hostname where the collector resides and listens for syslog data.
Port	Required	The target port where the collector resides and listens for syslog data.
Buffer size	Required	The size in bytes of the socket's buffer. The default for TCP is 65536. The default for UDP is 8192.
Connection timeout	Required	The number of seconds of inactivity after which the TCP connection is dropped. Default: 60
TLS certificate	Required	The path and certificate filename. For example: /path/to/cert.pem
TLS certificate key	Required	The path and certificate key filename. For example: /path/to/cert.key
Minimum TLS version	Required	The minimum TLS version. Example: TLSv1_3
TLS insecure skip verification	Required	Enables SSL certification verification. Default: disabled

WebProxy

In addition to specifying the field values for Google SecOps Forwarder on Windows, install the Npcap library on the Windows machine or device. This is not required for Google SecOps Forwarder on Linux systems.

Field	Required or optional for this type	Description
Network interface	Required	The interface to listen to for web proxy data.
Berkeley packet filter	Required	The Berkeley Packet Filter (BPF) for the web proxy. Example: udp port 53

Troubleshooting

Syslog data is not received by the forwarder

Make sure the collector's syslog settings are configured to use the correct connection protocol (TCP or UDP) for the incoming data. For more information, see Edit a collector.