Installing and configuring the forwarder on Linux

This document describes how to install and configure the forwarder on Linux. You can install the forwarder on a variety of Linux distributions (Debian, Ubuntu, Red Hat, Suse, etc.). Google Cloud provides the software using a Docker container. You can run and manage the Docker container on either a physical or virtual machine running Linux.

Customize the configuration files

Based on the information you submitted prior to deployment, Google Cloud provides you with the forwarder software and configuration files. Google Cloud taylors your configuration files to the forwarder instance on your network. If you need to alter the configuration, contact Chronicle Support.

System requirements

The following are general recommendations. For recommendations specific to your system, contact Chronicle Support.

  • RAM—1.5 GB for each collected data type. For example, endpoint detection and response (EDR), DNS, and DHCP are all separate data types. You would need 4.5 GB of RAM to collect data for all three.

  • CPU—2 CPUs are sufficient to handle less than 10,000 events per second (EPS) (total for all data types). If you expect to forward more than 10,000 EPS, provision 4 to 6 CPUs.

  • Disk—100 MB of disk space is sufficient, regardless of how much data the Chronicle forwarder handles. See also Disk Buffering if you need to buffer backlogged messages to disk as opposed to memory. The Chronicle forwarder buffers to memory by default.

Verify the firewall configuration

If you have firewalls or authenticated proxies in between the Chronicle forwarder container and the Internet, they require rules to open access to the following hosts:

Connection Type Destination Port
TCP malachiteingestion-pa.googleapis.com 443
TCP accounts.google.com 443
TCP gcr.io 443

Install Docker

You can install Docker on a variety of host operating systems. The actual installation of Docker is dependent on the host environment. Google Cloud provides limited documentation to assist you in installing Docker on several of the more popular Linux distributions. However, Docker is open source and substantial documentation is already available.

Once you have installed Docker on your system, the rest of the Chronicle forwarder installation process is identical, regardless of which Linux distribution you are using.

Check that Docker is installed on your system by executing the following command (elevated privileges):

# docker ps

The following response indicates that Docker has been installed properly:

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

You can gather additional information about your Docker installation using the following command:

# docker info

If you have problems with Docker, Chronicle Support might request the output from this command to help with debugging.

Install the forwarder

This section describes how to install Chronicle Forwarder using a Docker container on a Linux system.

Step 1. Download, transfer and install the forwarder configuration files

Google Cloud provides forwarder configuration files specific to your operating system (Linux or Windows). Download the files from the link provided by your Chronicle representative to a local directory on your laptop (for example, named chronicle). After you complete the following steps, transfer the configuration files from your laptop to your forwarder ~/config directory within the user’s home directory.

  1. Connect to your Linux forwarder via terminal.

  2. Create a new user on the Linux forwarder.

    # adduser <username>

    # passwd <username>

    # usermod -aG wheel <username>

  3. Change directory to the home directory of the new user that will run Docker Container.

  4. Create a directory to store the Chronicle forwarder configuration files: # mkdir ~/config

  5. Change directory

  6. Once files have been transferred, confirm the configuration files are located in the ~/config directory:

    # ls -l

Step 2. Install the key for the Chronicle Container Registry

Google Cloud provided JSON file (the file has a .json extension) contains the credentials needed to access the Chronicle Docker registry.

  1. Copy the following command text (including the line breaks) to a text editor and rename the .json file name within brackets to your Chronicle Docker Authentication file name.

      docker login \
    
         -u _json_key \
    
         --password-stdin \
    
         https://gcr.io < ./chronicle-container-c2da10b71454-oneline.json
    
  2. Once edit is complete, copy the full command text and run the command from within the ~/config directory on the Linux forwarder.

  3. Output of the above docker login command should be Login Succeeded.

If you run into an issue, confirm the Docker version running:

# docker version

Older versions of Docker (for example, 1.13.1) may require different command line arguments. The differences are noted below.

Step 2 for older versions of Docker that don't support --password-stdin

For older versions of Docker (versions that do not accept --password-stdin), copy the contents of your JSON file and paste them at the password prompt.

  1. Issue the following command:

    $ cat ./<filename>.json

  2. Copy the output of the cat command.

  3. Login to Docker:

    docker login -u _json_key https://gcr.io

  4. At the Password: prompt, paste the clipboard contents.

    Regardless of how you provide the password (--password-stdin or copy and paste), the output of the docker login command should be Login Succeeded.

Step 3. Run the forwarder within the Docker container

You can use the following procedures to start Chronicle forwarder the first time as well as to upgrade to the latest version of the Chronicle container:

The --log-opt options have been available since Docker 1.13. These options limit the size of the container log files and should be used as long as your version of Docker supports them.

  1. If you are upgrading, start by cleaning up any previous Docker runs. In the following example, the name of the Docker container is cfps, then obtain the latest Docker image from Google Cloud with the # docker pull command below.

    # docker stop cfps
    # docker rm cfps
    
  2. Obtain the latest Docker image from Google Cloud:

    # docker pull gcr.io/chronicle-container/cf_production_stable

  3. Start Chronicle forwarder from the Docker container:

    docker run \
    
      --detach \
    
      --name cfps \
    
      --restart=always \
    
      --log-opt max-size=100m \
    
      --log-opt max-file=10 \
    
      --net=host \
    
      -v ~/config:/opt/chronicle/external \
    
      gcr.io/chronicle-container/cf_production_stable
    

These commands are also provided as a shell script called run_docker_production_stable.sh.

The Docker container (and Chronicle forwarder) persist after system reboots.

Step 4. Monitor and manage the forwarder

The following Docker commands help you to monitor and manage Chronicle forwarder:

  • Check if the Docker container is running:

    docker ps

  • Display the logs from the container. Note that this can generate a substantial volume of output, but is useful for debugging:

    docker logs cfps

  • To see what is running inside the container:

    docker top cfps

  • To stop the container:

    docker stop cfps

Collect Splunk data

You can configure Chronicle forwarder to forward your Splunk data to Chronicle. Google Cloud configures Chronicle forwarder with the following information to forward your data from Splunk:

  • URL for the Splunk REST API (for example, https://10.0.113.15:8889).

  • Splunk queries to generate data for each of the required data types (for example, index=dns).

You need to make your Splunk account credentials available to Chronicle forwarder.

Create a local file for your Splunk credentials and name it creds.txt. Place your username on the first line and the password on the second line:

  cat creds-file

  myusername
  mypassword

For customers who are using Chronicle forwarder to access a Splunk instance, copy the creds.txt file to the config directory (the same directory as the configuration files). For example:

cp creds-file ~/config/creds.txt

Verify the creds.txtem> file is in its proper location:

ls ~/config

Collect syslog data

Chronicle forwarder can operate as a Syslog server, meaning you can configure any appliance or server that supports sending syslog data over a TCP or UDP connection to forward their data to Chronicle forwarder. You can control exactly what data the appliance or server sends to Chronicle forwarder. Chronicle forwarder can then forward the data to Chronicle.

The configuration file (provided by Google Cloud) specifies which ports to monitor for each type of forwarded data (for example, port 10514). By default, Chronicle forwarder accepts both TCP and UDP connections.

Configure rsyslog

To configure rsyslog, you need to specify a target for each port (for example, each data type). Consult your system documentation for the correct syntax. The following examples illustrate an rsyslog target configuration:

  • TCP log traffic: dns.* @192.168.0.12:10514

  • UDP log traffic: dns.* @192.168.0.12:10514

Enable TLS for syslog configurations

You can enable TLS for the Syslog connection to the Chronicle forwarder. In the Chronicle forwarder configuration file, specify the location of your certificate and certificate key as shown in the following example:

certificate "/opt/chronicle/external/certs/edb3ae966a7bbe1f.pem"
certificate_key "/opt/chronicle/external/certs/forwarder.key"

Based on the example shown, the Chronicle forwarder configuration would be modified as follows:

  collectors:
- syslog:
    common:
      enabled: true
      data_type: WINDOWS_DNS
      data_hint:
      batch_n_seconds: 10
      batch_n_bytes: 1048576
  tcp_address: 0.0.0.0:10515
  connection_timeout_sec: 60
  certificate: "/opt/chronicle/external/certs/edb3ae966a7bbe1f.pem"
  certificate_key: "/opt/chronicle/external/certs/forwarder.key"

You can create a certs directory under the configuration directory and store the certificate files there.

Collect packet data

Chronicle forwarder can capture packets directly from a network interface using libpcap on Linux. Packets are captured and sent to Chronicle instead of log entries. Packet capture is handled from a local interface only. To enable packet capture for your system, contact Chronicle Support.

Google Cloud configures Chronicle forwarder with the Berkeley Packet Filter (BPF) expression used when capturing packets (for example, port 53 and not localhost).

Toggle data compression

Log compression reduces network bandwidth consumption when transferring logs to Chronicle. However, compression might cause an increase in CPU usage. The tradeoff between CPU usage and bandwidth depends on many factors, including the type of log data, the compressibility of that data, the availability of CPU cycles on the host running the forwarder and the need for reducing network bandwidth consumption.

For example, text based logs compress well and can provide substantial bandwidth savings with low CPU usage. However, encrypted payloads of raw packets do not compress well and incur higher CPU usage.

Since most of the log types ingested by the forwarder are efficiently compressible, log compression is enabled by default to reduce bandwidth consumption. However, if the increased CPU usage outweighs the benefit of the bandwidth savings, you can disable compression by setting the compression field to false in the Chronicle forwarder configuration file as shown in the following example:

output:
  compression: false
    url: malachiteingestion-pa.googleapis.com:443
    identity:
      identity:
      collector_id: 10479925-878c-11e7-9421-10604b7cb5c1
      customer_id: ebdc4bb9-878b-11e7-8455-10604b7cb5c1
      secret_key: |
        {
          "type": "service_account",
...

Disk Buffering

Disk buffering enables you to buffer backlogged messages to disk as opposed to memory. The backlogged messages can be stored in case the forwarder crashes or the underlying host crashes. Be aware that enabling disk buffering can impact performance.

If you are running the Forwarder using Docker, Google recommends mounting a volume separate from your configuration volume for isolation purposes. Also, each input should be isolated with its own directory or volume to avoid conflicts.

Example configuration: disk buffering

The following configuration includes syntax to enable disk buffering:

collectors:
- syslog:
    common:
      write_to_disk_buffer_enabled: true
      # /buffers/NIX_SYSTEM is part of the external mounted volume for the forwarder
      write_to_disk_dir_path: /buffers/NIX_SYSTEM
      max_file_buffer_bytes: 1073741824
      batch_n_bytes: 1048576
      batch_n_seconds: 10
      data_hint: null
      data_type: NIX_SYSTEM
      enabled: true
    tcp_address: 0.0.0.0:30000
    connection_timeout_sec: 60
- syslog:
    common:
      batch_n_bytes: 1048576
      batch_n_seconds: 10
      data_hint: null
      data_type: WINEVTLOG
      enabled: true
    tcp_address: 0.0.0.0:30001
    connection_timeout_sec: 60

Regular Expression Filters

Regular expression filters enable you to filter logs based on regular expression matches against raw logs.

The filters employ the RE2 syntax described here: https://github.com/google/re2/wiki/Syntax

The filters must include a regular expression and, optionally, define a behavior when there is a match. The default behavior on a match is block (you can also explicitly configure it as block).

Alternatively, you can specify filters with the allow behavior. If you specify any allow filters, the forwarder blocks any logs that do not match at least one allow filter.

It is possible to define an arbitrary number of filters. Block filters take precedence over allow filters.

When filters are defined, they must be assigned a name. The names of active filters will be reported to Chronicle via Forwarder health metrics. Filters defined at the root of the configuration are merged with filters defined at the collector level. Collector level filters take precedence in cases of conflicting names. If no filters are defined either at the root or collector level, the behavior is to allow all.

Example configuration: regular expression filters

In the following Forwarder configuration, WINEVTLOG logs that do not match the root filter (allow_filter) are blocked. Given the regular expression, the filter only allows logs with priorities between 0 and 99. However, any NIX_SYSTEM logs containing 'foo' or 'bar' are blocked, despite allow_filter. This is because the filters use a logical OR. All logs are processed until a filter is triggered.

regex_filters:
  allow_filter:
    regexp: ^<[1-9][0-9]?$>.*$
    behavior_on_match: allow
collectors:
- syslog:
    common:
      regex_filters:
        block_filter_1:
          regexp: ^.*foo.*$
          behavior_on_match: block
        block_filter_2:
          regexp: ^.*bar.*$
      batch_n_bytes: 1048576
      batch_n_seconds: 10
      data_hint: null
      data_type: NIX_SYSTEM
      enabled: true
    tcp_address: 0.0.0.0:30000
    connection_timeout_sec: 60
- syslog:
    common:
      batch_n_bytes: 1048576
      batch_n_seconds: 10
      data_hint: null
      data_type: WINEVTLOG
      enabled: true
    tcp_address: 0.0.0.0:30001
    connection_timeout_sec: 60

Arbitrary labels

Labels are used to attach arbitrary metadata to logs using key and value pairs. Labels can be configured for an entire Forwarder or within a specific collector of a Forwarder. If both are provided, the labels are merged with the collector's keys taking precedence over the Forwarder's keys if the keys overlap.

Example configuration: arbitrary labels

In the following Forwarder configuration, the 'foo=bar' and 'meow=mix' key and value pairs are both attached to WINEVTLOG logs, and the 'foo=baz' and 'meow=mix' key and value pairs are attached to the NIX_SYSTEM logs.

metadata:
  labels:
    foo: bar
    meow: mix
collectors:
syslog:
    common:
      metadata:
        labels:
          foo: baz
          meow: mix
      batch_n_bytes: 1048576
      batch_n_seconds: 10
      data_hint: null
      data_type: NIX_SYSTEM
      enabled: true
    tcp_address: 0.0.0.0:30000
    connection_timeout_sec: 60
syslog:
    common:
      batch_n_bytes: 1048576
      batch_n_seconds: 10
      data_hint: null
      data_type: WINEVTLOG
      enabled: true
    tcp_address: 0.0.0.0:30001
    connection_timeout_sec: 60

Namespaces

Use namespace labels to identify logs from distinct network segments and deconflict overlapping IP addresses. You can configure a namespace label for an entire Forwarder or within a specific collector of the Forwarder. If both are included, the specific collector's namespace takes precedence.

Any namespace configured for the Forwarder appears with the associated assets in the Chronicle user interface. You can also search for namespaces using the Chronicle Search feature.

For information about how to view namespaces in the Chronicle user interface, see here.

Example configuration: namespaces

In the following Forwarder configuration, WINEVTLOG logs are attached to the FORWARDER namespace and NIX_SYSTEM logs are attached to the CORPORATE namespace.

metadata:
  namespace: FORWARDER
collectors:
- syslog:
      common:
        metadata:
          namespace: CORPORATE
        batch_n_bytes: 1048576
        batch_n_seconds: 10
        data_hint: null
        data_type: NIX_SYSTEM
        enabled: true
      tcp_address: 0.0.0.0:30000
      connection_timeout_sec: 60
- syslog:
      common:
        batch_n_bytes: 1048576
        batch_n_seconds: 10
        data_hint: null
        data_type: WINEVTLOG
        enabled: true
      tcp_address: 0.0.0.0:30001
      connection_timeout_sec: 60

Kafka Input

You can ingest data from Kafka topics just as you can for syslog. Consumer groups are leveraged to enable you to deploy up to 3 Forwarders and pull data from the same Kafka topic.

For more information on Kafka consumer groups, see the following: https://docs.confluent.io/platform/current/clients/consumer.html

Example configuration: Kafka input

The following Forwarder configuration shows how to setup the forwarder to ingest data from Kafka topics:

collectors:
- kafka:
      common:
        batch_n_bytes: 1048576
        batch_n_seconds: 10
        data_hint: null
        data_type: NIX_SYSTEM
        enabled: true
      username: user
      password: password
      topic: example-topic
      group_id: chronicle-forwarder
      timeout: 60
      brokers:
      - broker-1:9093
      - broker-2:9093
      tls:
        insecureSkipVerify: true
        certificate: "/path/to/cert.pem"
        certificate_key: "/path/to/cert.key"
- syslog:
      common:
        batch_n_bytes: 1048576
        batch_n_seconds: 10
        data_hint: null
        data_type: WINEVTLOG
        enabled: true
      tcp_address: 0.0.0.0:30001
      connection_timeout_sec: 60

Load Balancing and High Availability

The Chronicle Linux forwarder can be deployed in an environment where a Layer 4 load balancer is installed between the data source and forwarder instances. This allows a customer to distribute log collection across multiple forwarders or send logs to a different forwarder if one fails. This feature is supported with the syslog collection type only.

The Linux forwarder includes a built-in HTTP server that responds to HTTP health checks from the load balancer. The HTTP server also helps ensure that logs are not lost during startup or shutdown of a forwarder.

Configure the HTTP server, load balancing, and high availability options under the server section of the forwarder configuration file. These options support setting timeout durations and status codes returned in response to health checks received in container scheduler and orchestration-based deployments, as well as from traditional load balancers.

Use the following url paths for health, readiness, and liveness checks. The <host:port> values are defined in the forwarder configuration.

  • http://<host:port>/meta/available: liveness checks for container schedulers/orchestrators, such as Kubernetes.
  • http://<host:port>/meta/ready: readiness checks and traditional load balancer health checks.

The following forwarder configuration is an example for load balancing and high availability:

collectors:
- syslog:
    common:
      batch_n_bytes: 1048576
      batch_n_seconds: 10
      data_hint: null
      data_type: NIX_SYSTEM
      enabled: true
    tcp_address: 0.0.0.0:30000
    connection_timeout_sec: 60
- syslog:
    common:
      batch_n_bytes: 1048576
      batch_n_seconds: 10
      data_hint: null
      data_type: WINEVTLOG
      enabled: true
    tcp_address: 0.0.0.0:30001
    connection_timeout_sec: 60
server:
  graceful_timeout: 15s
  drain_timeout: 10s
  http:
    port: 8080
    host: 0.0.0.0
    read_timeout: 3s
    read_header_timeout: 3s
    write_timeout: 3s
    idle_timeout: 3s
    routes:
    - meta:
        available_status: 204
        ready_status: 204
        unready_status: 503
Configuration path Description
server : graceful_timeout Amount of time the forwarder returns a bad readiness/health check and still accepts new connections. This is also the time to wait between receiving a signal to stop and actually beginning the shutdown of the server itself. This allows the load balancer time to remove the forwarder from the pool.
server : drain_timeout Amount of time the forwarder waits for active connections to successfully close on their own before being closed by the server.
server : http : port Port number that the HTTP server listens on for health checks from the load balancer. Must be between 1024-65535.
server : http : host IP address, or hostname that can be resolved to IP addresses, that the server should listen on. If empty, the default value is local system (0.0.0.0).
server : http : read_timeout Used to tune the HTTP server. Typically, does not need to be changed from the default setting. Maximum amount of time allowed to read the entire request, both the header and the body. You can set both read_timeout and read_header_timeout.
server : http : read_header_timeout Used to tune the HTTP server. Typically, does not need to be changed from the default setting. Maximum amount of time allowed to read request headers. The connection's read deadline is reset after reading the header.
server : http : write_timeout Used to tune the HTTP server. Typically, does not need to be changed from the default setting. Maximum amount of time allowed to send a response. It is reset when a new request header is read.
server : http : idle_timeout Used to tune the HTTP server. Typically, does not need to be changed from the default setting. Maximum amount of time to wait for the next request when idle connections are enabled. If idle_timout is zero, the value of read_timout is used. If both are zero, read_header_timeout is used.
routes : meta : ready_status Status code the forwarder returns when it is ready to accept traffic in either of the following situations:
  • Readiness check is received from a container scheduler or orchestrator, such as Kubernetes.
  • Health check is received from a traditional load balancer.
routes : meta : unready_status Status code the forwarder returns when it is not ready to accept traffic.
routes : meta : available_status Status code the forwarder returns when a liveness check is received and the forwarder is available. Liveness checks are often sent by container schedulers/orchestrators such as Kubernetes.

Frequently asked questions

What is a Docker container?

  • Docker containers, like virtual machines, provide additional security, isolation and resource management.

  • Virtual Machines—have both a privileged space (linux kernel) and a user space (everything you interact with: libc, python, ls, tcpdump, etc).

  • Containers—have only a user space (everything you interact with: libc, python, ls, tcpdump, etc) and relies on the host's privilege space.

Why distribute Chronicle forwarder using a container?

  • Better security through isolation:
    • Customer environment and requirements do not affect Chronicle forwarder.
    • Chronicle forwarder environment and requirements do not affect the customer.
    • Container distribution mechanism already exists and can be private and separate for Google Cloud and customers. https://cloud.google.com/container-registry/

Why only Linux for containers? What about Windows?

  • Containers were developed for Linux first and are production ready.

  • Windows support for Containers is improving. Containers are available for Windows Server 2016 and Windows 10.

Do you need to learn advanced Docker commands?

  • Chronicle forwarder uses a single container, so there is no need to learn about Swarm, orchestration, Kubernetes, or other advanced Docker concepts or commands.