Troubleshoot common Linux forwarder issues

This page helps you recognize and troubleshoot common issues that you may face while using the Google Security Operations Linux forwarder.

Forwarder does not start

Forwarder fails to start and is in a continuous restart loop with the following error in the logs:

F0510 06:17:39.013603 202 main_linux.go:153] open /opt/chronicle/external/*.conf: no such file or directory

Possible cause 1: Incorrect mapping in the configuration file

To resolve this issue, ensure that you are passing the correct path to the configuration file and mapping it to an external folder.

Possible cause 2: SELinux is enabled

Check the configuration file by getting into the container without starting the forwarder and run the following command:

docker run --name cfps --log-opt max-size=100m --log-opt   max-file=10 --net=host -v ~/configuration:/opt/chronicle/external --entrypoint=/bin/bash \
-it gcr.io/chronicle-container/cf_production_

This command will put you in a shell from within the container.

Run the following command:
```
ls -lrt /opt.chronicle/external/ 
```
If you get a permission denied error, it indicates that the forwarder is unable to open the configuration file and therefore, not able to start.

To resolve this issue, do the following:

Check the SELinux status by running the following command:
```
  sestatus
```
If SELinux status is enabled in the output, run the following command to disable it:
```
  setenforce 0
```

Logs not reaching the Google Security Operations tenant

Possible cause 1: DNS resolution

Check if the host is unable to resolve addresses or to reach Google Security Operations by running the following command:

  nslookup malachiteingestion-pa.googleapis.com

If the command fails, contact your networking team for resolution.

Possible cause 2: Firewall

Check if the local firewall is blocking communication between Google Security Operations and the forwarder by running the following command:

  firewall-cmd --state

If the firewall is enabled, disable it by running the following command:

  systemctl stop firewalld

Possible cause 3: Buffer size

Check if it's a buffer size issue by looking for the following error in the logs:

Memory ceiling (1073741824) reached, freeing a batch from the backlog

To resolve this issue, do the following:

Enable compression in the forwarder configuration file.
Increase the buffer size by updating the max_memory_buffer_bytes and max_file_buffer_bytes parameters in the configuration file. These parameters indicate the buffer of backlog batches that are stored in the memory or disk.

Forwarder and host not receiving logs

If the host and the forwarder do not receive logs, check the port status by running the following command for each port:

  netstat -a | grep PORT

Replace PORT with the port ID that you want to check.

If the command does not give any output, it indicates that the host is not listening on that port and you should consult your network administrator.

If the output from the command indicates that the host is listening on a port and the forwarder is still not receiving logs, do the following:

Stop Docker by running the following command:
```
docker stop cfps
```
Run one of the following commands based on your network setup.

For TCP:
```
nc -l PORT
```
For UDP:
```
nc -l -u PORT
```
Replace PORT with the port ID that you want to troubleshoot.
Restart your external service and check if the issue is resolved. If the issue persists, [contact Google Security Operations support.

Forwarder not receiving logs but host is receiving logs

If the host is receiving logs but the forwarder is not, it indicates that the forwarder is not listening on the port specified in the configuration file.

To resolve this issue, do the following:

Open two terminal windows on your system, one to configure the forwarder and one to send a test message to the host.
1. On the first terminal (forwarder), start Docker without starting the forwarder by running the following command:
```
docker run \
--name cfps \
--log-opt max-size=100m \ 
--log-opt max-file=10 \
--net=host \
-v ~/config:/opt/chronicle/external \
--entrypoint=/bin/bash \
-it gcr.io/chronicle-container/cf_production_stable
```
2. Specify the port that the forwarder should listen on:
```
nc -l PORT
```
  Replace PORT with the port ID that you want to troubleshoot.
On the second terminal (host), send the test message on the port by running the following command:
```
echo "test message" | nc localhost PORT
```
Replace PORT with the port ID that you want to troubleshoot.

Rerun the docker command. Specify the -p flag with the ports on which the forwarder should listen by running the following command:

docker run \
--detach \
–name cfps \
--restart=always \
--log-opt max-size=100m \
--log-opt max-file=10 --net=host \
—v /root/config:/opt/chronicle/external \
-p 11500:11800 \
gcr.io/chronicle-container/cf_production_stable