Use log rotation for Ops Agent self logs

The Ops Agent writes to a log file called logging-module.log. This "self log" file can, when running unattended for long periods or when a problem occurs, consume all available disk space. This document describes how to use log rotation to prevent this problem.

Set up self log file rotation on Linux VMs

To limit the size of the logging sub-agent log at /var/log/google-cloud-ops-agent/subagents/logging-module.log, install and configure the logrotate utility.

Install the logrotate utility by running the following command:

On Debian and Ubuntu
```
sudo apt install logrotate
```
On CentOS, RHEL and Fedora
```
sudo yum install logrotate
```

Create a logrotate config file at /etc/logrotate.d/google-cloud-ops-agent.conf.

sudo tee /etc/logrotate.d/google-cloud-ops-agent.conf > /dev/null << EOF
# logrotate config to rotate Google Cloud Ops Agent self log file.
# See https://manpages.debian.org/jessie/logrotate/logrotate.8.en.html for
# the full options.
/var/log/google-cloud-ops-agent/subagents/logging-module.log
{
    # Log files are rotated every day.
    daily
    # Log files are rotated this many times before being removed. This
    # effectively limits the disk space used by the Ops Agent self log files.
    rotate 30
    # Log files are rotated when they grow bigger than maxsize even before the
    # additionally specified time interval
    maxsize 256M
    # Skip rotation if the log file is missing.
    missingok
    # Do not rotate the log if it is empty.
    notifempty
    # Old versions of log files are compressed with gzip by default.
    compress
    # Postpone compression of the previous log file to the next rotation
    # cycle.
    delaycompress
}
EOF

Set up crontab or systemd timer to trigger the logrotate utility periodically.

After the log rotation takes effect, you see rotated files in the /var/log/google-cloud-ops-agent/subagents/ directory. The results look similar to the following output:

/var/log/google-cloud-ops-agent/subagents$ ls -lh
total 24K
-rw-r--r-- 1 root root  717 Sep  3 19:54 logging-module.log
-rw-r--r-- 1 root root 6.8K Sep  3 19:51 logging-module.log.1
-rw-r--r-- 1 root root  874 Sep  3 19:50 logging-module.log.2.gz
-rw-r--r-- 1 root root  873 Sep  3 19:50 logging-module.log.3.gz
-rw-r--r-- 1 root root 3.2K Sep  3 19:34 logging-module.log.4.gz

To test log rotation, do the following:

Temporarily reduce the file size at which rotation is triggered by setting the maxsize value to 1k in the /etc/logrotate.d/google-cloud-ops-agent.conf file.
Trigger the agent self log file to be larger than 1K by restarting the agent a few times:
```
sudo service google-cloud-ops-agent restart
```
Wait for the crontab or systemd timer to take effect to trigger the logrotate utility, or trigger the logrotate utility manually by running this command:
```
sudo logrotate /etc/logrotate.d/google-cloud-ops-agent.conf
```
Verify that you see rotated log files in the /var/log/google-cloud-ops-agent/subagents/ directory.
Reset the log-rotation configuration by restoring the original maxsize value.