Use log rotation for Ops Agent self logs

The Ops Agent writes to a log file called logging-module.log. When the agent runs unattended for long periods or when a problem occurs, this "self log" file can consume all available disk space. This document describes how to use log rotation to prevent this problem.

Ops Agent version 2.31.0 introduces a configurable log-rotation feature built into the agent. If you are running Ops Agent version 2.31.0 or newer, then you can use the built-in log-rotation feature; see Configure log rotation in the Ops Agent.

You can also manage log rotation manually. You might need a manual process if you are using a version of the Ops Agent without built-in log rotation, or if you prefer to rotate your logs manually. See Set up self-log file rotation on Linux VMs for one possible approach.

Configure log rotation in the Ops Agent

This section describes how to modify the default log-rotation configuration used by Ops Agent to rotate its logs automatically. Using this feature requires Ops Agent version 2.31.0 or newer.

Default configuration

The Ops Agent uses the default_self_log_file_rotation entry to configure log rotation. This configuration entry takes three options; the following snippet shows the options and their default values:

  default_self_log_file_rotation:
    enabled: true
    max_file_size_megabytes: 400
    backup_count: 1

The default_self_log_file_rotation configuration takes three options:

enabled: Whether log rotation is enabled; default is true.
max_file_size_megabytes: The maximum size the log file can reach before it is backed up by log rotation. Measured in megabytes (1024² bytes). Default is 400, minimum valid value is 1.
backup_count: The number of old log files to retain. Default is 1, minimum valid value is 1.

User configuration of log rotation

To modify the default log-rotation configuration, you override that configuration by redefining the configuration in the Ops Agent user-configuration file:

On Linux: /etc/google-cloud-ops-agent/config.yaml
On Windows: C:\Program Files\Google\Cloud Operations\Ops Agent\config\config.yaml

To configure log rotation in the Ops Agent, add a global section to the user configuration file and include the configuration element default_self_log_file_rotation in the global section. You might already have logging or metrics pipelines in the this configuration file; add the global section after your pipelines. The result, specifying all options and default values, looks like the following:

logging:  ...
metrics:  ...
global:
  default_self_log_file_rotation:
    enabled: true
    max_file_size_megabytes: 400
    backup_count: 1

Example configurations

To disable log rotation by the Ops Agent, specify the enabled option with the value false:

logging:  ...
metrics:  ...
global:
  default_self_log_file_rotation:
    enabled: false

To rotate the log when the log file reaches 20 MB and keep 5 backups (6 files total):

logging:  ...
metrics:  ...
global:
  default_self_log_file_rotation:
    max_file_size_megabytes: 20
    backup_count: 5

To rotate the log when the log file reaches 2,000 MB (2 GB) and keep 1 backup (2 files total):

logging:  ...
metrics:  ...
global:
  default_self_log_file_rotation:
    max_file_size_megabytes: 2000

To rotate the log when the log file reaches 400 MB and keep 2 backups (3 files total):

logging:  ...
metrics:  ...
global:
  default_self_log_file_rotation:
    backup_count: 2

If you make frequent changes as you refine your log-rotation configuration, remember to restart the agent to apply your changes.

Set up self-log file rotation on Linux VMs

To limit the size of the logging sub-agent log at /var/log/google-cloud-ops-agent/subagents/logging-module.log, install and configure the logrotate utility.

Install the logrotate utility by running the following command:

On Debian and Ubuntu
```
sudo apt install logrotate
```
On CentOS, RHEL and Fedora
```
sudo yum install logrotate
```

Create a logrotate config file at /etc/logrotate.d/google-cloud-ops-agent.conf.

sudo tee /etc/logrotate.d/google-cloud-ops-agent.conf > /dev/null << EOF
# logrotate config to rotate Google Cloud Ops Agent self log file.
# See https://manpages.debian.org/jessie/logrotate/logrotate.8.en.html for
# the full options.
/var/log/google-cloud-ops-agent/subagents/logging-module.log
{
    # Log files are rotated every day.
    daily
    # Log files are rotated this many times before being removed. This
    # effectively limits the disk space used by the Ops Agent self log files.
    rotate 30
    # Log files are rotated when they grow bigger than maxsize even before the
    # additionally specified time interval
    maxsize 256M
    # Skip rotation if the log file is missing.
    missingok
    # Do not rotate the log if it is empty.
    notifempty
    # Old versions of log files are compressed with gzip by default.
    compress
    # Postpone compression of the previous log file to the next rotation
    # cycle.
    delaycompress
}
EOF

Set up crontab or systemd timer to trigger the logrotate utility periodically.

After the log rotation takes effect, you see rotated files in the /var/log/google-cloud-ops-agent/subagents/ directory. The results look similar to the following output:

/var/log/google-cloud-ops-agent/subagents$ ls -lh
total 24K
-rw-r--r-- 1 root root  717 Sep  3 19:54 logging-module.log
-rw-r--r-- 1 root root 6.8K Sep  3 19:51 logging-module.log.1
-rw-r--r-- 1 root root  874 Sep  3 19:50 logging-module.log.2.gz
-rw-r--r-- 1 root root  873 Sep  3 19:50 logging-module.log.3.gz
-rw-r--r-- 1 root root 3.2K Sep  3 19:34 logging-module.log.4.gz

To test log rotation, do the following:

Temporarily reduce the file size at which rotation is triggered by setting the maxsize value to 1k in the /etc/logrotate.d/google-cloud-ops-agent.conf file.
Trigger the agent self log file to be larger than 1K by restarting the agent a few times:
```
sudo service google-cloud-ops-agent restart
```
Wait for the crontab or systemd timer to take effect to trigger the logrotate utility, or trigger the logrotate utility manually by running this command:
```
sudo logrotate /etc/logrotate.d/google-cloud-ops-agent.conf
```
Verify that you see rotated log files in the /var/log/google-cloud-ops-agent/subagents/ directory.
Reset the log-rotation configuration by restoring the original maxsize value.