Configuring the Ops Agent

This feature is covered by the Pre-GA Offerings Terms of the Google Cloud Platform Terms of Service. Pre-GA features may have limited support, and changes to pre-GA features may not be compatible with other pre-GA versions. For more information, see the launch stage descriptions.

Contact us at [e-mail] if you have any questions, need support, or would like to offer feedback.

This page provides details about the Ops Agent's default and custom configurations.

Most users don't need to read this page. Read this page if any of the following applies to you:

• You're interested in learning the technical details of the Ops Agent's configuration.

• You want to change the configuration of the Ops Agent to achieve the following goals:

  • Customize the file path of the log files that the agent collects logs from.

  • Customize the structured log format (for example, json or regex) that the agent uses to process the log entries.

  • Customize which group or groups of metrics to enable. The agent collects all system metrics, like cpu and memory, by default.

Root configuration file location

The Ops Agent configuration file lives in the following location:

• For Linux: /etc/google-cloud-ops-agent/config.yaml.

• For Windows: C:\Program Files\Google\Cloud Operations\Ops Agent\config\config.yaml.

Note: If you make any configuration changes, you must restart the agent to apply the updated configurations.

Default configuration

This section describe the default configuration for the Ops Agent. The configuration defines building blocks of receivers, processors, and exporters, and then defines a pipeline, or a set of pipelines, to tie those blocks together.

On Linux, by default the agent collects file-based syslog and host metrics See details in the definition of hostmetrics type metrics receiver.

logging:
  receivers:
    syslog:
      type: files
      include_paths:
      - /var/log/messages
      - /var/log/syslog
  exporters:
    google:
      type: google_cloud_logging
  service:
    pipelines:
      default_pipeline:
        receivers: [syslog]
        exporters: [google]

metrics:
  receivers:
    hostmetrics:
      type: hostmetrics
      collection_interval: 60s
  exporters:
    google:
      type: google_cloud_monitoring
  service:
    pipelines:
      default_pipeline:
        receivers: [hostmetrics]
        exporters: [google]

On Windows, by default the agent collects Windows event logs from System, Application and Security channels, and IIS and SQL Server metrics, also known as "host metrics".

logging:
  receivers:
    windows_event_log:
      type: windows_event_log
      channels: [System,Application,Security]
  exporters:
    google:
      type: google_cloud_logging
  service:
    pipelines:
      default_pipeline:
        receivers: [windows_event_log]
        exporters: [google]

metrics:
  receivers:
    hostmetrics:
      type: hostmetrics
      collection_interval: 60s
    mssql:
      type: mssql
      collection_interval: 60s
    iis:
      type: iis
      collection_interval: 60s
  exporters:
    google:
      type: google_cloud_monitoring
  service:
    pipelines:
      default_pipeline:
        receivers: [hostmetrics,mssql,iis]
        exporters: [google]

Logging configurations

The logging configuration has the following configuration sub-sections:

• receivers: describes what data, for example, log files, to map into the model.

• processors: a list of processor definitions. The processors in this list can be shared among multiple log input.

• exporters: a list of output definitions. The output in this list can be shared among multiple receivers.

• service: contains a pipelines section that is a list of pipeline definitions. A pipeline connects a list of receivers, a list of processors, and a list of exporters to form the data flow.

Logging receivers

This section contains a map from receiver IDs to receivers. A receiver describes where to retrieve the logs, for example by tailing files, via a tcp port, or from the Windows Event Log. The receivers in this list can be shared among multiple pipelines. Each receiver must specify a unique receiver ID, the type, and additional parameters specific to that type.

Common parameters among all logging receivers:

• type: Required. The type of the logging receiver. Supported types: files and syslog.

Parameters exclusive to files type receivers:

• include_paths: Required. A list of filesystem paths to read by tailing each file. A wild card can be used as part of the path. For example, /var/log/*.log.

• exclude_paths: Optional. A list of filesystem path patterns to exclude from those matched by include_paths.

Parameters exclusive to syslog type receivers:

• transport_protocol: Optional. Supported values: tcp or udp. The default is tcp.

• listen_host: Optional. An IP address to listen on. Only applicable when the transport_protocol is tcp. The default is 0.0.0.0.

• listen_port: Optional. A port to listen on. Only applicable when the transport_protocol is tcp. The default is 5140.

Parameter exclusive to windows_event_log type receivers:

Note: This receiver is for Windows only.

• channels: Required. A list of Windows Event Log channels to read logs from.

Sample files type logging receiver configuration:

receivers:
  <receiver_id>:
    type: files

    include_paths: [/var/log/*.log]
    exclude_paths: [/var/log/not-this-one.log]

Sample syslog type logging receiver configuration:

receivers:
  <receiver_id>:
    type: syslog

    transport_protocol: tcp
    listen_host: 0.0.0.0
    listen_port: 5140

Sample windows_event_log type logging receiver configuration:

receivers:
  <receiver_id>:
    type: windows_event_log

    channels: [System,Application,Security]

Logging processors

This section contains a map from processor IDs to processors. A processor describes additional processing of the logs, for example, parse to json or parse by regex. The processors in this list can be shared among multiple pipelines. Each processor must specify a unique processor ID, the type, and additional parameters specific to that type.

Common parameters for all logging processors:

• type: Required. The type of the logging processor. Supported values: parse_json and parse_regex.

Common parameters for the parse_json and parse_regex logging processors:

• field: Optional. The name of the field in the record to parse. If field isn't provided, then the processor parses the message field.

• time_key: Optional. If the log entry provides a field with a timestamp, this option specifies the name of that field.

• time_format: Only required when time_key is specified. This option specifies the format of the time_key field so it can be recognized and analyzed properly. For details of the format, refer to the strptime(3) guide.

Parameter exclusive to parse_regex type processors:

• regex: Required. The regular expression for parsing the field. The expression must include key names for the matched subexpressions, for example, "^(<time>[^ ]*) (<severity>[^ ]*) (<msg>.*)$".

Sample parse_json type logging processor configuration:

processors:
  <processor_id>:
    type: parse_json

    field:       message
    time_key     time
    time_format  %Y-%m-%dT%H:%M:%S.%L%Z

Sample parse_regex type logging processor configuration:

processors:
  <processor_id>:
    type: parse_regex

    field:       message
    regex:       "^(<time>[^ ]*) (<severity>[^ ]*) (<msg>.*)$"
    time_key     time
    time_format  %Y-%m-%dT%H:%M:%S.%L%Z

Logging exporters

This section contains a map from ID to exporter. An exporter describes which backend to send the logs to. The exporters in this list can be shared among multiple pipelines. Each exporter must specify a unique exporter ID and type.

Common parameter among all logging exporters:

• type: Required. The type of the logging exporter. Supported value: google_cloud_logging. When google_cloud_logging is specified, the Ops Agent translates the processed records into the Google LogEntry proto and sends them to the Logging API. Only one receiver with type google_cloud_logging is allowed.

When Ops Agent receives an unstructured log record, it maps it to textPayload.

For structured log records, Ops Agent treats the following fields specially. You can set specific fields in the LogEntry object that the agent writes to the Logging API.

All fields in the following table are stripped from the payload if present.

Record field	LogEntry field
Option 1 "timestamp": { "seconds": CURRENT_SECONDS, "nanos": CURRENT_NANOS, } Option 2 { "timestampSeconds": CURRENT_SECONDS, "timestampNanos": CURRENT_NANOS, }	timestamp
receiver_id (not a record field)	logName
logging.googleapis.com/httpRequest	httpRequest
logging.googleapis.com/severity	severity
logging.googleapis.com/labels	labels
logging.googleapis.com/operation	operation
logging.googleapis.com/sourceLocation	sourceLocation

Any remaining structured record fields remain part of jsonPayload.

Sample google_cloud_logging type logging exporter configuration:

exporters:
  <exporter_id>:
    type: google_cloud_logging

Logging service

The preceding sections, receivers, processors, and exporters, define components that you can instantiate in the service section.

The service section has a single subsection, pipelines, that contains a map from user-defined pipeline IDs to pipeline definitions. Each pipeline definition consists of three parts:

• receivers: a list of receiver IDs, as defined in the Logging receivers section. The pipeline collects data from all of the listed receivers. The order of the receivers doesn't matter.

• processors: a list of processor IDs, as defined in the Logging processors section. The order of processors here matters. Each record is run through each processor in the listed order.

• exporters: a list of exporter IDs, as defined in the Logging exporters section. The pipeline sends data to all of the listed exporters. The order of the exporters doesn't matter.

Example service section:

service:
  pipelines:
    <pipeline_id>:
      receivers:  [...]  # list of receiver_id
      processors: [...]  # list of processor_id
      exporters:  [...]  # list of exporter_id

Metrics configurations

The metrics configuration has the following configuration sub-sections:

• receivers: a list of receiver definitions. A receiver describes where to retrieve the metrics (for example, system metrics like cpu or memory). The receivers in this list can be shared among multiple pipelines.

• exporters: a list of exporter definitions. An exporter describes which backend to send the metrics to. The exporters in this list can be shared among multiple pipelines.

• service: contains a pipelines section that is a list of pipeline definitions. A pipeline connects a list of receivers, a list of processors, and a list of exporters to form the data flow.

Metrics receivers

This section contains a map from receiver IDs to receivers. A receiver describes where to retrieve the metrics, such as like cpu and memory. The receivers in this list can be shared among multiple pipelines. Each receiver must specify a unique receiver ID, the type, and additional parameters specific to that type.

Common parameters for all metrics receivers:

• type: Required. The type of the metrics receiver. Supported metrics receiver types: "hostmetrics", "iis" (Windows only), "mssql" (Windows only). Only one receiver with each type is allowed.

• collection_interval: Optional. The interval between each metrics collection in the format of a duration. For example, 30s or 2m. The default duration is 60s.

Sample hostmetrics type metrics receiver config:

receivers:
  <receiver_id>:
    type: hostmetrics
    collection_interval: 60s

hostmetrics ingests the following metric data. See details in the Agent Metrics page.

pattern	Metric
cpu	CPU load at 1 minute intervals CPU load at 5 minute intervals CPU load at 15 minute intervals CPU usage, with labels for CPU number and CPU state CPU usage percent, with labels for CPU number and CPU state
disk	Disk bytes read, with label for device Disk bytes written, with label for device Disk I/O time, with label for device Disk weighted I/O time, with label for device Disk pending operations, with label for device Disk merged operations, with labels for device and direction Disk operations, with labels for device and direction Disk operation time, with labels for device and direction Disk usage, with labels for device and state Disk utilization, with labels for device and state
interface	Linux only Total count of network errors Total count of packets sent over the network Total number of bytes sent over the network
memory	Memory usage, with label for state (buffered, cached, free, slab, used) Memory usage percent, with label for state (buffered, cached, free, slab, used)
network	Network error counts, with labels for device and direction Network packets sent, with labels for device and direction Network bytes sent, with labels for device and direction TCP connection count, with labels for port and TCP state
swap	Swap I/O operations, with label for direction Swap bytes used, with labels for device and state Swap percent used, with labels for device and state
pagefile	Windows only Current percentage of pagefile used by state
process	Process count, with label for state Processes forked count
perprocess	Process CPU time, with labels for process name + others (omitted for brevity) Process disk read I/O, with labels for process name + others Process disk write I/O, with labels for process name + others Process RSS usage, with labels for process name + others Process VM usage, with labels for process name + others

Sample iis type metrics receiver config:

receivers:
  <receiver_id>:
    type: iis
    collection_interval: 60s

iis ingests the following metric data. See details in the Agent Metrics page.

pattern	Metric
iis	Windows only Currently open connections to IIS Network bytes transferred by IIS Connections opened to IIS Requests made to IIS

Sample mssql type metrics receiver config:

receivers:
  <receiver_id>:
    type: mssql
    collection_interval: 60s

mssql ingests the following metric data. See details in the Agent Metrics page.

pattern	Metric
mssql	Windows only Currently open connections to SQL server SQL server total transactions per second SQL server write transactions per second

Metrics exporters

This section contains a map from exporter IDs to exporters. An exporter describes which backend to send the metrics to. The exporters in this list can be shared among multiple pipelines. Each exporter must specify a unique exporter ID and type.

Common parameter for all metrics exporters:

• type: Required. The type of the metrics exporter. Supported value: google_cloud_monitoring. When google_cloud_monitoring is specified, the Ops Agent translates the processed records into the Google TimeSeries and sends them to the Monitoring API. Only one receiver with type google_cloud_monitoring is allowed.

Sample google_cloud_monitoring type metrics exporter configuration:

exporters:
  <exporter_id>:
    type: google_cloud_monitoring

Metrics service

The preceding sections, receivers, processors, and exporters, define components that you can instantiate in the service section.

The service section has a single subsection, pipelines, that contains a map from user-defined pipeline IDs to pipeline definitions. Each pipeline definition consists of three parts:

• receivers: a list of receiver IDs, as defined in the Metrics receivers section. The pipeline collects data from all of the listed receivers. The order of the receivers doesn't matter.

• processors: a list of processor IDs, as defined in the Metrics processors section. The order of processors here matters. Each record is run through each processor in the listed order.

• exporters: a list of exporter IDs, as defined in the Metrics exporters section. The pipeline sends data to all of the listed exporters. The order of the exporters doesn't matter.

Example service section:

service:
  pipelines:
    <pipeline_id>:
      receivers:  [...]  # list of receiver_id
      processors: [...]  # list of processor_id
      exporters:  [...]  # list of exporter_id