Contact us at google-cloud-ops-agent@google.com if you have any questions, need support, or would like to offer feedback.
This page provides details about the Ops Agent's default and custom configurations.
Most users don't need to read this page. Read this page if any of the following applies to you:
You're interested in learning the technical details of the Ops Agent's configuration.
You want to change the configuration of the Ops Agent to achieve the following goals:
Customize the file path of the log files that the agent collects logs from.
Customize the structured log format (for example, json or regex) that the agent uses to process the log entries.
Customize which group or groups of metrics to enable. The agent collects all system metrics, like
cpu
andmemory
, by default.
Root configuration file location
The Ops Agent configuration file lives in the following location:
For Linux:
/etc/google-cloud-ops-agent/config.yaml
.For Windows:
C:\Program Files\Google\Cloud Operations\Ops Agent\config\config.yaml
.
Default configuration
This section describe the default configuration for the Ops Agent. The
configuration defines building blocks of receivers
, processors
, and
exporters
, and then defines a pipeline
, or a set of pipelines
, to tie
those blocks together.
On Linux, by default the agent collects file-based syslog
and host metrics
See details in the definition of hostmetrics
type metrics receiver.
logging:
receivers:
syslog:
type: files
include_paths:
- /var/log/messages
- /var/log/syslog
exporters:
google:
type: google_cloud_logging
service:
pipelines:
default_pipeline:
receivers: [syslog]
exporters: [google]
metrics:
receivers:
hostmetrics:
type: hostmetrics
collection_interval: 60s
exporters:
google:
type: google_cloud_monitoring
service:
pipelines:
default_pipeline:
receivers: [hostmetrics]
exporters: [google]
On Windows, by default the agent collects Windows event logs from System
,
Application
and Security
channels, and IIS and SQL Server metrics, also
known as "host metrics".
logging:
receivers:
windows_event_log:
type: windows_event_log
channels: [System,Application,Security]
exporters:
google:
type: google_cloud_logging
service:
pipelines:
default_pipeline:
receivers: [windows_event_log]
exporters: [google]
metrics:
receivers:
hostmetrics:
type: hostmetrics
collection_interval: 60s
mssql:
type: mssql
collection_interval: 60s
iis:
type: iis
collection_interval: 60s
exporters:
google:
type: google_cloud_monitoring
service:
pipelines:
default_pipeline:
receivers: [hostmetrics,mssql,iis]
exporters: [google]
Logging configurations
The logging configuration has the following configuration sub-sections:
receivers
: describes what data, for example, log files, to map into themodel. processors
: a list of processor definitions. The processors in this list can be shared among multiple log input.exporters
: a list of output definitions. The output in this list can be shared among multiplereceivers
.service
: contains apipelines
section that is a list ofpipeline
definitions. Apipeline
connects a list ofreceivers
, a list ofprocessors
, and a list ofexporters
to form the data flow.
Logging receivers
This section contains a map from receiver
IDs to receivers
. A receiver
describes where to retrieve the logs, for example by tailing files, via a tcp
port, or from the Windows Event Log. The receivers
in this list can be shared
among multiple pipelines
. Each receiver
must specify a
unique receiver
ID, the type
, and additional parameters specific to that
type
.
Common parameters among all logging receivers
:
type
: Required. The type of the loggingreceiver
. Supportedtypes
:files
andsyslog
.
Parameters exclusive to files
type receivers
:
include_paths
: Required. A list of filesystem paths to read by tailing each file. A wild card can be used as part of the path. For example,/var/log/*.log
.exclude_paths
: Optional. A list of filesystem path patterns to exclude from those matched byinclude_paths
.
Parameters exclusive to syslog
type receivers
:
transport_protocol
: Optional. Supported values:tcp
orudp
. The default istcp
.listen_host
: Optional. An IP address to listen on. Only applicable when thetransport_protocol
istcp
. The default is0.0.0.0
.listen_port
: Optional. A port to listen on. Only applicable when thetransport_protocol
istcp
. The default is5140
.
Parameter exclusive to windows_event_log
type receivers
:
channels
: Required. A list of Windows Event Log channels to read logs from.
Sample files
type logging receiver
configuration:
receivers:
<receiver_id>:
type: files
include_paths: [/var/log/*.log]
exclude_paths: [/var/log/not-this-one.log]
Sample syslog
type logging receiver
configuration:
receivers:
<receiver_id>:
type: syslog
transport_protocol: tcp
listen_host: 0.0.0.0
listen_port: 5140
Sample windows_event_log
type logging receiver
configuration:
receivers:
<receiver_id>:
type: windows_event_log
channels: [System,Application,Security]
Logging processors
This section contains a map from processor
IDs to processors
. A processor
describes additional processing of the logs, for example, parse to json or parse
by regex. The processors
in this list can be shared among multiple
pipelines
. Each processor
must specify a unique
processor
ID, the type
, and additional parameters specific to that type
.
Common parameters for all logging processors
:
type
: Required. The type of the loggingprocessor
. Supported values:parse_json
andparse_regex
.
Common parameters for the parse_json
and parse_regex
logging processors
:
field
: Optional. The name of the field in the record to parse. Iffield
isn't provided, then theprocessor
parses themessage
field.time_key
: Optional. If the log entry provides a field with a timestamp, this option specifies the name of that field.time_format
: Only required whentime_key
is specified. This option specifies the format of thetime_key
field so it can be recognized and analyzed properly. For details of the format, refer to thestrptime(3)
guide.
Parameter exclusive to parse_regex
type processors
:
regex
: Required. The regular expression for parsing the field. The expression must include key names for the matched subexpressions, for example,"^(<time>[^ ]*) (<severity>[^ ]*) (<msg>.*)$"
.
Sample parse_json
type logging processor
configuration:
processors:
<processor_id>:
type: parse_json
field: message
time_key time
time_format %Y-%m-%dT%H:%M:%S.%L%Z
Sample parse_regex
type logging processor
configuration:
processors:
<processor_id>:
type: parse_regex
field: message
regex: "^(<time>[^ ]*) (<severity>[^ ]*) (<msg>.*)$"
time_key time
time_format %Y-%m-%dT%H:%M:%S.%L%Z
Logging exporters
This section contains a map from ID to exporter
. An exporter
describes which
backend to send the logs to. The exporters
in this list can be shared among
multiple pipelines
. Each exporter
must specify a unique
exporter
ID and type
.
Common parameter among all logging exporters:
type
: Required. The type of the loggingexporter
. Supported value:google_cloud_logging
. Whengoogle_cloud_logging
is specified, the Ops Agent translates the processed records into the Google LogEntry proto and sends them to the Logging API. Only onereceiver
with typegoogle_cloud_logging
is allowed.
When Ops Agent receives an unstructured log record, it maps it to
textPayload
.
For structured log records, Ops Agent treats the following fields
specially. You can set specific fields in the LogEntry
object that the agent
writes to the Logging API.
All fields in the following table are stripped from the payload if present.
Record field | LogEntry field |
---|---|
Option 1
Option 2
|
timestamp |
receiver_id (not a record field) | logName |
logging.googleapis.com/httpRequest |
httpRequest |
logging.googleapis.com/severity |
severity |
logging.googleapis.com/labels |
labels |
logging.googleapis.com/operation |
operation |
logging.googleapis.com/sourceLocation |
sourceLocation |
Any remaining structured record fields remain part of jsonPayload
.
Sample google_cloud_logging
type logging exporter
configuration:
exporters:
<exporter_id>:
type: google_cloud_logging
Logging service
The preceding sections, receivers
, processors
, and exporters
, define
components that you can instantiate in the service
section.
The service
section has a single subsection, pipelines
, that contains a map
from user-defined pipeline
IDs to pipeline
definitions. Each pipeline
definition consists of three parts:
receivers
: a list ofreceiver
IDs, as defined in the Logging receivers section. Thepipeline
collects data from all of the listedreceivers
. The order of thereceivers
doesn't matter.processors
: a list ofprocessor
IDs, as defined in the Logging processors section. The order ofprocessors
here matters. Each record is run through each processor in the listed order.exporters
: a list ofexporter
IDs, as defined in the Logging exporters section. Thepipeline
sends data to all of the listedexporters
. The order of theexporters
doesn't matter.
Example service
section:
service:
pipelines:
<pipeline_id>:
receivers: [...] # list of receiver_id
processors: [...] # list of processor_id
exporters: [...] # list of exporter_id
Metrics configurations
The metrics configuration has the following configuration sub-sections:
receivers
: a list of receiver definitions. Areceiver
describes where to retrieve the metrics (for example, system metrics likecpu
ormemory
). Thereceivers
in this list can be shared among multiplepipelines
.exporters
: a list ofexporter
definitions. Anexporter
describes which backend to send the metrics to. Theexporters
in this list can be shared among multiplepipelines
.service
: contains apipelines
section that is a list ofpipeline
definitions. Apipeline
connects a list ofreceivers
, a list ofprocessors
, and a list ofexporters
to form the data flow.
Metrics receivers
This section contains a map from receiver
IDs to receivers
. A receiver
describes where to retrieve the metrics, such as like cpu
and memory
. The
receivers
in this list can be shared among multiple
pipelines
. Each receiver
must specify a unique
receiver
ID, the type
, and additional parameters specific to that type
.
Common parameters for all metrics receivers
:
type
: Required. The type of the metrics receiver. Supported metrics receiver types: "hostmetrics", "iis" (Windows only), "mssql" (Windows only). Only one receiver with each type is allowed.collection_interval
: Optional. The interval between each metrics collection in the format of a duration. For example,30s
or2m
. The default duration is60s
.
Sample hostmetrics
type metrics receiver config:
receivers:
<receiver_id>:
type: hostmetrics
collection_interval: 60s
hostmetrics
ingests the following metric data. See details in the
Agent Metrics page.
pattern |
Metric |
---|---|
cpu |
CPU load at 1 minute intervals CPU load at 5 minute intervals CPU load at 15 minute intervals CPU usage, with labels for CPU number and CPU state CPU usage percent, with labels for CPU number and CPU state |
disk |
Disk bytes read, with label for device Disk bytes written, with label for device Disk I/O time, with label for device Disk weighted I/O time, with label for device Disk pending operations, with label for device Disk merged operations, with labels for device and direction Disk operations, with labels for device and direction Disk operation time, with labels for device and direction Disk usage, with labels for device and state Disk utilization, with labels for device and state |
interface |
Linux only Total count of network errors Total count of packets sent over the network Total number of bytes sent over the network |
memory |
Memory usage, with label for state (buffered, cached, free, slab, used) Memory usage percent, with label for state (buffered, cached, free, slab, used) |
network |
Network error counts, with labels for device and direction Network packets sent, with labels for device and direction Network bytes sent, with labels for device and direction TCP connection count, with labels for port and TCP state |
swap |
Swap I/O operations, with label for direction Swap bytes used, with labels for device and state Swap percent used, with labels for device and state |
pagefile |
Windows only Current percentage of pagefile used by state |
process |
Process count, with label for state Processes forked count |
perprocess |
Process CPU time, with labels for process name + others (omitted for brevity) Process disk read I/O, with labels for process name + others Process disk write I/O, with labels for process name + others Process RSS usage, with labels for process name + others Process VM usage, with labels for process name + others |
Sample iis
type metrics receiver config:
receivers:
<receiver_id>:
type: iis
collection_interval: 60s
iis
ingests the following metric data. See details in the
Agent Metrics page.
pattern |
Metric |
---|---|
iis |
Windows only Currently open connections to IIS Network bytes transferred by IIS Connections opened to IIS Requests made to IIS |
Sample mssql
type metrics receiver config:
receivers:
<receiver_id>:
type: mssql
collection_interval: 60s
mssql
ingests the following metric data. See details in the
Agent Metrics page.
pattern |
Metric |
---|---|
mssql |
Windows only Currently open connections to SQL server SQL server total transactions per second SQL server write transactions per second |
Metrics exporters
This section contains a map from exporter
IDs to exporters
. An exporter
describes which backend to send the metrics to. The exporters
in this list can
be shared among multiple pipelines
. Each exporter
must
specify a unique exporter
ID and type
.
Common parameter for all metrics exporters
:
type
: Required. Thetype
of the metricsexporter
. Supported value:google_cloud_monitoring
. Whengoogle_cloud_monitoring
is specified, the Ops Agent translates the processed records into the Google TimeSeries and sends them to the Monitoring API. Only one receiver with typegoogle_cloud_monitoring
is allowed.
Sample google_cloud_monitoring
type metrics exporter configuration:
exporters:
<exporter_id>:
type: google_cloud_monitoring
Metrics service
The preceding sections, receivers
, processors
, and exporters
, define
components that you can instantiate in the service
section.
The service
section has a single subsection, pipelines
, that contains a map
from user-defined pipeline
IDs to pipeline
definitions. Each pipeline
definition consists of three parts:
receivers
: a list ofreceiver
IDs, as defined in the Metrics receivers section. Thepipeline
collects data from all of the listedreceivers
. The order of thereceivers
doesn't matter.processors
: a list ofprocessor
IDs, as defined in the Metrics processors section. The order ofprocessors
here matters. Each record is run through each processor in the listed order.exporters
: a list ofexporter
IDs, as defined in the Metrics exporters section. Thepipeline
sends data to all of the listedexporters
. The order of theexporters
doesn't matter.
Example service
section:
service:
pipelines:
<pipeline_id>:
receivers: [...] # list of receiver_id
processors: [...] # list of processor_id
exporters: [...] # list of exporter_id