Configuring the Agent

This page provides details about the Stackdriver Logging agent's default and custom configurations.

Most users will not need to read this page. Read this page if:

  • You are interested in learning deep technical details of the Stackdriver Logging agent's configuration.

  • You want to change the configuration of the Stackdriver Logging agent.

Default configuration

The Logging agent google-fluentd is a modified version of the fluentd log data collector. The Stackdriver Logging agent comes with a default configuration; in most common cases, no additional configuration is required.

In its default configuration, the Logging agent streams logs, as included in the list of default logs, to Stackdriver Logging. You can configure the agent to stream additional logs; see the Customizing the Stackdriver Logging agent configuration section below.

How the Logging Agent Works

The Logging agent uses fluentd input plugins to retrieve and pull event logs from external sources, such as files on disk, or to parse incoming log records. Input plugins are bundled with the agent or can be installed separately as Ruby gems; see the list of bundled plugins.

The agent reads log records stored in log files using on the VM instance via fluentd's built-in in_tail plugin. Each log record is converted to a log entry structure for Stackdriver Logging. The content of each log record is mostly recorded in the payload of the log entries, but log entries also contain standard elements like a timestamp and severity. The Stackdriver Logging agent requires every log record to be tagged with a string-format tag; all of the filters and output plugins match a specific set of tags. The log name usually follows the format, projects/[PROJECT-ID]/logs/[TAG]. For example, this log name includes the tag structured-log:

    projects/my-sample-project-12345/logs/structured-log

The output plugin transforms each internalized structured message to a log entry in Stackdriver Logging. The payload becomes the text or JSON payload.

The following sections on this page discuss the default configuration in detail.

Default configuration definitions

The following sections describe the default configuration definitions for syslog, the forward input plugin, input configurations for third-party application logs, such as those in the list of default logs, and our Google Cloud fluentd output plugin.

Syslog configuration

  • Configuration file locations:

    • Linux: /etc/google-fluentd/config.d/syslog.conf
    • Windows: C:\Program Files (x86)\Stackdriver\LoggingAgent\fluent.conf

      If you are running a Logging agent before v1-5, the location is: C:\GoogleStackdriverLoggingAgent\fluent.conf

  • Description: This file includes the configuration to specify syslog as a log input.

  • See the config repository.

Configuration name Type Default Description
format string /^(?<message>(?<time>[^ ]*\s*[^ ]* [^ ]*) .*)$/ The format of the syslog.
path string /var/log/syslog The path of the syslog file.
pos_file string /var/lib/google-fluentd/pos/syslog.pos The path of the position file for this log input. fluentd will record the position it last read into this file. See detailed fluentd documentation.
read_from_head bool true Whether to start to read the logs from the head of file instead of the bottom. See detailed fluentd documentation.
tag string syslog The log tag for this log input.

in_forward input plugin configuration

  • Configuration file location: /etc/google-fluentd/config.d/forward.conf
  • Description: This file includes the configuration to configure the in_forward fluentd input plugin. The in_forward input plugin allows you to pass in logs via a TCP socket.
  • See the detailed fluentd documentation for this plugin and the config repository.
Configuration name Type Default Description
port int 24224 The port to monitor.
bind string 127.0.0.1 The bind address to monitor. By default, only connections from localhost are accepted. To open this up, this configuration needs to be changed to 0.0.0.0.

Third-party application log input configuration

  • Configuration file location: /etc/google-fluentd/config.d/[APPLICATION_NAME].conf.
  • Description: This directory includes configuration files to specify third-party applications’ log files as log inputs. Each file, except syslog.conf and forward.conf, represents one application (e.g., apache.conf for the Apache application).
  • See the config repository.
Configuration name Type Default Description
format string Varies per application The format of the log. See the detailed fluentd documentation.
path string Varies per application The path of the log file(s). Multiple paths can be specified, separated by ‘,’. * and strftime format can be included to add/remove watch file dynamically. See detailed fluentd documentation.
pos_file string Varies per application The path of the position file for this log input. fluentd will record the position it last read into this file. See detailed fluentd documentation.
read_from_head bool true Whether to start to read the logs from the head of file instead of the bottom. See detailed fluentd documentation.
tag string Varies; the name of the application. The log tag for this log input.

Google Cloud fluentd output plugin configuration

  • Configuration file location: /etc/google-fluentd/google-fluentd.conf
  • Description: This file includes configuration options to control the Google Cloud fluentd output plugin’s behavior.
  • See the config repository.
Configuration name Type Default Description
buffer_chunk_limit string 1M As log records come in, those that cannot be written to downstream components fast enough will be pushed into a queue of chunks. This configuration sets the size limit of each chunk. By default, we set the chunk limit conservatively to avoid exceeding the recommended chunk size of 5MB per write request in Logging API. A buffer chunk gets flushed if one of the two conditions are met:
1. flush_interval kicks in.
2. The buffer size reaches buffer_chunk_limit.
flush_interval string 5s As log records come in, those that cannot be written to downstream components fast enough will be pushed into a queue of chunks. The configuration sets how long before we have to flush a chunk buffer. A buffer chunk gets flushed if one of the two conditions are met:
1. flush_interval kicks in.
2. The buffer size reaches buffer_chunk_limit.
disable_retry_limit bool false Enforces a limit on the number of retries of failed flush of buffer chunks. See detailed specifications in retry_limit, retry_wait and max_retry_wait.
retry_limit int 3 When a buffer chunk fails to be flushed, fluentd by default will retry later. This configuration sets how many retries to perform before dropping one problematic buffer chunk.
retry_wait int 10s When a buffer chunk fails to be flushed, fluentd by default will retry later. This configuration sets the wait interval in seconds before the first retry. The wait interval will be doubled on each following retry (20s, 40s,...) until eitherretry_ limit or max_retry_wait is reached.
max_retry_wait int 300 When a buffer chunk fails to be flushed, fluentd by default will retry later. The wait interval will be doubled on each following retry (20s, 40s,...) This configuration sets the maximum of wait intervals in seconds. If the wait interval reaches this limit, the doubling stops.
num_threads int 8 The number of simultaneous log flushes that can be processed by the output plugin.
use_grpc bool true Whether to use gRPC instead of REST/JSON to communicate to the Logging API. With gRPC enabled, CPU usage will typically be lower.
partial_success bool true Whether to support partial success for logs ingestion. If true, invalid log entries in a full set are dropped, and valid log entries are successfully ingested into the Logging API. If false, the full set would be dropped if it contained any invalid log entries.
enable_monitoring bool true When set to true, the Logging agent will expose two metrics, a request count metric that keeps track of the number of log entries requested to be sent to Stackdriver Logging and an ingested entry count that keeps track of the actual number of log entries successfully ingested by Stackdriver Logging. When false, these metrics are not exposed.
monitoring_type string prometheus The type of monitoring. The only option today is prometheus, but we may support additional options in the future. If enable_monitoring is true and monitoring_type is prometheus, the Logging agent will expose some local metrics in Prometheus format at localhost:24231/metrics. See details.
autoformat_stackdriver_trace bool true When set to true, the trace will be reformatted if the value of structured payload field logging.googleapis.com/trace matches ResourceTrace traceId format. Details of the autoformatting can be found under Special fields in structured payloads.

Processing payloads

Most of the supported logs under the default configuration of the Logging agent are from log files and are ingested as unstructured (text) payloads in the log entries.

The only exception is that the in_forward input plugin, which is also enabled by default, only accepts structured logs and ingests them as structured (JSON) payloads in the log entries. See Streaming structured (JSON) log records via in_forward plugin for more information about this plugin.

You can customize the agents's configuration to support ingesting structured logs from additional resources. See Streaming structured (JSON) log records to Stackdriver Logging for details.

The payload of log records streamed by a custom-configured Logging agent can be either a single unstructured text message (textPayload) or a structured JSON message (jsonPayload).

Special fields in structured payloads

When the Logging agent receives a structured log record, it treats the following fields specially, allowing you to set specific fields in the LogEntry object that get written to the Logging API.

All fields in the table below are stripped from the payload if present.

JSON log field LogEntry field Logging agent function
severity severity The Logging agent attempts to match a variety of common severity strings. This includes the list of LogSeverity strings recognized by the Logging API.
message textPayload (or part of jsonPayload) If, after stripping special-purpose fields, there is only a message field left, that message is saved as textPayload. If your log entry contains an exception stack trace, the exception stack trace should be set in this message JSON log field, so that the exception stack trace can be parsed and saved to Stackdriver Error Reporting.
log textPayload (or part of jsonPayload) Only applies to Cloud Functions and Google Kubernetes Engine: if, after stripping special purpose fields, there is only a log field left, that log is saved as textPayload.
httpRequest httpRequest This is stripped from jsonPayload and assigned to the LogEntry HttpRequest field.
time-related fields timestamp For details, see Time-related fields.
logging.googleapis.com/trace trace logging.googleapis.com/trace is stripped from jsonPayload and assigned to the LogEntry trace field. For example, assume the value of logging.googleapis.com/trace is [V]. [V] should be formatted as projects/[PROJECT-ID]/traces/[TRACE-ID], so it can be used by the Logs Viewer and the Trace Viewer to group log entries and display them in line with traces. If autoformat_stackdriver_trace is true and [V] matches the format of ResourceTrace traceId, the LogEntry trace field will have the value projects/[PROJECT-ID]/traces/[V].
logging.googleapis.com/spanId spanId This field is stripped from jsonPayload and assigned to the LogEntry spanId field.
logging.googleapis.com/operation operation This is stripped from jsonPayload and assigned to LogEntryOperation. This field is also used by the Logs Viewer to group related log entries.
logging.googleapis.com/sourceLocation sourceLocation This is stripped from jsonPayload and assigned to LogEntrySourceLocation.

Any remaining structured record fields become part of jsonPayload. If there is only a message field remaining, the value of that field is stored as textPayload in the log entry.

Time-related fields

The Logging agent can receive and process time-related fields in several JSON formats. If one of the following JSON timestamp representations is present in a structured record, the Logging agent strips the fields from jsonPayload and collapses them into a single representation in the timestamp field in the LogEntry object:

  • jsonPayload contains a timestamp field that includes the seconds and nanos fields:

    {
      "timestamp": {
        "seconds": CURRENT_SECONDS,
        "nanos": CURRENT_NANOS
      }
    }
    
  • jsonPayload contains both the timestampSeconds and timestampNanos fields:

    {
       "timestampSeconds": CURRENT_SECONDS,
       "timestampNanos": CURRENT_NANOS
    }
    
  • jsonPayload contains a time field:

    {
        "time": CURRENT_TIME
    }
    

Once the Logging agent detects a timestamp representation, no further timestamp-related stripping occurs, even if additional representations of acceptable formatting are present in the structured record.

Customizing agent configuration

Besides the list of default logs that the Logging agent streams by default, you can customize the Logging agent to send additional logs to Logging or to adjust agent settings by adding input configurations.

The configuration definitions in these sections apply to the fluent-plugin-google-cloud output plugin only and specify how logs are transformed and ingested into Stackdriver Logging.

  • Configuration file location: /etc/google-fluentd/google-fluentd.conf
  • Description: This file includes configuration options to control the fluent-plugin-google-cloud output plugin’s behavior.
  • See the config repository.

Streaming logs from additional inputs

You can customize the Logging agent to send additional logs to Logging by adding input configurations.

Streaming unstructured (text) logs via log files

  1. Create a log file using the following command:

    $ touch /tmp/test-unstructured-log.log
    
  2. Create a new configuration file in /etc/google-fluentd/config.d with the following configuration. Alternatively, you can add the configuration into any existing configuration files in /etc/google-fluentd as well:

    $ sudo vim /etc/google-fluentd/config.d/test-unstructured-log.conf
    $ cat /etc/google-fluentd/config.d/test-unstructured-log.conf
    <source>
        @type tail
        # Format 'none' indicates the log is unstructured (text).
        format none
        # The path of the log file.
        path /tmp/test-unstructured-log.log
        # The path of the position file that records where in the log file
        # we have processed already. This is useful when the agent
        # restarts.
        pos_file /var/lib/google-fluentd/pos/test-unstructured-log.pos
        read_from_head true
        # The log tag for this log input.
        tag unstructured-log
    </source>
    
  3. Restart the agent to apply the configuration changes:

    $ sudo service google-fluentd restart
    
  4. Generate a log record into the log file:

    $ echo 'This is a log from the log file at test-unstructured-log.log' >> /tmp/test-unstructured-log.log
    
  5. Check the Logs Viewer to see the ingested log entry:

      {
       insertId:  "eps2n7g1hq99qp"
       labels: {
        compute.googleapis.com/resource_name:  "add-unstructured-log-resource"
       }
       logName:  "projects/my-sample-project-12345/logs/unstructured-log"
       receiveTimestamp:  "2018-03-21T01:47:11.475065313Z"
       resource: {
        labels: {
         instance_id:  "3914079432219560274"
         project_id:  "my-sample-project-12345"
         zone:  "us-central1-c"
        }
        type:  "gce_instance"
       }
       textPayload:  "This is a log from the log file at test-unstructured-log.log"
       timestamp:  "2018-03-21T01:47:05.051902169Z"
      }
    

Streaming structured (JSON) logs via log files

You can enable structured logging for certain log inputs when installing the Logging agent. For more information, see Structured Logging.

You can also customize the Logging agent to add a JSON log input. Creating this configuration makes the agent expect each log record to be a JSON object:

  1. Create a log file:

    $ touch /tmp/test-structured-log.log
    
  2. Create a new configuration file in /etc/google-fluentd/config.d with the following configuration. Alternatively, you can add the configuration into any existing configuration files in /etc/google-fluentd as well:

    $ sudo vim /etc/google-fluentd/config.d/test-structured-log.conf
    $ cat /etc/google-fluentd/config.d/test-structured-log.conf
    <source>
        @type tail
        # Format 'JSON' indicates the log is structured (JSON).
        format json
        # The path of the log file.
        path /tmp/test-structured-log.log
        # The path of the position file that records where in the log file
        # we have processed already. This is useful when the agent
        # restarts.
        pos_file /var/lib/google-fluentd/pos/test-structured-log.pos
        read_from_head true
        # The log tag for this log input.
        tag structured-log
    </source>
    
  3. Restart the agent to apply the configuration changes:

    $ sudo service google-fluentd restart
    
  4. Generate a log record into the log file:

    $ echo '{"code": "structured-log-code", "message": "This is a log from the log file at test-structured-log.log"}' >> /tmp/test-structured-log.log
    
  5. Check the Logs Viewer to see the ingested log entry:

    {
     insertId:  "1m9mtk4g3mwilhp"
     jsonPayload: {
      code:  "structured-log-code"
      message:  "This is a log from the log file at test-structured-log.log"
     }
     labels: {
      compute.googleapis.com/resource_name:  "add-structured-log-resource"
     }
     logName:  "projects/my-sample-project-12345/logs/structured-log"
     receiveTimestamp:  "2018-03-21T01:53:41.118200931Z"
     resource: {
      labels: {
       instance_id:  "5351724540900470204"
       project_id:  "my-sample-project-12345"
       zone:  "us-central1-c"
      }
      type:  "gce_instance"
     }
     timestamp:  "2018-03-21T01:53:39.071920609Z"
    }
    

For additional options to customize your log input format for common third-party applications, see Common Log Formats and How To Parse Them.

Streaming structured (JSON) logs via in_forward plugin

Additionally, you can send logs via the fluentd in_forward plugin. fluentd-cat is a built-in tool that helps easily send logs to the in_forward plugin. The fluentd documentation contains more details for this tool.

To send logs via the fluentd in_forward plugin, see the following instructions:

  1. Execute the following command on the VM with the Logging agent installed::

     $ echo '{"code": "send-log-via-fluent-cat", "message": "This is a log from in_forward plugin."}' | /opt/google-fluentd/embedded/bin/fluent-cat log-via-in-forward-plugin
    
  2. Check the Logs Viewer to see the ingested log entry:

      {
       insertId:  "1kvvmhsg1ib4689"
       jsonPayload: {
        code:  "send-log-via-fluent-cat"
        message:  "This is a log from in_forward plugin."
       }
       labels: {
        compute.googleapis.com/resource_name:  "add-structured-log-resource"
       }
       logName:  "projects/my-sample-project-12345/logs/log-via-in-forward-plugin"
       receiveTimestamp:  "2018-03-21T02:11:27.981020900Z"
       resource: {
        labels: {
         instance_id:  "5351724540900470204"
         project_id:  "my-sample-project-12345"
         zone:  "us-central1-c"
        }
        type:  "gce_instance"
       }
       timestamp:  "2018-03-21T02:11:22.717692494Z"
      }
    

Streaming structured (JSON) log records from application code

You can enable connectors in various languages to send structured logs from application code; for more information, see the fluentd documentation. These connectors are built based on the in_forward plugin.

Setting log entry labels

The configuration options below let you override LogEntry labels and MonitoredResource labels when ingesting logs to Stackdriver Logging. All log entries are associated with monitored resources; for more information, see list of Stackdriver Logging monitored resource types.

Configuration name Type Default Description
label_map hash nil label_map (specified as a JSON object) is an unordered set of fluentd field names whose values are sent as labels rather than as part of the structured payload. Each entry in the map is a {field_name: label_name} pair. When field_name (as parsed by the input plugin) is encountered, a label with the corresponding label_name is added to the log entry. The value of the field is used as the value of the label. The map gives you the additional flexibility in specifying label names, including the ability to use characters which would not be legal as part of fluentd field names. See Setting labels in structured log entries for example.
labels hash nil labels (specified as a JSON object) is a set of custom labels provided at configuration time. It allows you to inject extra environmental information into every message or to customize labels otherwise detected automatically. Each entry in the map is a {label_name: label_value} pair.

The Logging agent output plugin supports three ways to set LogEntry labels:

Setting labels in structured log entries

Suppose you wrote a structured log entry payload like this:

{ "message": "This is a log message", "timestamp": "Aug 10 20:07:00", "env": "production" }

And suppose you want to translate the above payload field env to a metadata label environment. To do this, add the following to your output plugin configuration in /etc/google-fluentd/google-fluentd.conf:

  # Configure all sources to output to Stackdriver Logging
  <match **>
    type google_cloud
    label_map {
      "env": "environment"
    }
    ...
  </match>

The label_map setting here replaces the env label in the payload with environment, so the resulting log entry will have a label environment with the value production.

Setting labels statically

If you don't have this information in the payload, and simply want to add a static metadata label called environment, add the following to your output plugin configuration in /etc/google-fluentd/google-fluentd.conf:

  # Configure all sources to output to Stackdriver Logging
  <match **>
    type google_cloud
    labels {
      "environment": "production"
    }
    ...
  </match>

In this case, instead of using a map to replace one label with another, we use a labels setting to attach a label with a given literal value to a log entry, regardless of whether the entry already has a label or not. This approach can be used even if you are sending unstructured logs.

For more on how to configure labels, label_map and other Logging agent settings, see Setting Log Entry Labels.

Modifying Log Records

Fluentd provides built-in filter plugins that can be used to modify log entries.

The most commonly used filter plugin is filter_record_transformer. It enables you to:

  • Add new fields to log entries
  • Update fields in log entries
  • Delete fields in log entries

Some output plugins also let you modify log entries. The fluent-plugin-record-reformer output plugin provides functionality similar to the filter_record_transformer filter plugin, except that it also allows you to modify log tags. More resource usage is expected with this plugin: each time a log tag is updated, it generates a new log entry with the new tag. Note that the tag field in the configuration is required; we also recommend that you modify this field to avoid entering a dead loop.

The fluent-plugin-detect-exceptions output plugin scans a log stream, either unstructured (text) or JSON-format log records, for multi-line exception stack traces. If a consecutive sequence of log entries forms an exception stack trace, the log entries are forwarded as a single, combined log message. Otherwise, the log entry is forwarded as it was.

Advanced (non-default) configuration definitions

If you want to customize the configuration of your Logging agent, beyond its default configuration, see the following sections.

The configuration options below let you adjust the Logging agent's internal buffering mechanism.

Configuration name Type Default Description
buffer_type string buf_memory Records that cannot be written to the Logging API fast enough will be pushed into a buffer. The buffer can be in memory or in actual files. Recommended value: buf_file. The default buf_memory is fast but not persistent. There is risk of losing logs. If buffer_type is buf_file, buffer_path needs to be specified as well.
buffer_path string User-specified The path where buffer chunks are stored. This parameter is required if buffer_type is file. This configuration must be unique to avoid a race condition.
buffer_queue_limit int 64 Specifies the length limit of the chunk queue. When the buffer queue reaches this many chunks, the buffer behavior is controlled by buffer_queue_full_action. By default, it will throw exceptions.
buffer_queue_full_action string exception Controls the buffer behavior when the buffer queue is full. Possible values:
1. exception: Throw BufferQueueLimitError when the queue is full. How BufferQueueLimitError is handled depends on input plugins. For example, the in_tail input plugin stops reading new lines while the in_forward input plugin returns an error.
2. block: This mode stops input plugin thread until the buffer full condition is resolved. This action is a good for batch-like use cases. fluentd does not recommend using block action to avoid BufferQueueLimitError. If you hit BufferQueueLimitError frequently, it means your destination capacity is insufficient for your traffic.
3. drop_oldest_chunk: This mode drops the oldest chunks.

The configuration options below let you manually specify a project and certain fields from the MonitoredResource object. These values are automatically gathered by the Logging agent; it is not recommended that you manually specify them.

Configuration name Type Default Description
project_id string nil If specified, this overrides the project_id identifying the underlying GCP or AWS project in which the Logging agent is running.
zone string nil If specified, this overrides the zone.
vm_id string nil If specified, this overrides the VM id.
vm_name string nil If specified, this overrides the VM name.

Other output plugin configuration options

Configuration name Type Default Description
detect_json1 bool false Whether to try to detect if the log record is a text log entry with JSON content that needs to be parsed. If this configuration is true, and an unstructured (text) log entry is detected as in JSON format, it will be parsed and sent as structured (JSON) payload.
coerce_to_utf8 bool true Whether to allow non-UTF-8 characters in user logs. If set to true, any non-UTF-8 character would be replaced by the string specified by non_utf8_replacement_string. If set to false, any non-UTF-8 character would trigger the plugin to error out.
require_valid_tags bool false Whether to reject log entries with invalid tags. If this option is set to false, tags will be made valid by converting any non-string tag to a string, and sanitizing any non-UTF-8 or other invalid characters.
non_utf8_replacement_string string ""(space) If coerce_to_utf8 is set to true, any non-UTF-8 character would be replaced by the string specified here.

1When the record has only a single textual message field, the output plugin can sometimes detect when this message is JSON and transform the payload to JSON. This feature is enabled by default in VM instances running on App Engine flexible environment and Google Kubernetes Engine.

Applying customized agent configuration

Customizing the Logging agent allows you to add your own fluentd configuration files:

Linux instance

  1. Copy your configuration files into the following directory:

    /etc/google-fluentd/config.d/
    
  2. Restart the agent by running the following command:

    sudo service google-fluentd force-reload
    

The Logging agent installation script populates this directory with the default catch-all configuration files. For more information, see Getting the Logging agent source code.

Windows instance

  1. Copy your config files into the config.d subdirectory of your agent-installation directory. If you accepted the default installation directory, this directory is:

    C:\Program Files (x86)\Stackdriver\LoggingAgent\config.d\
    
  2. Restart the agent by running the following commands in a command-line shell:

    net stop  StackdriverLogging
    net start StackdriverLogging
    

For more information on fluentd configuration files, see fluentd's Configuration File Syntax documentation.

Was this page helpful? Let us know how we did:

Send feedback about...

Stackdriver Logging