Structured Logging

In Stackdriver Logging, structured logs refer to log entries that use the jsonPayload field to add structure to their payloads. If you use the Stackdriver Logging API or the command-line utility, gcloud logging, you can control the structure of your payloads. If you use the Stackdriver Logging agent to get your log entries, you can specify that the Stackdriver Logging agent convert your payloads to JSON format. See LogEntry for more information about different payload formats.

Here is an example of a log entry, with the structured payload highlighted in bold:


        {
         insertId:  "1m9mtk4g3mwilhp"
         "jsonPayload: {
          code:  "structured-log-code"
          message:  "This is a log from the log file at test-structured-log.log"
         }
         labels: {
          compute.googleapis.com/resource_name:  "add-structured-log-resource"
         }
         logName:  "projects/my-sample-project-12345/logs/structured-log"
         receiveTimestamp:  "2018-03-21T01:53:41.118200931Z"
         resource: {
          labels: {
           instance_id:  "5351724540900470204"
           project_id:  "my-sample-project-12345"
           zone:  "us-central1-c"
          }
          type:  "gce_instance"
         }
         timestamp:  "2018-03-21T01:53:39.071920609Z"
        }

Structured logging operations

The Stackdriver Logging agent google-fluentd is a modified version of the Fluentd log data collector. The Stackdriver Logging agent comes with the default Fluentd configuration and uses Fluentd input plugins to pull event logs from external sources such as files on disk, or to parse incoming log records.

Fluentd has a list of supported parsers that extract logs and convert them into structured (JSON) payloads.

By configuring a log source with format [PARSER_NAME], you can leverage the built-in parsers provided by Fluentd.

The following code samples show the Fluentd configuration, the input log record, and the output structured payload, which is part of a Stackdriver Logging log entry:

  • Fluentd configuration:

      <source>
        @type tail
    
        format syslog # <--- This uses a predefined log format regex named
                      # `syslog`. See details at https://docs.fluentd.org/v1.0/articles/parser_syslog.
    
        path /var/log/syslog
        pos_file /var/lib/google-fluentd/pos/syslog.pos
        read_from_head true
        tag syslog
      </source>
    
  • Log record (input):

      <6>Feb 28 12:00:00 192.168.0.1 fluentd[11111]: [error] Syslog test
    
  • Structured payload:

        jsonPayload: {
            "pri": "6",
            "host": "192.168.0.1",
            "ident": "fluentd",
            "pid": "11111",
            "message": "[error] Syslog test"
        }
    

For more information about how the syslog parser works, see the detailed Fluentd documentation.

Standard parsers enabled by default

The table below includes the standard parsers that are included in the agent if you enable structured logging:

Parser Name Configuration file
syslog /etc/google-fluentd/config.d/syslog.conf
nginx /etc/google-fluentd/config.d/nginx.conf
apache2 /etc/google-fluentd/config.d/apache.conf
apache_error /etc/google-fluentd/config.d/apache.conf

See the Installation section below for instructions on enabling structured logging when installing the Stackdriver Logging agent.

Installation

To enable structured logging, you must change the default configuration of the Stackdriver Logging agent when installing (or reinstalling) it. Enabling structured logging gets you the latest version of the agent and replaces the previously listed configuration files. There is no change to the operation of the agent itself.

When you enable structured logging, the listed logs will be converted to log entries with different formats than they had before enabling structured logs. If the logs are being exported out of Stackdriver Logging, the change could affect any post-processing applications. In the case of BigQuery exports, BigQuery will reject the new log entries for the rest of the day as having an incorrect schema.

Following are the instructions for installing the agent with structured logging enabled:

  1. Open a terminal connection to your VM instance using SSH or a similar tool.

  2. Download the Stackdriver Logging agent's installation script by running the following command on your VM instance:

        curl -sSO "https://dl.google.com/cloudagents/install-logging-agent.sh"
    
  3. Run the installation script with the following command:

        sudo bash install-logging-agent.sh --structured
    
  4. To verify that the installation was successful, look for the agent's test log entry in the Logs Viewer.

  5. You may delete the installation script after it runs successfully.

You can find the Stackdriver Logging agent configuration files at /etc/google-fluentd/config.d/, which should now include the Standard Parsers Enabled by Default.

For more details, see Installing the Stackdriver Logging agent.

Writing your own parser

If your logs are not supported by the standard parsers, you can write your own. Parsers consist of a regular expression that is used to match log records and apply labels to the pieces.

The following 3 code examples show a log line in the log record, a configuration with a regular expression that indicates the log line's format and the ingested log entry:

  • A log line in the log record:

    REPAIR CAR $500
    
  • A configuration with a regular expression that indidates the log line's format:

    $ sudo vim /etc/google-fluentd/config.d/test-structured-log.conf
    $ cat /etc/google-fluentd/config.d/test-structured-log.conf
    <source>
      @type tail
    
      # Format indicates the log should be translated from text to
      # structured (JSON) with three fields, "action", "thing" and "cost",
      # using the following regex:
      format /(?<action>\w+) (?<thing>\w+) \$(?<cost>\d+)/
      # The path of the log file.
      path /tmp/test-structured-log.log
      # The path of the position file that records where in the log file
      # we have processed already. This is useful when the agent
      # restarts.
      pos_file /var/lib/google-fluentd/pos/test-structured-log.pos
      read_from_head true
      # The log tag for this log input.
      tag structured-log
    </source>
    
  • The resulting log entry:

     {
      insertId:  "eps2n7g1hq99qp"
      jsonPayload: {
        action:  "REPAIR"
        thing:  "CAR"
        cost:  "500"
      }
      labels: {
        compute.googleapis.com/resource_name:  "add-structured-log-resource"
      }
      logName:  "projects/my-sample-project-12345/logs/structured-log"
      receiveTimestamp:  "2018-03-21T01:47:11.475065313Z"
      resource: {
        labels: {
          instance_id:  "3914079432219560274"
          project_id:  "my-sample-project-12345"
          zone:  "us-central1-c"
        }
        type:  "gce_instance"
      }
      timestamp:  "2018-03-21T01:47:05.051902169Z"
     }
    

Viewing structured logs

To search logs and view log entries with the Logs Viewer, see Viewing Logs.

To read log entries using the SDK or API, see Reading log entries.

Troubleshooting

To troubleshoot common issues found with installing or interacting with the Stackdriver Logging agent, see Troubleshooting the Agent.

Send feedback about...

Stackdriver Logging