Reading and Writing Application Logs

Overview

When a request is sent to your app, a request log is automatically written by App Engine. During the handling of the request, your app can also write application logs. In this page, you'll learn how to write application logs from your application, how to read both application and request logs programmatically using the Logs API, how to view logging in the Google Cloud Platform Console, and how to understand the request log data that App Engine writes during the request.

Request logs vs application logs

There are two categories of log data: request logs and application logs. A request log is automatically written by App Engine for each request handled by your app, and contains information such as the project ID, HTTP version, and so forth. For a complete list of available properties for request logs, see RequestLogs. See also the request log table for descriptions of the request log fields.

Each request log contains a list of application logs (AppLogLine) associated with that request, returned in the RequestLogs.getAppLogLines() method. Each app log contains the time the log was written, the log message, and the log level.

Writing application logs

The App Engine Java SDK allows a developer to log the following levels of severity:

  • SEVERE
  • WARNING
  • INFO
  • CONFIG
  • FINE
  • FINER
  • FINEST

In order to add logging to your Java app, you need to add the appropriate system properties to your project's appengine-web.xml file, and you might also need to modify the logging.properties file to set the desired log level. Both of these files are created for you when you create a new App Engine Java project using Maven. You can find these files at the following locations:

Maven Project Layout

Edit the appengine-web.xml file to add the following inside the <appengine-web-app> tags:

    <system-properties>
       <property name="java.util.logging.config.file" value="WEB-INF/logging.properties"/>
    </system-properties>

The default log level in logging.properties is WARNING, which suppresses INFO messages from the output. To change the log level for all classes in your app, edit the logging.properties file to change the level. For example you could change .level = WARNING to .level = INFO.

In your application code, write log messages using the java.util.logging.Logger API. The example below writes an Info log message:

import java.util.logging.Logger;
//...

public class MyClass {

  private static final Logger log = Logger.getLogger(MyClass.class.getName());
  log.info("Your information log message.");
  //....

When your app runs, App Engine records the messages and makes them available. You can read them programmatically using the Logs API as shown next, or you can browse them in the Logs Viewer. You can also download the log files using the AppCfg tool.

Reading logs via API

The general process of getting logs using the Logs API is as follows:

  1. Use LogQuery to specify which logs to return.
  2. Use LogServiceFactory.getLogService() to create the LogService
  3. Invoke LogServiceFactory.getLogService().fetch() to return an iterator for the request logs.
  4. In each iteration, for each RequestLogs, process the request properties as desired.
  5. Optionally, use RequestLogs.getAppLogLines() to get the list of app logs (AppLogLine) associated with this request.
  6. If you retrieved the app logs list, for each AppLogLine, process the property data as desired.

Sample code

The following sample displays 5 request logs at at time, along with their application logs. It lets you cycle through each set of 5 logs using a Next link.

package com.example.appengine.logs;

import com.google.appengine.api.log.AppLogLine;
import com.google.appengine.api.log.LogQuery;
import com.google.appengine.api.log.LogServiceFactory;
import com.google.appengine.api.log.RequestLogs;

import org.joda.time.DateTime;

import java.io.IOException;
import java.io.PrintWriter;

import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;


// Get request logs along with their app log lines and display them 5 at
// a time, using a Next link to cycle through to the next 5.
public class LogsServlet extends HttpServlet {
  @Override
  public void doGet(HttpServletRequest req, HttpServletResponse resp)
         throws IOException {

    resp.setContentType("text/html");
    PrintWriter writer = resp.getWriter();
    writer.println("<!DOCTYPE html>");
    writer.println("<meta charset=\"utf-8\">");
    writer.println("<title>App Engine Logs Sample</title>");

    // We use this to break out of our iteration loop, limiting record
    // display to 5 request logs at a time.
    int limit = 5;

    // This retrieves the offset from the Next link upon user click.
    String offset = req.getParameter("offset");

    // We want the App logs for each request log
    LogQuery query = LogQuery.Builder.withDefaults();
    query.includeAppLogs(true);

    // Set the offset value retrieved from the Next link click.
    if (offset != null) {
      query.offset(offset);
    }

    // This gets filled from the last request log in the iteration
    String lastOffset = null;
    int count = 0;

    // Display a few properties of each request log.
    for (RequestLogs record : LogServiceFactory.getLogService().fetch(query)) {
      writer.println("<br>REQUEST LOG <br>");
      DateTime reqTime = new DateTime(record.getStartTimeUsec() / 1000);
      writer.println("IP: " + record.getIp() + "<br>");
      writer.println("Method: " + record.getMethod() + "<br>");
      writer.println("Resource " + record.getResource() + "<br>");
      writer.println(String.format("<br>Date: %s", reqTime.toString()));

      lastOffset = record.getOffset();

      // Display all the app logs for each request log.
      for (AppLogLine appLog : record.getAppLogLines()) {
        writer.println("<br>" + "APPLICATION LOG" + "<br>");
        DateTime appTime = new DateTime(appLog.getTimeUsec() / 1000);
        writer.println(String.format("<br>Date: %s", appTime.toString()));
        writer.println("<br>Level: " + appLog.getLogLevel() + "<br>");
        writer.println("Message: " + appLog.getLogMessage() + "<br> <br>");
      }

      if (++count >= limit) {
        break;
      }
    }

    // When the user clicks this link, the offset is processed in the
    // GET handler and used to cycle through to the next 5 request logs.
    writer.println(String.format("<br><a href=\"/?offset=%s\">Next</a>", lastOffset));
  }
}

In the sample, notice that the GET handler expects to be re-invoked by the user clicking on the Next link, and so it extracts the offset param, if present. That offset is used in the subsequent re-invocation of LogServiceFactory.getLogService().fetch() to "page through" each group of 5 request logs. There is nothing special about the number 5; it can be anything you want.

Log URL format in the Google Cloud Platform Console

See the following sample URL for an example of the log URL format in the Cloud Platform Console:

https://console.cloud.google.com/logs?filters=request_id:000000db00ff00ff827e493472570001737e73686966746361727331000168656164000100

Reading logs in the console

To view logs using the Log Viewer:

  1. In the Cloud Platform Console, go to the logs for your project.

  2. Use the desired filter to retrieve the logs you want to see. You can filter by various combinations of time, log level, module, and log filter label or regular expression.

    Notice that labels are regular expressions for filtering the logs by logging fields. Valid labels include the following:

    • day
    • month
    • year
    • hour
    • minute
    • second
    • tzone
    • remotehost
    • identd_user
    • user
    • status
    • bytes
    • referrer
    • useragent
    • method
    • path
    • querystring
    • protocol
    • request_id

    For example, path:/foo.* useragent:.*Chrome.* gets logs for all requests to a path starting with /foo that were issued from a Chrome browser.

A typical App Engine log contains data in the Apache combined log format, along with some special App Engine fields, as shown in the following sample log:

192.0.2.0 - test [27/Jun/2014:09:11:47 -0700] "GET / HTTP/1.1" 200 414
"http://www.example.com/index.html"
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36"
"1-dot-calm-sylph-602.appspot.com" ms=195 cpu_ms=42 cpm_usd=0.000046
loading_request=1 instance=00c61b117cfeb66f973d7df1b7f4ae1f064d app_engine_release=1.9.46

Understanding request log fields

The following table lists the fields in order of occurrence along with a description:

Field Order Field Name Always Present? Description
1 Client address Yes Client IP address. Example: 192.0.2.0
2 RFC1413 identity No RFC1413 identity of the client. This is nearly always the character -
3 User No Present only if the app uses the Users API and the user is logged in. This value is the "nickname" portion of the Google Account, for example, if the Google Account is test@example.com, the nickname that is logged in this field is test.
4 Timestamp Yes Request timestamp. Example: [27/Jun/2014:09:11:47 -0700]
5 Request querystring Yes First line of the request, containing method, path, and HTTP version. Example: GET / HTTP/1.1
6 HTTP Status Code Yes Returned HTTP status code. Example: 200
7 Response size Yes Response size in bytes. Example: 414
8 Referrer path No If there is no referrer, the log contains no path, but only -. Example referrer path: "http://www.example.com/index.html".
9 User-agent Yes Identifies the browser and operating system to the web server. Example: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
10 Hostname Yes The hostname used by the client to connect to the App Engine application. Example : (1-dot-calm-sylph-602.appspot.com)
11 Wallclock time Yes Total clock time in milliseconds spent by App Engine on the request. This time dur