Hide
App Engine

Using Cloud Logging in App Engine Apps

Deepak Tiwari, Doug Anderson
November 2014

Google Cloud Logging provides facilities for importing, viewing, storing, and retrieving logs from applications running on the Google Cloud Platform.

What is available

Google Cloud Logging includes the following features for App Engine:

  • Request logs and application logs for App Engine applications.
  • Custom log files for applications using Managed VMs.
  • A Logs API (Python, Java, PHP, Go) for accessing logs.

Cloud Logging and Managed VMs apps

Applications using App Engine Managed VMs should write custom log files to the VM's log directory at /var/log/app_engine/custom_logs. These files are automatically collected and made available in the Logs Viewer.

Custom log files must have the suffix .log or .log.json. If the suffix is .log.json, the logs must be in JSON format with one JSON object per line. If the suffix is .log, log entries are treated as plain text.

Using the Logs Viewer

The Logs Viewer provides a web-based UI to navigate, display, and search your logs. With the Logs Viewer you can view and search logs from all your instances and apply filters to narrow in on a specific event, regardless of where it was generated.

To access the Logs Viewer:

  1. Open your project in the Developers Console.

  2. Click Monitoring > Logs.

  3. Ensure App Engine is selected in the dropdown menu.

  4. If your app runs in a Managed VM and writes custom logs, select the desired log name.

Filtering and finding logs

You can choose to view only those logged events that are above a certain severity. Debug events are considered the lowest severity, Critical events are considered the highest. For instance, if you select a minimum severity of 'Warning', all warning, error, and critical log events will be displayed.

When viewing the Request log, you can combine field filters (for example, status:, protocol:, and so forth) and regular expressions in the same query, using the filter text box in the Logs Viewer. In addition, you can add and remove filters to help drill down and then zoom out again until you find what you’re looking for. Simply modify the query and press Enter to refresh the logs.

When you click the search bar, the Logs Viewer shows possible completions for filter fields as you type. For example, typing u would produce the following possible completions:

Filter logs

Filters of the same type are ORed together, while different filter types are ANDed together. So for example, status:400 status:500 regex:quota would produce all requests that returned HTTP status of either 400 OR 500, AND have the word quota in the log.

Field filters aren't yet supported for Managed VM custom logs, but you can use regular expressions to search custom logs.

Search or scroll through all of your logs

When you scroll through your logs in the new Logs Viewer, results are fetched until the console window is full. To retrieve additional logs that match the query, scroll down for newer results or up for older ones.

This provides you with a continuous view of your events to enable you to move forward and backward in time without requiring you to click Next or refresh the console. While results are being fetched you will see a Loading… indicator at the top or bottom of the viewer.

You can also click the refresh icon to load the latest logs, or click the play icon to live stream the logs for all of your logs from all of your instances.

Downloading logs from App Engine

For download information, see the log downloading instructions for Python, Java, or Go.

How to read the log information

The log fields are documented in the Logs API pages for Python, Java, Go, and PHP.

Using Logging in your app

This part of the article describes how to add logging to your App Engine Python SDK app or your Java SDK app. Click the tab for details in the SDK you are interested in.

Python SDK

You may find it useful to read the documentation for the standard Python logging module at Python.org.

The Python logging module allows a developer to log 5 levels of severity:

  • Debug
  • Info
  • Warning
  • Error
  • Critical

The following example shows how to use some of these levels to log events in a simple Guestbook application:

import logging
import os
import datetime
import wsgiref.handlers

from google.appengine.api import users
from google.appengine.ext import db
from google.appengine.ext import webapp
from google.appengine.ext.webapp import template

class Post(db.Model):
    author = db.StringProperty()
    content = db.StringProperty()
    date = db.DateTimeProperty()

class MainPage(webapp.RequestHandler):

  def get(self):
      # Retrieve existing posts from the datastore when getting the Main Page
      try:
          post_query = db.GqlQuery('SELECT * FROM Post ORDER BY date DESC')
          posts = [post for post in post_query]
      except:
          logging.error('There was an error retrieving posts from the datastore')

      template_values = {
          'posts': posts,
      }
      path = os.path.join(os.path.dirname(__file__), 'index.html')
      self.response.out.write(template.render(path, template_values))

class Guestbook(webapp.RequestHandler):
    # The user has posted a comment to the guest book
    def post(self):
        logging.debug('Start guestbook signing request')

        post = Post()

        if users.get_current_user():
            logging.info('Signed by user %s', users.get_current_user().nickname())
            post.author = users.get_current_user().user_id()
        else:
            logging.info('Signed by anonymous user')

        post.content = self.request.get('content')
        post.date = datetime.datetime.now()

        try:
            post.put()
        except:
            logging.error('There was an error saving comment %s', self.request.get('content'))

        logging.debug('Finish guestbook signing')
        self.redirect('/')

application = webapp.WSGIApplication([('/', MainPage),
                                      ('/sign', Guestbook)],
                                     debug=True)

def main():
    # Set the logging level in the main function
    # See the section on <a href="/appengine/docs/python/#Python_App_caching">Requests and App Caching</a> for information on how
    # App Engine reuses your request handlers when you specify a main function
    logging.getLogger().setLevel(logging.DEBUG)
    webapp.util.run_wsgi_app(application)

if __name__ == '__main__':
    main()

In our example, we call debug() to provide information about the requests coming into our handler. We call info() to provide information about the users of our application, and we call error() to tell us where our application is experiencing errors.

Later, we can use this information to filter out only the actions we are interested in. This will help us troubleshoot our code, improve the user experience, or add additional features to our application.

Java SDK

The App Engine Java SDK allows a developer to log 5 levels of severity:

  • DEBUG
  • INFO
  • WARNING
  • ERROR
  • CRITICAL

In order to add logging to your Java app, you'll first need to add the appropriate system properties to your project's appengine-web.xml file, and you may also need to modify the logging.properties file to set the desired log level. Both of these files are created for you when you create a new App Engine Java project using Maven. You can find these files at the following locations:

Maven Project Layout

Edit the appengine-web.xml file to add the following inside the <appengine-web-app> tags:

    <system-properties>
        <property name="java.util.logging.config.file" value="WEB-INF/logging.properties"/>
    </system-properties>

Notice that the default log level in logging.properties is WARNING, which suppresses INFO messages from the output. To change the log level for all classes in your app, edit the logging.properties file to change the level. For example change:

.level = WARNING

to:

.level = INFO

Inside your application code, log messages as desired using the java.util.logging.Logger API. When your app runs, App Engine records the messages and makes them available for browsing in the Admin Console, and available for downloading using the AppCfg tool. The Admin Console lets you browse messages by log level.