- Getting log data
- Log URL formats in App Engine and developer consoles
- How to read a log
- Quotas and limits
- The development server and Logs API
The Logs API provides access to the application and request logs for your application. You can also access the logs for your application in the Logs Viewer provided in the Google Developers Console by clicking Monitoring > Logs in the left navigation panelImportant: Be aware that logs by default do not persist when you use the development app server. For information on how to make logs persist, see The development server and Logs API.
Log categories: request logs and app logs
There are two categories of log data: request logs and application logs. A request log is written for each request handled by your app, and contains information such as the app ID, HTTP version, and so forth. For a complete list of available properties for request logs, see RequestLog.
Each request log contains a list of application logs (AppLog) associated with that request, returned in the
RequestLog.app_logs property. Each app log contains the time the log was written, the log message, and the log level.
Getting log data
The general process of getting logs is as follows:
fetch()to return an iterator for the request logs.
- In each iteration, process each
- Optionally, use
RequestLog.app_logsto get the list of related AppLogs.
- If you retrieved the app logs list, for each
AppLogLine, process the
AppLogproperty data as desired.
The following sample displays 5 request logs at at time, along with their application logs. It lets you cycle through each set of 10 logs using a Next link.
import base64 import datetime import logging import time import urllib import webapp2 from google.appengine.api.logservice import logservice # This sample gets the app request logs up to the current time, displays 5 logs # at a time, including all AppLogs, with a Next link to let the user "page" # through the results, using the RequestLog offset property. class MainPage(webapp2.RequestHandler): def get(self): logging.info('Starting Main handler') # Get the incoming offset param from the Next link to advance through # the logs. (The first time the page is loaded, there won't be any offset.) try: offset = self.request.get('offset') or None if offset: offset = base64.urlsafe_b64decode(str(offset)) except TypeError: offset = None # Set up end time for our query. end_time = time.time() # Count specifies the max number of RequestLogs shown at one time. # Use a boolean to initially turn off visiblity of the "Next" link. count = 5 show_next = False last_offset = None # Iterate through all the RequestLog objects, displaying some fields and # iterate through all AppLogs beloging to each RequestLog count times. # In each iteration, save the offset to last_offset; the last one when # count is reached will be used for the link. i = 0 for req_log in logservice.fetch(end_time=end_time, offset=offset, minimum_log_level=logservice.LOG_LEVEL_INFO, include_app_logs=True): self.response.out.write('<br /> REQUEST LOG <br />') self.response.out.write( 'IP: %s <br /> Method: %s <br /> Resource: %s <br />' % (req_log.ip, req_log.method, req_log.resource)) self.response.out.write( 'Date: %s<br />' % datetime.datetime.fromtimestamp(req_log.end_time).strftime('%D %T UTC')) last_offset= req_log.offset i += 1 for app_log in req_log.app_logs: self.response.out.write('<br />APP LOG<br />') self.response.out.write( 'Date: %s<br />' % datetime.datetime.fromtimestamp(app_log.time).strftime('%D %T UTC')) self.response.out.write('<br />Message: %s<br />' % app_log.message) if i >= count: show_next = True break # Prepare the offset URL parameters, if any. if show_next: query = self.request.GET query['offset'] = base64.urlsafe_b64encode(last_offset) next_link = urllib.urlencode(query) self.response.out.write('<a href="/?%s">Next</a>' % next_link) app = webapp2.WSGIApplication([('/', MainPage)], debug=True)
In the sample, notice that the GET handler expects to be re-invoked by the user clicking on the Next link, and so it extracts the offset param, if present. That offset is used in the subsequent re-invocation of
logservice.fetch() to "page through" each group of 5 request logs. There is nothing special about the number 5; it can be anything you want.
Log URL formats in App Engine and developer consoles
The log URL format is different in the App Engine Admin Console compared to the
Google Developers console. In particular, the Google Developers console does not
filter_type, as can be seen in the following sample log URLs:
App Engine Admin Console format:
Google Developer Console format:
How to read a log
To view logs using the Log Viewer:
Visit the developer console in your browser.
Open the project whose logs you wish to see and select Compute > App Engine > Logs.
Use the desired filter to retrieve the logs you want to see. You can filter by various combinations of time, log level, module, and log filter label or regular expression.
Notice that labels are regular expressions for filtering the logs by logging fields. Valid labels include the following:
path:/foo.* useragent:.*Chrome.*gets logs for all requests to a path starting with
/foothat were issued from a Chrome browser.
A typical App Engine log contains data in the Apache combined log format, along with some special App Engine fields, as shown in the following sample log:
192.0.2.0 - test [27/Jun/2014:09:11:47 -0700] "GET / HTTP/1.1" 200 414 "http://www.example.com/index.html" Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36" "1-dot-calm-sylph-602.appspot.com" ms=195 cpu_ms=42 cpm_usd=0.000046 loading_request=1 instance=00c61b117cfeb66f973d7df1b7f4ae1f064d app_engine_release=1.9.27
The following table lists the fields in order of occurrence along with a description:
|Field Order||Field Name||Always Present?||Description|
|1||Client address||Yes||Client IP address. Example:
|2||RFC1413 identity||No||RFC1413 identity of the client. This is nearly always the character
|3||User||No||Present only if the app uses the Users API and the user is logged in. This value is the "nickname" portion of the Google Account, for example, if the Google Account is
|4||Timestamp||Yes||Request timestamp. Example:
|5||Request querystring||Yes||First line of the request, containing method, path, and HTTP version. Example:
|6||HTTP Status Code||Yes||Returned HTTP status code. Example:
|7||Response size||Yes||Response size in bytes. Example:
|8||Referrer path||No||If there is no referrer, the log contains no path, but only
|9||User-agent||Yes||Identifies the browser and operating system to the web server. Example:
|10||Hostname||Yes||The hostname used by the client to connect to the App Engine application. Example : (
|11||Wallclock time||Yes||Total clock time in milliseconds spent by App Engine on the request. This time duration does not include time spent between the client and the server running the instance of your application. Example:
|12||CPU milliseconds||Yes||CPU milliseconds required to fulfill the request. This is the number of milliseconds spent by the CPU actually executing your application code, expressed in terms of a baseline 1.2 GHz Intel x86 CPU. If the CPU actually used is faster than the baseline, the CPU milliseconds can be larger than the actual clock time defined above. Example:
|13||Exit code||No||Only present if the instance shut down after getting the request. In the format
|14||Estimated cost||Yes||Estimated cost of 1000 requests just like this one, in USD. Example:
|15||Queue name||No||The name of the task queue used. Only present if request used a task queue. Example:
|16||Task name||No||The name of the task executed in the task queue for this request. Only present if the request resulted in the queuing of a task. Example:
|17||Pending queue||No||Only present if a request spent some time in a pending queue. If there are many of these in your logs and/or the values are high, it might be an indication that you need more instances to serve your traffic. Example:
|18||Loading request||No||Only present if the request is a loading request. This means an instance had to be started up. Ideally, your instances should be up and healthy for as long as possible, serving large numbers of requests before being recycled and needing to be started again. Which means you shouldn't see too many of these in your logs. Example:
|19||Instance||Yes||Unique identifier for the instance that handles the request. Example:
|20||Version||Yes||The current App Engine release version used in production App Engine: 1.9.27|
Quotas and limits
Your application is affected by the following logs-related quotas:
- Logs data retrieved via the Logs API.
- Log storage, also called logs retention.
Quota for data retrieved
The first 100 megabytes of logs data retrieved per day via the Logs API calls are free. After this amount is exceeded, no further Logs API calls will succeed unless billing is enabled for your app. If billing is enabled for your app, data in excess of 100 megabytes results in charges of $0.12/GB.
You can control how much log data your application stores by means of its log retention settings in the Admin Console. By default, logs are stored for an application free of charge with the following per-application limits: a maximum of 1 gigabyte for a maximum of up to 90 days. If either limit is exceeded, more recent logs will be shown and older logs will be deleted to stay within the size limit. Logs older than the maximum retention time are also deleted.
If your app has billing enabled, you can pay for higher log size limits by specifying the desired maximum log size in gigabytes in the Admin Console. You can also set the retention time by specifying the desired number of days to keep logs, up to a maximum of 365 days. The cost of this extra log storage is $0.026 per gigabyte utilized per month.
|Limit||Amount||Cost past free threshold|
|Maximum days storage per log||90 days free, 365 days if paid||$0.026 per gigabyte utilized per month|
|Maximum total logs storage||1 gigabyte free, unlimited if paid||$0.026 per gigabyte utilized per month|
The development server and Logs API
By default, logs are stored in memory only in the development server and are accessible if you wish to test the Logs API feature. If you wish to persist logs from the development server to disk at the default location
/tmp/dev_appserver.logs, supply the
--persist_logs command line option as follows:
dev_appserver.py --persist_logs your-app-directory
If you wish to persist the logs from the development server to disk at a location of your own choosing, supply the desired path and filename to the
--logs_path command line option as follows:
dev_appserver.py --logs_path=your-path/your-logfile-name your-app-directory