Note: Python 2.7 has reached end of support on January 31, 2024. Your existing Python 2.7 applications will continue to run and receive traffic. However, App Engine might block re-deployment of applications that use runtimes after their end of support date. We recommend that you migrate to the latest supported version of Python.

google.appengine.ext.appstats.recording module

Summary

Userland RPC instrumentation for App Engine.

google.appengine.ext.appstats.recording.AppStatsDjangoMiddleware: alias of AppstatsDjangoMiddleware

class google.appengine.ext.appstats.recording.AppstatsDjangoMiddlewaresource

Bases: object

Django Middleware to install the instrumentation.

To start recording your app’s RPC statistics, add

‘google.appengine.ext.appstats.recording.AppstatsDjangoMiddleware’,

to the MIDDLEWARE_CLASSES entry in your Django settings.py file. It’s best to insert it in front of any other middleware classes, since some other middleware may make RPC calls and those won’t be recorded if that middleware is invoked before this middleware.

See http://docs.djangoproject.com/en/dev/topics/http/middleware/.

process_request(request)source: Called by Django before deciding which view to execute.

process_response(request, response)source: Called by Django just before returning a response.

class google.appengine.ext.appstats.recording.ConfigDefaultssource

Bases: object

Configurable constants.

To override appstats configuration valuess, define values like this in your appengine_config.py file (in the root of your app):

appstats_MAX_STACK = 5 appstats_MAX_LOCALS = 0

More complete documentation for all configurable constants can be found in the file sample_appengine_config.py.

CALC_RPC_COSTS = False

CHANNEL_CREATE_COST = 10000

DATASTORE_DETAILS = False

DATASTORE_READ_OP_COST = 70

DATASTORE_SMALL_OP_COST = 10

DATASTORE_WRITE_OP_COST = 100

DEBUG = False

DEFAULT_SCRIPT = "print 'Hello, world.'"

DUMP_LEVEL = -1

FULL_SUFFIX = ':full'

KEY_DISTANCE = 100

KEY_MODULUS = 1000

KEY_NAMESPACE = '__appstats__'

KEY_PREFIX = '__appstats__'

KEY_TEMPLATE = ':%06d'

LOCK_SUFFIX = '<lock>'

LOCK_TIMEOUT = 1

MAIL_RECIPIENT_COST = 1000

MAX_DEPTH = 10

MAX_LOCALS = 10

MAX_REPR = 100

MAX_STACK = 10

PART_SUFFIX = ':part'

RECORD_FRACTION = 1.0

RE_STACK_BOTTOM = 'dev_appserver\\.py'

RE_STACK_SKIP = 'recording\\.py|apiproxy_stub_map\\.py'

SHELL_OK = False

TZOFFSET = 28800

XMPP_STANZA_COST = 100

extract_key(request)source

Extract a canonical key from a StatsProto instance.

This default implementation calls config.normalize_path() on the path returned by request.http_path(), and then prepends the HTTP method and a space, unless the method is ‘GET’, in which case the method and the space are omitted (so as to display a more compact key in the user interface).

Parameters

request – a StatsProto instance.

Returns

A string, typically something like ‘/foo/bar/X’ or ‘POST /foo/bar’.

normalize_path(path)source

Transform a path to a canonical key for that path.

Parameters

path – A string, e.g. ‘/foo/bar/12345’.

Returns

A string derived from path, e.g. ‘/foo/bar/X’.

should_record(env)source

Return a bool indicating whether we should record this request.

Parameters

env – The CGI or WSGI environment dict.

Returns

True if this request should be recorded, False if not.

The default implementation returns True if the request matches FILTER_LIST (see above) and random.random() < RECORD_FRACTION.

stats_url = '/_ah/stats'

class google.appengine.ext.appstats.recording.Recorder(env)source

Bases: object

In-memory state for the current request.

An instance is created soon after the request is received, and set as the Recorder for the current request in the RequestLocalRecorderProxy in the global variable ‘recorder_proxy’. It collects information about the request and about individual RPCs made during the request, until just before the response is sent out, when the recorded information is saved to memcache by calling the save() method.

add_full_info_to_proto(proto)source: Update a protobuf representing with additional data.

dump(level=None)source

Log the recorded data, for debugging.

This logs messages using logging.info(). The amount of data logged is controlled by the level argument, which defaults to config.DUMP_LEVEL; if < 0 (the default) nothing is logged.

get_both_protos_encoded()source

Return a string representing all recorded info an encoded protobuf.

This constructs the full proto and calls its .Encode() method; if the resulting string is too large, it tries a number of increasingly aggressive strategies for chopping the data down.

get_call_stack(trace)source

Extract the current call stack.

The stack is limited to at most config.MAX_STACK frames; frames recognized by config.RE_STACK_SKIP are skipped; a frame recognized by config.RE_STACK_BOTTOM terminates the stack search.

Parameters

trace – An IndividualRpcStatsProto instance that will be updated.

get_frame_summary(frame, trace)source

Return a frame summary.

Parameters

frame – A Python stack frame object.
trace – An IndividualRpcStatsProto instance that will be updated.

Returns

False if this stack frame matches config.RE_STACK_BOTTOM. True otherwise.

get_full_proto()source: Return the full protobuf, wrapped in a StatsProto.

get_rpcstats()source

Compute RPC statistics (how often each RPC endpoint is called).

Returns

A dict mapping ‘service.call’ keys to an array of objects giving call counts (int), call costs (int), and billed ops (dict from op to pb).

get_summary_proto()source: Return a protobuf representing a summary of this recorder.

get_summary_proto_encoded()source

Return a string representing a summary an encoded protobuf.

This calls self.get_summary_proto() and calls the .Encode() method of the resulting object.

get_total_api_mcycles()source

Compute the total amount of API time for all RPCs.

Deprecated. This value is no longer meaningful.

Returns

An integer expressing megacycles.

http_method()source: Return the request method, e.g. ‘GET’ or ‘POST’.

http_path()source: Return the request path, e.g. ‘/’ or ‘/foo/bar’, excluding the query.

http_query()source

Return the query string, if any, with ‘?’ prefix.

If there is no query string, an empty string is returned (i.e. not ‘?’).

classmethod init_sys_path_entries()source

Initialize the class variable path_entries.

The variable will hold a list of (i, entry) tuples where entry == sys.path[i], sorted from shortest to longest entry.

record_allocate_ids_details(trace)source

Records cost information for the AllocateIds RPC.

Parameters

trace – IndividualStatsProto where information must be recorded.

record_channel_details(call, trace)source

Records information relating to channel RPCs.

Parameters

call – The call name, e.g. ‘CreateChannel’.
trace – IndividualStatsProto where information must be recorded.

record_commit_details(response, trace)source

Records cost information for the Commit RPC.

Parameters

response – The response protocol message of the Commit RPC call.
trace – IndividualStatsProto where information must be recorded.

record_custom_event(label, data=None, start=None, end=None)source

Record a custom event.

Parameters

label – A string to use as event label; a ‘custom.’ prefix will be added.
data – Optional value to record. This can be anything; the value will be formatted using format_value() before it is recorded.
start – time.time() of the start of the time range, default now. Supply start when you want a range intead of a 0ms duration event.
end – time.time() of the end of the time range, default now. Supply end when the range was in the past or when you want the range to match other monitoring/analytics. Supplying end without start is an error.

record_datastore_details(call, request, response, trace)source

Records additional information relating to datastore RPCs.

Parses requests and responses of datastore related RPCs, and records the primary keys of entities that are put into the datastore or fetched from the datastore. Non-datastore RPCs are ignored. Keys are recorded in the form of Reference protos. Currently the information is logged for the following calls: Get, Put, RunQuery and Next. The code may be extended in the future to cover more RPC calls. In addition to the entity keys, useful information specific to each call is recorded. E.g., for queries, the entity kind and cursor information is recorded; For gets, a flag indicating if the requested entity key is present or not is recorded.

Also collects RPC costs.

Parameters

call – The call name, e.g. ‘Get’.
request – The request protocol message corresponding to the call.
response – The response protocol message corresponding to the call.
trace – IndividualStatsProto where information must be recorded.

record_delete_details(response, trace)source

Records cost information for the Delete RPC.

Parameters

response – The response protocol message of the Delete RPC call.
trace – IndividualStatsProto where information must be recorded.

record_get_details(request, response, trace)source

Records additional get details based on config options.

Details include: Keys of entities requested, whether or not the requested key was successfully fetched, and cost information for the Get RPC.

Parameters

request – The request protocol message of the Get RPC call.
response – The response protocol message of the Get RPC call.
trace – IndividualStatsProto where information must be recorded.

record_http_status(status)source: Record the HTTP status code and the end time of the HTTP request.

record_mail_details(call, request, trace)source

Records information relating to mail RPCs.

Parameters

call – The call name, e.g. ‘Send’.
request – The request protocol message corresponding to the call.
trace – IndividualStatsProto where information must be recorded.

record_put_details(response, trace)source

Records additional put details based on config options.

Details include: Keys of entities written and cost information for the Put RPC.

Parameters

response – The response protocol message of the Put RPC call.
trace – IndividualStatsProto where information must be recorded.

record_query_details(call, request, response, trace)source

Records additional query details based on config options.

Details include: Keys of entities fetched by a datastore query and cost information.

Information is recorded for both the RunQuery and Next calls. For RunQuery calls, we record the entity kind and ancestor (if applicable) and cursor information (which can help correlate the RunQuery with a subsequent Next call). For Next calls, we record cursor information of the Request (which helps associate this call with the previous RunQuery/Next call), and the Response (which helps associate this call with the subsequent Next call). For key only queries, entity keys are not recorded since entities are not actually fetched. In the future, we might want to record the entities but also record a flag indicating whether this is a key only query.

Parameters

call – The call name, e.g. ‘RunQuery’ or ‘Next’
request – The request protocol message of the RPC call.
response – The response protocol message of the RPC call.
trace – IndividualStatsProto where information must be recorded.

record_rpc_request(service, call, request, response, rpc)source

Record the request of an RPC call.

Parameters

service – The service name, e.g. ‘memcache’.
call – The call name, e.g. ‘Get’.
request – The request object.
response – The response object (ignored).
rpc – The RPC object; may be None.

record_rpc_response(service, call, request, response, rpc)source

Record the response of an RPC call.

Parameters

service – The service name, e.g. ‘memcache’.
call – The call name, e.g. ‘Get’.
request – The request object.
response – The response object (ignored).
rpc – The RPC object; may be None.

This first tries to match the request with an unmatched request trace. If no matching request trace is found, this is logged as a new trace.

record_xmpp_details(call, request, trace)source

Records information relating to xmpp RPCs.

Parameters

call – The call name, e.g. ‘SendMessage’.
request – The request protocol message corresponding to the call.
trace – IndividualStatsProto where information must be recorded.

save()source

Save the recorded data to memcache and log some info.

This wraps the _save() method, which does the actual work; this function just logs the total time it took and some other statistics.

sys_path_entries = None

class google.appengine.ext.appstats.recording.RequestLocalRecorderProxysource

Bases: object

A Recorder proxy that dispatches to a Recorder for the current request.

clear_for_current_request(*args)source

get_for_current_request(*args)source

has_recorder_for_current_request(*args)source

set_for_current_request(*args)source

class google.appengine.ext.appstats.recording.StatsProto(proto=None)source

Bases: object

A wrapper for RequestStatProto with a number of extra attributes.

This exists mainly so that ui.py can pass an instance of this class directly to a Django template, and give the Django template access to formatted times and megacycles converted to milliseconds without using custom tags. (Though arguably the latter would be more convenient for the Java version of Appstats.)

This adds the following methods:

.start_time_formatted(): .start_time_milliseconds() nicely formatted.
.processor_milliseconds(): .processor_mcycles() converted to milliseconds.
.combined_rpc_count(): total number of RPCs, computed from

.rpc_stats_list(). (This is cached as .__combined_rpc_count.)
.combined_rpc_cost(): total cost of RPCs, computed from

.rpc_stats_list(). (This is cached as .__combined_rpc_cost.)
.combined_rpc_billed_ops(): total billed ops for RPCs, computed from

.rpc_stats_list(). (This is cached as .__combined_rpc_billed_ops.)

All these are methods to remain close in style to the protobuffer access methods.

api_milliseconds()source

Return an int giving .api_mcycles() converted to milliseconds.

Deprecated. This value is no longer meaningful.

Returns

An integer expressing milliseconds.

combined_rpc_billed_ops()source: Return the total billed ops for RPCs across .rpc_stats_list().

combined_rpc_cost_micropennies()source: Return the total cost of RPCs across .rpc_stats_list().

combined_rpc_count()source: Return the total number of RPCs across .rpc_stats_list().

processor_mcycles()source

processor_milliseconds()source: Return an int giving .processor_mcycles() converted to milliseconds.

start_time_formatted()source: Return a string representing .start_timestamp_milliseconds().

google.appengine.ext.appstats.recording.appstats_wsgi_middleware(app)source

WSGI Middleware to install the instrumentation.

Normally you specify this middleware in your appengine_config.py file, like this:

def webapp_add_wsgi_middleware(app):: from google.appengine.ext.appstats import recording app = recording.appstats_wsgi_middleware(app) return app

See Python PEP 333, http://www.python.org/dev/peps/pep-0333/ for more information about the WSGI standard.

google.appengine.ext.appstats.recording.billed_ops_to_str(billed_ops_list)source: Formats a list of BilledOpProtos for display in the appstats UI.

google.appengine.ext.appstats.recording.dont_record()source: API to prevent recording of the current request. Used by ui.py.

google.appengine.ext.appstats.recording.end_recording(status, firepython_set_extension_data=None)source

Stop recording RPC traces and save all traces to memcache.

This clears the recorder set for this request in ‘recorder_proxy’.

Parameters

status – HTTP Status, a 3-digit integer.

google.appengine.ext.appstats.recording.format_time(timestamp)source

Utility to format a timestamp in UTC.

Parameters

timestamp – A float representing a standard Python time (see make_key()).

google.appengine.ext.appstats.recording.format_value(val)source

Format an arbitrary value as a compact string.

This wraps formatting._format_value() passing it our config variables.

google.appengine.ext.appstats.recording.individual_billed_ops_to_str(self)source

Formats a list of BilledOpProtos for display in the appstats UI.

We attach this method to IndividualRpcStatsProto, which keeps the django-templates we use to render the appstats UI simpler and multi-language friendly.

Parameters

self – the linter is harrassing me, what am I supposed to put here?

Returns

A display-friendly string representation of a list of BilledOpsProtos

google.appengine.ext.appstats.recording.load_full_proto(timestamp, java_application=False)source

Load the full record for a given timestamp.

Parameters

timestamp – The start_timestamp of the record, as a float in seconds (see make_key() for details).
java_application – Boolean. If true, this function is being invoked by the download_appstats tool on a java application.

Returns

A StatsProto instance if the record exists and can be loaded; None otherwise.

google.appengine.ext.appstats.recording.load_summary_protos(java_application=False)source

Load all valid summary records from memcache.

Parameters

java_application – Boolean. If true, this function is being invoked by the download_appstats tool on a java application.

Returns

A list of StatsProto instances, in reverse chronological order (i.e. most recent first).

NOTE: This is limited to returning at most config.KEY_MODULUS records, since there are only that many distinct keys. See also make_key().

google.appengine.ext.appstats.recording.lock_key()source: Return the key name to use for the memcache lock.

google.appengine.ext.appstats.recording.make_key(timestamp)source

Return the key (less suffix) to which a timestamp maps.

Parameters

timestamp – A timestamp, expressed using the standard Python convention for timestamps (a float giving seconds and fractional seconds since the POSIX timestamp epoch).

Returns

A string, formed by concatenating config.KEY_PREFIX and config.KEY_TEMPLATE with some of the lower digits of the timestamp converted to milliseconds substituted in the template (which should contain exactly one %-format like ‘%d’).

google.appengine.ext.appstats.recording.mcycles_to_msecs(mcycles)source: Helper function to convert megacycles to milliseconds.

google.appengine.ext.appstats.recording.mcycles_to_seconds(mcycles)source: Helper function to convert megacycles to seconds.

google.appengine.ext.appstats.recording.post_call_hook(service, call, request, response, rpc=None, error=None)source

Post-Call hook function for apiproxy_stub_map.

The signature is determined by the CallHooks protocol. In certain cases, rpc and/or error may be omitted.

Once registered, this fuction will be called right after any kind of RPC call made through apiproxy_stub_map returns. The call is passed on to the record_rpc_request() method of the global ‘recorder_proxy’ variable, unless the latter does not have a Recorder set for this request.

google.appengine.ext.appstats.recording.pre_call_hook(service, call, request, response, rpc=None)source

Pre-Call hook function for apiprixy_stub_map.

The signature is determined by the CallHooks protocol. In certain cases, rpc may be omitted.

Once registered, this fuction will be called right before any kind of RPC call is made through apiproxy_stub_map. The arguments are passed on to the record_rpc_request() method of the global ‘recorder_proxy’ variable, unless the latter does not have a Recorder set for this request.

google.appengine.ext.appstats.recording.start_recording(env=None)source

Start recording RPC traces.

This creates a Recorder instance and sets it for the current request in the global RequestLocalRecorderProxy ‘recorder_proxy’.

Parameters

env – Optional WSGI environment; defaults to os.environ.

google.appengine.ext.appstats.recording.total_billed_ops_to_str(self)source

Formats a list of BilledOpProtos for display in the appstats UI.

We attach this method to AggregateRpcStatsProto, which keeps the django-templates we use to render the appstats UI simpler and multi-language friendly.

Parameters

self – the linter is harrassing me, what am I supposed to put here?

Returns

A display-friendly string representation of a list of BilledOpsProtos

google.appengine.ext.appstats.recording module

Summary

Contents