Migrating to Python 2.7

This page covers the measures necessary to upgrade your Python application to the Python 2.7 runtime. After following this walkthrough, your application can take advantage of the Python 2.7 runtime's many new capabilities, including multithreading, the Jinja2 templating engine, bytecode access and uploading, and several new included third-party libraries.

Prerequisites and Considerations

In order to use Python 2.7, an application must meet the following requirements:

  • If the application uses Django, it must use version 1.2 or later. See the Django documentation for upgrade details.
  • If you wish to use concurrent requests, the application must use Web Server Gateway Interface (WSGI) script handlers, as covered in Using WSGI.

In addition to meeting these general prerequisites, you must use specific versions of some App Engine features and third-party libraries. Be sure to update the versions that you include and import into your application, and test the application extensively after upgrading. The following list identifies the primary compatibility issues and points to further resources to resolve them:

  • Performance: Application performance may change after upgrading to Python 2.7. If you experience increased response latency, you can increase the frontend instance class and enable concurrent requests. Concurrent requests allow your application to perform faster with a lower instance cost on larger instances. For more information, see Request Handling in Instances.
  • Django: You must use Django 1.2 or later with Python 2.7. For information on upgrading, see the Django Release Notes.
  • PyCrypto: Crypto.Util.randpool has been deprecated in favor of Crypto.Random. For more information, see What to do about RandomPool.
  • webapp: webapp templates are deprecated in Python 2.7. Instead, you can use Django templates directly, jinja2, or a different template engine of your choice.
  • WebOb: Python 2.7 supports WebOb version 1.1. This version is not fully backward compatible with the old version (0.9). If the application uses WebOb, you need to test it extensively to catch any errors resulting from the upgrade.
  • zipimport: Python 2.7 does not support zipimport, but Python 2.7 can natively import from .zip files.
  • simplejson: Python 2.7 does not support simplejson, but Python 2.7 includes the equivalent, and much faster, standard library json module.

Concurrent Requests and WSGI

The Python 2.7 feature that most affects your application's design and performance is its support for multithreaded applications that can handle concurrent requests. The ability to handle concurrent requests results in improved utilization that can significantly enhance your application's performance, especially for applications utilizing higher instance classes that exploit multiple CPU cores.

In order to enable multithreading, applications must move from the Common Gateway Interface (CGI)-based approach of the previous Python runtimes to a Web Server Gateway Interface (WSGI)-based approach. This is because CGI scripts, which were designed to handle requests serially, rely on environment variables for access to the input and output streams.

While the WSGI model for handling requests gives applications more direct access to the input and output streams (enabling concurring requests), handling multiple requests in parallel can cause race conditions when a request handler's logic relies on or interacts with data with a greater-than-local scope, such as application state. For this reason, it is important to code defensively to deal with race conditions in order to ensure your new WSGI application is threadsafe.

See Making Your Application Threadsafe for more details.

Updating app.yaml

Python 2.7 requires a special runtime configuration element in the header of app.yaml. Note that the threadsafe: [ true | false ] element is required for Python 2.7 applications. If true, App Engine sends requests concurrently; if false, App Engine sends them serially. The following app.yaml header enables concurrent requests: 

application: myapp
version: 1
runtime: python27
api_version: 1
threadsafe: true
...

Using WSGI

The Python 2.7 runtime optionally allows you to directly run a Web Server Gateway Interface (WSGI) application, instead of using the run_wsgi_app adapter to run the program as a CGI script. To do this, replace the CGI handler (e.g. myapp.py) in app.yaml with a WSGI application name (e.g. myapp.app).

...
handlers:
- url: /.*
  script: myapp.app
...

You also need to move your WSGI application object to the global scope:

import webapp2

class MainPage(webapp2.RequestHandler):
  def get(self):
    self.response.headers['Content-Type'] = 'text/plain'
    self.response.out.write('Hello, WebApp World!')

app = webapp2.WSGIApplication([('/', MainPage)])

""" Old code:
def main():
  run_wsgi_app(app)

if __name__ == '__main__':
  main()
"""

You can still specify CGI script handlers in app.yaml; however, requests handled by CGI scripts are processed serially, not concurrently. Also, you can not have an app.yaml file that mixes CGI scripts and WSGI applications, and you can not set threadsafe to true if you define any CGI handlers.

Some conventions from previous Python runtimes, such as the use of main() and the check for __name__ == 'main', are now deprecated. These measures helped the CGI scripts of the past stay cached, but now that you're executing WSGI applications directly, these steps are no longer necessary.

Using the App Root Directory

In Python 2.5, CGI scripts ran with the current working directory set to the directory that contained the script. This has changed in Python 2.7. With WSGI, the current working directory at the beginning of the request handler's lifetime is the application root directory.

Review your application code, and make sure that all handlers are written to expect the current working directory to be the application root.

Configuring Libraries

The Python 2.7 runtime includes some third-party modules. Some of these are available by default; others are only available if configured. You can specify which version you want to use. 

libraries:
- name: PIL
  version: "1.1.7"
- name: webob
  version: "1.1.1"

You can specify that the application should use the latest version of the module. This is useful if you're developing an application that doesn't have users yet: you don't need to track new versions. But if your application is being actively used, beware: you might be surprised that your application starts using a new not-backward-compatible library version. To use the latest version: 

libraries:
- name: PIL
  version: latest

For a list of supported libraries, please see Third-party Libraries.

Making Your Application Threadsafe

Handling concurrent requests is simple if each handler only interacts with variables within its scope. But it gets tricky quickly if one handler modifies resources as another is reading them. Making sure that your application behaves as expected—even though multiple requests might be manipulating the same data and interfering with one another—is known as making your application "threadsafe."

The primary rule when designing an application to be threadsafe is to limit use of shared resources (such as state information or global variables) as often as possible. However, it's usually not possible to completely rule out their use, and that's where synchronization mechanisms like lock objects come in.

In Python 2.7, you have access to Python's threading library, which allows you to declare a lock on a block of logic that forces the code inside to be executed serially instead of concurrently. Consider the following code:

class Configuration(ndb.Model):
  some_config_data = ndb.StringProperty()
  _config_cache = None
  _config_lock = threading.Lock()
  @classmethod
  def get_config(cls):
    with cls._config_lock:
      if not cls._config_cache:
        cls._config_cache = cls.get_by_id('config')
    return cls._config_cache

This code shows the creation of a cache of some global configuration variables into a variable called _config_cache. Here, the use of a lock object named _config_lock ensures that the check for a pre-existing _config_cache behaves reliably. Otherwise, this variable could waste time making multiple trips to the Datastore to set the same variable multiple times with the same data, because the competing requests all found that _config_cache was empty.

Do not choose lightly to use locks. A lock forces any other threads executing this method to block. This can be a bottleneck to performance.

Updating Your Application to webapp2

Note: if you are not using webapp as your request handler, you can skip this section.

The web framework included with the Python 2.7 runtime has been upgraded from webapp to webapp2. Among other things, webapp2 adds improved URI routing and exception handling, a full-featured response object, and a more flexible dispatching mechanism.

webapp templates are now deprecated. In their place, you can use Jinja2, Django, or a templating system of your choice (as long as it's written in pure Python).

In App Engine, webapp2 has been aliased to webapp, and webapp2 is backward compatible. However, after upgrading, you still need to thoroughly test your application and familiarize yourself with the new syntax and capabilities of webapp2, rather than continue to rely on backwards compatibility.

And your upgrade is complete!

Once you've uploaded the application, you need to test it extensively to ensure backward compatibility. If you are experiencing issues, check out the forums.