The Python 3 runtime in the App Engine standard runtime doesn't include bundled services such as Memcache and Task Queues. Instead, Google Cloud provides standalone products that are equivalent to most of the bundled services in the Python 2 runtime. For the bundled services that are not available as separate products in Google Cloud, such as image processing, search, and messaging, this guide recommends third-party providers or other workarounds.
This page introduces the migration path for each bundled service. We will provide detailed migration guides for some of these services over the next few months.
Understanding Google Cloud permissions
Since your migrated app and the Google Cloud services that it uses are no longer running in the same "sandboxed" environment, your app needs authorization to access each service. For example, to interact with Datastore or Cloud Tasks, your app needs to supply the credentials of an account that is authorized to access those services.
By default apps in the App Engine standard runtime supply the credentials of the App Engine default service account, which is authorized to access databases in the same project as the app.
If any of the following conditions are true, you need to use an alternative authentication technique that explicitly provides credentials:
Your app and the Memorystore database are in different Google Cloud projects.
You have changed the roles assigned to the default App Engine service account.
For information about alternative authentication techniques, see Setting up Authentication for Server to Server Production Applications.
Authentication for local development
To develop or test your app locally, we recommend creating and using a service account. Do not use the App Engine default service account, since it has a high level of permissions for everything in your project. Instead, create and use a service account with the lowest level of permissions you need for your specific development and testing task.
For instructions on setting up a service account and connecting it to your app, see Obtaining and providing service account credentials manually.
Installing client libraries
The easiest way to use Google Cloud services from a Python app is to install the service's Python client library. (Google Cloud services also provide JSON REST and other interfaces.) The steps for installing the client libraries into the App Engine runtime are different for Python 2 apps and Python 3 apps.
Installing libraries for Python 2 apps
To install a library for your app to use when it's running in the Python 2 runtime:
Create a directory to store your third-party libraries, such as
requirements.txtfile in the same folder as your
app.yamlfile and add the name of a client library along with the
Python 2 apps require
googleapis_common_protosto access Google Cloud services.
For example, the following file can be used to install libraries for
googleapis_common_protos, Pub/Sub and Cloud Tasks:
googleapis_common_protos google-cloud-pubsub google-cloud-tasks
pip installto install the libraries into the folder that you created. For example:
pip install -t lib -r requirements.txt
In your app's
app.yamlfile, specify the Google RPC and SSL libraries in the
libraries: - name: grpcio version: 1.0.0 - name: ssl version: latest
Some client libraries don't need the SSL library. If you don't include the SSL library for a client library that needs it, you'll see an SSL error in the Logs Explorer when you app receives a request.
appengine_config.pyfile in the same folder as your
app.yamlfile if you do not already have one. Add the following to your
# appengine_config.py import pkg_resources from google.appengine.ext import vendor # Set path to your libraries folder. path = 'lib' # Add libraries installed in the path folder. vendor.add(path) # Add libraries to pkg_resources working set to find the distribution. pkg_resources.working_set.add_entry(path)
Be sure to use the
pkg_resourcesmodule, which ensures that your app uses the right distribution of the client libraries.
appengine_config.pyfile in the preceding example assumes that the the
libfolder is located in the current working directory. If you can't guarantee that
libwill always be in the current working directory, specify the full path to the
libfolder. For example:
import os path = os.path.join(os.path.dirname(os.path.realpath(__file__)), 'lib')
App Engine uploads all of the libraries in the directory you specified
appengine_config.py file to the Python 2 runtime.
Installing libraries for Python 3 apps
To install a library for your app to use when it's running in the Python 3 runtime:
Add the library name your app's
requirements.txtfile. For example:
App Engine automatically uploads all libraries listed in the
requirements.txt file to the Python 3 runtime.
Migration paths for App Engine bundled services
DatastoreIf your Python 2 app uses NDB to interact with Datastore, migrate to the Cloud NDB Python 3 library. New Python 3 apps should use the Datastore mode client library.
For details, see:
You can serve images from Cloud Storage, serve them directly, or use a third-party content delivery network (CDN).
To resize, convert, and manipulate images, use an image processing library such as Pillow or a Python interface for ImageMagick. To use one of these third-party libraries, add the library as a dependency and update your code to call the library's APIs.
The App Engine Images service also provided functionality to avoid dynamic requests to your application by handling image resizing using a serving URL. If you want similar functionality, you can generate the re-sized images ahead of time and upload them to Cloud Storage for serving. Alternatively, you could use a third-party content delivery network (CDN) service that offers image resizing.
We recommend that you update your app to use Cloud Logging, which supports features such as viewing logs in the Logs Explorer, downloading logs, filtering messages by severity, and correlating app messages with specific requests. As an alternative, you can enable these features by writing log messages that contain specific data structured in a JSON object.For more information, see Migrating to Cloud Logging.
MemcacheIf your Python 2 app explicitly uses Memcache, you need to migrate to another caching solution before you can run your app in a Python 3 environment. We recommend using Memorystore for Redis as your caching service. For details, see Migrating Memcache to Memorystore.
To obtain information and modify your application's running services, use a combination of environment variables and the App Engine Admin API:
|Service information||How to access|
|Current application ID||
|Current project ID||
|Current service name||
|Current service version||
|Current instance ID||
|Default hostname||Admin API
|List of services||Admin API
|List of versions for a service||Admin API
|Default version for a service, including any traffic splits||Admin API
|List of running instances for a version||Admin API
The Namespaces API enabled multitenant apps to partition data across tenants simply by specifying a unique namespace string for each tenant.
While Datastore supports multitenancy directly, other Google Cloud services do not. If your multitenant app uses other Google Cloud services, you will need to handle multitenancy manually. To have completely isolated instances of services, you can create new projects programmatically using the Cloud Resource Manager API and access resources across projects.
Host any full-text search database such as Elasticsearch on Compute Engine and access it from your service.
Task queueMigrate pull queues to Pub/Sub and migrate push queues to Cloud Tasks. If your app uses both pull and push queues, migrate pull queues first to prevent unexpected behavior with Cloud Tasks.
For details on performing these migrations, see:
By default, the Python 2.7 runtime uses the URL Fetch service to handle outbound HTTP(S) requests, even if you use standard Python libraries to issue those requests. The Python 3 runtime doesn't need an intermediary service to handle outbound requests, so URL Fetch is not available in the Python 3 runtime.
If your app uses standard Python libraries to issue requests, the absence of URL Fetch in the Python 3 runtime should not affect your app's ability to make requests. However, we recommend that you test your app in an environment that doesn't use URL Fetch to make sure.
If your app uses URL Fetch APIs directly, for example to make asynchronous
requests, you need to migrate those requests to use a standard Python library
For more information, see Migrating Outbound Requests.
For an alternative to the Users API, use any HTTP-based authentication mechanism such as:
- OAuth 2.0 and OpenID Connect which provide federated identity from the provider of your choice. Google is an OpenID Connect identity provider. There are also several other providers available.
- Firebase Authentication, which provides authentication using username/password and federated identity using Google, Facebook, Twitter, and more.
- Google Identity Platform, which provides many options for authentication and authorization of Google user accounts.
- Auth0, which provides authentication with various identity providers and single sign-on features.