App Engine locations
App Engine is regional, which means the infrastructure that runs your apps is located in a specific region and is managed by Google to be redundantly available across all the zones within that region.
Meeting your latency, availability, or durability requirements are primary factors for selecting the region where your apps are run. You can generally select the region nearest to your app's users but you should consider the location of the other Google Cloud products and services that are used by your app. Using services across multiple locations can affect your app's latency as well as pricing.
App Engine is available in the following regions:
northamerica-northeast1
(Montréal)us-central
(Iowa)us-west2
(Los Angeles)us-east1
(South Carolina)us-east4
(Northern Virginia)southamerica-east1
(São Paulo)europe-west
(Belgium)europe-west2
(London)europe-west3
(Frankfurt)europe-west6
(Zürich)asia-northeast1
(Tokyo)asia-northeast2
(Osaka)asia-east2
(Hong Kong)asia-south1
(Mumbai)australia-southeast1
(Sydney)
You cannot change an app's region after you set it.
If you already created an App Engine application, you can view the
region by running the gcloud app describe
command or opening the
App Engine Dashboard in the Cloud Console.
The region of your App Engine application is listed under
http://[YOUR_PROJECT_ID].appspot.com
.
Google Cloud NDB is a client library for Python that replaces App Engine NDB. App Engine NDB enables Python 2 apps to store and query data in Datastore databases. Cloud NDB enables Python 2 and Python 3 apps to store and query data in the same databases, however the product that manages those databases has changed from Datastore to Cloud Firestore in Datastore mode.
We recommend that you migrate to Cloud NDB before you upgrade your app to Python 3, as the App Engine Python 3 runtime does not support App Engine NDB client library. This incremental approach to migration enables you to maintain a functioning and testable app throughout the migration process.
Cloud NDB is intended to replace the features in App Engine NDB, so it will not support new features of Cloud Firestore in Datastore mode. New Python 3 apps should use the Datastore mode client library instead of Cloud NDB.
You can view the source code for Cloud NDB in the GitHub repository.
Key differences between App Engine NDB and Cloud NDB
While Cloud NDB is designed to replace App Engine NDB, there are some key differences between the two libraries:
Cloud NDB doesn't use the App Engine Memcache service to cache data.
Instead, Cloud NDB can cache data in a Redis in-memory data store managed by Memorystore, Redis Labs, or other systems. While only Redis data stores are currently supported, Cloud NDB has generalized and defined caching in the abstract
GlobalCache
interface, which can support additional concrete implementations.Memorystore for Redis doesn't have a free tier. See Memorystore Pricing for details.
App Engine NDB APIs that rely on App Engine Python 2.7 runtime-specific services have either been updated or removed from Cloud NDB.
New features in Python 3 and Django have eliminated the need for
google.appengine.ext.ndb.django_middleware
. Instead, you can easily write your own middleware with just a few lines of code.App Engine NDB required apps and the Datastore database to be in the same Google Cloud project, with App Engine providing credentials automatically. Cloud NDB can access Datastore mode databases in any project, as long as you authenticate your client properly. This is consistent with other Google Cloud APIs and client libraries.
The full list of differences is available in the migration notes for the Cloud NDB GitHub project.
Before you start migrating
Setting up Memorystore for Redis
Cloud NDB can cache data in a Redis in-memory data store managed by Memorystore, Redis Labs, or other systems. This guide describes how to use Memorystore for Redis to cache data.
To set up Memorystore for Redis:
Create a Redis instance in Memorystore.
When prompted to select a region for your Redis instance, select the same region in which your App Engine app is located.
Note the IP address and port number of the Redis instance you create. You will use this information when you enable data caching for Cloud NDB.
Connect your App Engine to a VPC network. Your app can only communicate with Memorystore through a VPC connector.
Be sure to use the
gcloud beta
command to deploy your app updates. Only the beta command can update your app to use a VPC connector.
After you complete these steps, you can enable data caching for Cloud NDB.
Understanding Datastore mode permissions
Every interaction with a Google Cloud service needs to be authorized. For example, to store or query data in a Datastore mode database, your app needs to supply the credentials of an account that is authorized to access the database.
By default your app supplies the credentials of the App Engine default service account, which is authorized to access databases in the same project as your app.
You will need to use an alternative authentication technique that explicitly provides credentials if any of the following conditions are true:
Your app and the Datastore mode database are in different Google Cloud projects.
You have changed the roles assigned to the default App Engine service account.
For information about alternative authentication techniques, see Setting up Authentication for Server to Server Production Applications.
Overview of the migration process
To migrate to Cloud NDB:
-
Install the Cloud NDB client library.
Update import statements to import modules from Cloud NDB.
Add code that creates a Cloud NDB client. The client can read your app's environment variables and use the data to authenticate with Datastore mode.
Add code that uses the client's runtime context to keep caching and transactions separate between threads.
Remove or update code that uses methods and properties that are no longer supported.
Deploy your app to App Engine.
As with any change you make to your app, consider using traffic splitting to slowly ramp up traffic. Monitor the app closely for any database issues before routing more traffic to the updated app.
Updating your Python app
Installing the Cloud NDB library for Python 2 apps
Create a directory to store your third-party libraries, such as
lib/
.Create a
requirements.txt
file in the same folder as yourapp.yaml
file and add the name of a client library along with thegoogleapis_common_protos
library:googleapis_common_protos google-cloud-ndb
Python 2 apps require
googleapis_common_protos
to access Google Cloud services such as Cloud Firestore in Datastore mode.Use
pip
(version 6 or later) with the-t <directory>
flag to install the libraries into the folder you created in the previous step. For example:pip install -t lib -r requirements.txt
Specify the RPC library in the
libraries
section of yourapp.yaml
file:libraries: - name: grpcio version: 1.0.0
Create an
appengine_config.py
file in the same folder as yourapp.yaml
file if you do not already have one. Add the following to yourappengine_config.py
file:# appengine_config.py import pkg_resources from google.appengine.ext import vendor # Set path to your libraries folder. path = 'lib' # Add libraries installed in the path folder. vendor.add(path) # Add libraries to pkg_resources working set to find the distribution. pkg_resources.working_set.add_entry(path)
Be sure to use the
pkg_resources
module, which to ensures that your app uses the right distribution of the client libraries.The
appengine_config.py
file in the preceding example assumes that the thelib
folder is located in the current working directory. If you can't guarantee thatlib
will always be in the current working directory, specify the full path to thelib
folder. For example:import os path = os.path.join(os.path.dirname(os.path.realpath(__file__)), 'lib')
When you deploy your app, App Engine uploads all of the libraries in
the directory you specified in the appengine_config.py
file.
Installing the Cloud NDB library for Python 3 apps
App Engine's Python 3 runtime uses an app's
requirements.txt
file to determine which packages and
versions to install for the apps. To install the Cloud NDB library in
the Python 3 runtime, add the following line to your app's requirements.txt
file:
google-cloud-ndb
App Engine automatically uploads all libraries in the app's
requirements.txt
file when you deploy the
app.
Installing dependencies locally
When developing and testing your app locally, we highly recommend that
you use a virtual environment to isolate your app's dependencies from your).
system packages. This ensures that your app only loads the dependencies that are
declared in your app's requirements.txt
file are available, and synchronizes
their versions between your local and production environments.
In Python 2, you can use virtualenv to create a virtual environment.
In Python 3, venv is the recommended way to create a virtual environment.
Updating import statements
The location of the NDB module has moved to
google.cloud.ndb
. Update your app's import statements as shown
in the following table:
Remove | Replace with |
---|---|
from google.appengine.ext import ndb |
from google.cloud import ndb |
Creating a Cloud NDB Client
As with other client libraries that are based on Google Cloud APIs, the first
step in using Cloud NDB is to create a Client
object. The client contains credentials and other
data needed to connect to Datastore mode. For example:
from google.cloud import ndb
client = ndb.Client()
In the default authorization scenario described previously, the Cloud NDB client contains credentials from App Engine's default service account, which is authorized to interact with Datastore mode. If you aren't working in this default scenario, see Application Default Credentials (ADC) for information on how to provide credentials.
Using the client's runtime context
In addition to providing the credentials needed to interact with
Datastore mode, the Cloud NDB client contains the
context()
method which returns a runtime context.
The runtime context isolates caching and transaction requests from other
concurrent Datastore mode interactions.
All interactions with Datastore mode need to occur within an NDB runtime context. Since creating a model definition does not interact with Datastore mode, you can define your model class before creating a Cloud NDB client and retrieving a runtime context, and then use the runtime context in the request handler to get data from the database.
For example:
Multithreaded apps
The runtime context that the Cloud NDB client returns only applies to a single thread. If your app uses multiple threads for a single request, you need to retrieve a separate runtime context for each thread that will use the Cloud NDB library.
Using a runtime context with WSGI frameworks
If your web app uses a WSGI framework, you can automatically create a new runtime context for every request by creating a middleware object that retrieves the runtime context, and then wrapping the app in the middleware object.
In the following example of using middleware with Flask:
The
middleware
method creates a WSGI middleware object within the runtime context of the NDB client.The Flask app is wrapped in the middleware object.
Flask will the pass each request through the middleware object, which retrieves a new NDB runtime context for each request.
Using a runtime context with Django
The Django middleware provided by the App Engine NDB library
isn't supported by the Cloud NDB library. If you used this middleware
(google.appengine.ext.ndb.django_middleware
) in your app, follow these steps
to update your app:
Use Django's middleware system to create a new runtime context for every request.
In the following example:
The
ndb_django_middleware
method creates a Cloud NDB client.The
middleware
method creates a middleware object within the runtime context of the NDB client.
In the Django settings.py file, update the
MIDDLEWARE
setting so it lists the new middleware you created instead ofgoogle.appengine.ext.ndb.NdbDjangoMiddleware
.
Django will now pass each request through the middleware object you listed in
the MIDDLEWARE
setting, and this object will retrieve a new NDB runtime
context for each request.
Enabling data caching
Cloud NDB can cache data in a Redis in-memory data store managed by Memorystore, Redis Labs, or other systems. This guide describes how to use Memorystore for Redis to cache data.
After you set up Memorystore for Redis, complete the following steps:
Adding the Redis connection URL
You can connect to the Redis cache by adding the
REDIS_CACHE_URL
environment variable to your app's app.yaml
file. The value
of REDIS_CACHE_URL
takes the following form:
redis://IP address for your instance:port
For example, you can add the following lines to your app's app.yaml
file:
env_variables:
REDIS_CACHE_URL: redis://10.0.0.3:6379
Creating and using a Redis cache object
If you've set REDIS_CACHE_URL
as an environment variable, you can create a
RedisCache object with a single line of code, then use the cache by passing
it to Client.context()
when you set up the runtime context:
client = ndb.Client()
global_cache = ndb.RedisCache.from_environment()
with client.context(global_cache=global_cache):
books = Book.query()
for book in books:
print(book.to_dict())
If you don't set REDIS_CACHE_URL
as an environment variable, you'll need to
construct a Redis client and pass the client to the ndb.RedisCache()
constructor. For example:
global_cache = ndb.RedisCache(redis.StrictRedis(host=IP-address, port=redis_port))
Note that you don't need to declare a dependency on the Redis client library, since the Cloud NDB library already depends on the Redis client library.
Refer to the Memorystore sample application for an example of constructing a Redis client.
Updating code for removed or changed NDB APIs
NDB APIs that rely on App Engine-specific APIs and services have either been updated or removed from the Cloud NDB library.
You will need to update your code if it uses any of the following NDB APIs:
google.appengine.ext.ndb.Model
and model propertiesgoogle.appengine.ext.ndb.Key
google.appengine.ext.ndb.query.QueryOptions
andPropertyOrder
google.appengine.ext.ndb.utils
google.appengine.api.namespacemanager
google.appengine.api.ext.ndb.tasklets
google.appengine.api.datastore_errors
Models and Model properties
The following methods from
google.appengine.ext.ndb.Model
are not available in the Cloud NDB library because they use
App Engine-specific APIs that are no longer available.
Removed API | Replacement |
---|---|
Model.get_indexes and Model.get_indexes_async
|
None |
Model._deserialize and Model._serialize
|
None |
Model.make_connection
|
None |
The following table describes the specific
google.appengine.ext.ndb.Model
properties that have changed in the Cloud NDB library:
Property | Change |
---|---|
TextProperty
|
google.cloud.ndb.TextProperty cannot be indexed. If you
try to set google.cloud.ndb.TextProperty.indexed ,
a NotImplementedError will be raised.
|
StringProperty
|
StringProperty is
always indexed. If you try to set
google.cloud.ndb.StringProperty.indexed , a
NotImplementedError will be raised.
|
All properties with name or kind arguments in
the constructor.
|
name or kind must be str data types,
since unicode was replaced by str in Python 3.
|
The classes and methods in the following table are no longer available, because they use App Engine-specific resources that are no longer available.
Removed API | Replacement |
---|---|
google.appengine.ext.ndb.msgprop.MessageProperty and google.appengine.ext.ndb.msgprop.EnumProperty
|
None
If you try to create these objects, a |
from
google.appengine.ext.ndb.model.Property :_db_get_value _db_set_value _db_set_compressed_meaning _db_set_uncompressed_meaning __creation_counter_global
|
None
These methods rely on Datastore mode protocol buffers that have changed. |
Model.make_connection
|
None |
Keys
The following methods from
google.appengine.ext.ndb.Key
are not available in the Cloud NDB library. These methods were used to
pass keys to and from the DB Datastore API, which is no longer supported
(DB was the predecessor of App Engine NDB).
Removed API | Replacement |
---|---|
Key.from_old_key and Key.to_old_key
|
None |
In addition, note the following changes:
App Engine NDB | Cloud NDB |
---|---|
Kinds and string IDs must be less than 500 bytes | Kinds and string IDs must be less than 1500 bytes. |
Key.app() returns the project ID you specified when
you created the key.
|
The value returned by
google.cloud.ndb.Key.app() may differ from the original
ID passed into the constructor. This is because prefixed app IDs like
s~example are legacy identifiers from App Engine.
They have been replaced by equivalent project IDs, such as
example.
|
Queries
Like App Engine NDB, Cloud NDB provides a QueryOptions
class
(google.cloud.ndb.query.QueryOptions
) that enables you to reuse a specific set of
query options instead of redefining them for each query. However, QueryOptions
in Cloud NDB doesn't inherit from
google.appengine.datastore.datastore_rpc.Configuration
and therefore doesn't
support ...datastore_rpc.Configuration
methods.
In addition, google.appengine.datastore.datastore_query.Order
has been
replaced with
google.cloud.ndb.query.PropertyOrder
. Similar to Order
, the PropertyOrder
class
enables you to specify the sort order across multiple queries. The PropertyOrder
constructor is the same as the constructor for Order
. Only the name of the
class has changed.
Removed API | Replacement |
---|---|
from google.appengine.datastore.datastore_rpc.Configuration :deadline(value) on_completion(value) read_policy(value) force_writes(value) max_entity_groups_per_rpc(value) max_allocate_ids_keys(value) max_rpc_bytes(value) max_get_keys(value) max_put_entities(value) max_delete_keys(value)
See the source code for a description of these methods. |
None |
google.appengine.ext.ndb.Order For example: order=Order(-Account.birthday, Account.name)
|
google.cloud.ndb.PropertyOrder For example: google.cloud.ndb.PropertyOrder(-Account.birthday, Account.name)
|
Utils
The ndb.utils
module (google.appengine.ext.ndb.utils
) is no longer
available. Most of the methods in that module were internal to
App Engine NDB, some methods have been discarded due to implementation
differences in the new ndb, while other methods have been made obsolete by new
Python 3 features.
For example, the positional decorator in the old utils
module declared that only
the first n arguments of a function or method may be positional. However, Python
3 can do this using
keyword-only arguments. What used to be written as:
@utils.positional(2)
def function1(arg1, arg2, arg3=None, arg4=None)
pass
Can be written like this in Python 3:
def function1(arg1, arg2, *, arg3=None, arg4=None)
pass
Namespaces
Namespaces enable a multitenant application to use separate silos of data for each tenant while still using the same Datastore mode database. That is, each tenant stores data under its own namespace.
Instead of using the App Engine-specific
google.appengine.api.namespacemanager
,
you specify a default namespace when you create a Cloud NDB client and then use
the default namespace by calling Cloud NDB methods within the client's runtime
context. This follows the same pattern as other Google Cloud APIs that support
namespaces.
Removed API | Replacement |
---|---|
google.appengine.api.namespacemanager.setnamespace(str)
and google.appengine.api.namespacemanager.getnamespace()
|
client=google.cloud.ndb.Client(namespace="my namespace") with client.context() as context: key = ndb.Key("SomeKind", "SomeId")or key-non-default-namespace=ndb.Key("SomeKind," "AnotherId", namespace="non-default-nspace") |
All other google.appengine.api.namespacemanager methods |
None |
Tasklets
Tasklets can now use a standard return
statement to
return a result instead of raising a Return
exception. For example:
App Engine NDB library | Cloud NDB library |
---|---|
@ndb.tasklet def get_cart(): cart = yield CartItem.query().fetch_async() raise Return(cart) |
@ndb.tasklet def get_cart(): cart = yield CartItem.query().fetch_async() return cart |
Note that you can still return results in Cloud NDB by raising a Return
exception, but it's not recommended.
In addition, the following Tasklets
methods and subclasses are no longer
available, mainly because of changes in how an NDB context is created and used
in Cloud NDB library.
Removed API | Replacement |
---|---|
from
google.appengine.api.ext.ndb.tasklets :add_flow_exception make_context make_default_context set_context
|
None |
from
google.appengine.api.ext.ndb.tasklets :QueueFuture ReducedFuture SerialQueueFuture
|
None |
Exceptions
While the google.cloud.ndb.exceptions
module in the Cloud NDB library contains many of the
same exceptions from the App Engine NDB library, not all of the old
exceptions are available in the new library. The following table lists the
exceptions that are no longer available:
Removed API | Replacement |
---|---|
from
google.appengine.api.datastore_errors :BadKeyError BadPropertyError CommittedButStillApplying EntityNotFoundError InternalError NeedIndexError QueryNotFoundError ReferencePropertyResolveError Timeout TransactionFailedError TransactionNotFoundError
|
google.cloud.ndb.exceptions |
Testing your updates
To set up a test database and run your app locally before deploying it to App Engine:
Run the Datastore mode local emulator to store and retrieve data.
Make sure to follow the instructions for setting environment variables so your app connects to the emulator instead of the production Datastore mode environment.
You can also import data into the emulator if you want to start your test with data pre-loaded into the database.
Use the local development server to run your app.
To make sure the
GOOGLE_CLOUD_PROJECT
environment variable is set correctly during local development, initializedev_appserver
using the following parameter:--application=PROJECT_ID
PROJECT_ID should be your Google Cloud project ID. You can find your project ID by running the
gcloud config list project
command or looking at your project page in the Google Cloud Console.
Deploying your app
Once your app is running in the local development server without errors:
If the app runs without errors, use traffic splitting to slowly ramp up traffic for your updated app. Monitor the app closely for any database issues before routing more traffic to the updated app.