Accessing App Engine with Remote API

Region ID

The REGION_ID is an abbreviated code that Google assigns based on the region you select when you create your app. The code does not correspond to a country or province, even though some region IDs may appear similar to commonly used country and province codes. For apps created after February 2020, REGION_ID.r is included in App Engine URLs. For existing apps created before this date, the region ID is optional in the URL.

Learn more about region IDs.

OK

The Remote API library allows any Python client to access services available to App Engine applications.

For example, if your App Engine application uses Datastore or Google Cloud Storage, a Python client could access those storage resources using the Remote API.

You can use the Remote API to access your application's data store from an app running on your local machine, or from a local interactive Remote API shell. The Remote API interacts with real services, so this access does use quota and billable resources.

Caution: This solution is no longer recommended: Apps that use this API can only run in the Python 2 runtime and will need to upgrade to a recommended solution before migrating to the Python 3 runtime.

Enabling Remote API access in your app

The easiest way to enable the Remote API for your application is to use the builtins directive in the app.yaml file for your app, which specifies the default URL /_ah/remote_api/. However, you can instead use the url directive in that same file to specify some other URL.

builtin

The builtins directive in the app.yaml file makes the Remote API available at the default URL /_ah/remote_api:

runtime: python27
api_version: 1
threadsafe: true

builtins:
- remote_api: on

URL

Using the url directive in app.yaml lets you specify a different URL for use with the Remote API:

- url: /some-URL/*
  script: google.appengine.ext.remote_api.handler.application

Make sure you deploy your application to App Engine after making this change.

Using the Remote API shell

The Python SDK includes a Remote API shell, which allows you to invoke Python commands on App Engine services used by your application. You don't need to supply any additional authentication, because this automatically uses the same credentials you used to upload the app to App Engine.

To start the Remote API shell:

1. Invoke the following command from a terminal window on your local machine:

   SDK-INSTALL-DIRECTORY/remote_api_shell.py -s YOUR-PROJECT-ID. REGION_ID.r.appspot.com

   Replace [SDK-INSTALL-DIRECTORY] with the path to the App Engine SDK for Python, and [YOUR-PROJECT-ID] with your project ID.

2. In the interactive shell that is displayed, invoke the Python commands you want to run. For example, if your application uses Datastore, you could invoke the following ndb query to fetch 10 records:

    >>> from google.appengine.ext import ndb
    >>>
    >>> # Fetch 10 keys from the datastore
    >>> ndb.Query().fetch(10, keys_only=True)
   

Note: In the shell, you can access files and resources on your local machine in addition to the services running in Google Cloud Platform. You can also import modules from your application if you run the shell from your application directory.

Using the Remote API in a local client

You can also use the Remote API in local applications to access services used by your app running in App Engine.

To use the Remote API in a local application:

1. Enable the remote API.

2. Export the PYTHONPATH environment variable for your Python directory, for example:

    export PYTHONPATH=/usr/somedir/v3/bin/python2.7
   

   Replace that path with the actual values for your python location.

3. Add your App Engine SDK for Python location to PYTHONPATH:

    export GAE_SDK_ROOT="/usr/local/home/mydir/google_appengine"
    export PYTHONPATH=${GAE_SDK_ROOT}:${PYTHONPATH}
   

   Replace the SDK path shown above with your actual path to the App Engine SDK.

4. In your client code, import dev_appserver and call dev_appserver.fix_sys_path() to ensure all of the App Engine SDK modules import correctly:

   try:
       import dev_appserver
       dev_appserver.fix_sys_path()

5. Add the following remote_api_stub code to your application, making sure you pass it your project ID in your code:

   remote_api_stub.ConfigureRemoteApiForOAuth(
       '{}.appspot.com'.format(project_id),
       '/_ah/remote_api')

   If you don't use the default URL /_ah/remote_api for the Remote API, you'll have to change the code above to reflect the URL you are using. For the definition and documentation for remote_api_stub.ConfigureRemoteApiForOAuth, see the SDK file [SDK-INSTALL-DIRECTORY]/google/appengine/ext/remote_api/remote_api_stub.py.

6. Add any needed App Engine imports and Python code to access the desired App Engine services. The following sample code accesses the project's data store:

   
   import argparse
   
   try:
       import dev_appserver
       dev_appserver.fix_sys_path()
   except ImportError:
       print('Please make sure the App Engine SDK is in your PYTHONPATH.')
       raise
   
   from google.appengine.ext import ndb
   from google.appengine.ext.remote_api import remote_api_stub
   
   
   def main(project_id):
       remote_api_stub.ConfigureRemoteApiForOAuth(
           '{}.appspot.com'.format(project_id),
           '/_ah/remote_api')
   
       # List the first 10 keys in the datastore.
       keys = ndb.Query().fetch(10, keys_only=True)
   
       for key in keys:
           print(key)
   
   
   if __name__ == '__main__':
       parser = argparse.ArgumentParser(
           description=__doc__,
           formatter_class=argparse.RawDescriptionHelpFormatter)
       parser.add_argument('project_id', help='Your Project ID.')
   
       args = parser.parse_args()
   
       main(args.project_id)

7. With your application deployed to App Engine, start your Remote API client:

    python your-client.py YOUR-PROJECT-ID
   

   Replacing your-client.py with your client module, and YOUR-PROJECT-ID with your project ID. This assumes your client accepts project ID as the command-line input, following the client.py code sample.

Limitations and best practices

The remote_api module goes to great lengths to make sure that as far as possible, it behaves exactly like the native App Engine datastore. In some cases, this means doing things that are less efficient than they might otherwise be. When using remote_api, here's a few things to keep in mind:

Every datastore request requires a round-trip

Because you're accessing the datastore over HTTP, there's a bit more overhead and latency than when you access it locally. In order to speed things up and decrease load, try to limit the number of round-trips you do by batching gets and puts, and fetching batches of entities from queries. This is good advice not just for remote_api, but for using the datastore in general, because a batch operation is only considered to be a single Datastore operation. For example, instead of this:

for key in keys:
  rec = key.get()
  rec.foo = bar
  rec.put()

you can do this:

records = ndb.get_multi(keys)
for rec in records:
  rec.foo = bar
  ndb.put_multi(records)

Both examples have the same effect, but the latter requires only two roundtrips in total, while the former requires two roundtrips for each entity.

Requests to remote_api use quota

Because the remote_api operates over HTTP, every datastore call you make incurs quota usage for HTTP requests, bytes in and out, as well as the usual datastore quota you would expect. Bear this in mind if you're using remote_api to do bulk updates.

1 MB API limits apply

As when running natively, the 1MB limit on API requests and responses still applies. If your entities are particularly large, you may need to limit the number you fetch or put at a time to keep below this limit. This conflicts with minimising round-trips, unfortunately, so the best advice is to use the largest batches you can without going over the request or response size limitations. For most entities, this is unlikely to be an issue, however.

Avoid iterating over queries

One common pattern with datastore access is the following:

q = MyModel.query()
for entity in q:
  # Do something with entity

When you do this, the SDK fetches entities from the datastore in batches of 20, fetching a new batch whenever it uses up the existing ones. Because each batch has to be fetched in a separate request by remote_api, it's unable to do this as efficiently. Instead, remote_api executes an entirely new query for each batch, using the offset functionality to get further into the results.

If you know how many entities you need, you can do the whole fetch in one request by asking for the number you need:

entities = MyModel.query().fetch(100)
for entity in entities:
  # Do something with entity

If you don't know how many entities you will want, you can use cursors to efficiently iterate over large result sets. This also allows you to avoid the 1000 entity limit imposed on normal datastore queries.

Transactions are less efficient

In order to implement transactions via remote_api, it accumulates information on entities fetched inside the transaction, along with copies of entities that were put or deleted inside the transaction. When the transaction is committed, it sends all of this information off to the App Engine server, where it has to fetch all the entities that were used in the transaction again, verify that they have not been modified, then put and delete all the changes the transaction made and commit it. If there's a conflict, the server rolls back the transaction and notifies the client end, which then has to repeat the process all over again.

This approach works, and exactly duplicates the functionality provided by transactions on the local datastore, but is rather inefficient. By all means use transactions where they are necessary, but try to limit the number and complexity of the transactions you execute in the interest of efficiency.