Remote API for legacy bundled services

The Java SDK includes a library called Remote API that lets you transparently access App Engine services from any Java application. For example, you can use Remote API to access a production datastore from an app running on your local machine. You can also use Remote API to access the datastore of one App Engine app from a different App Engine app.

Configuring Remote API on the Server

The server component of Remote API is a Java servlet that is part of the App Engine runtime for Java. This servlet receives requests from the Remote API client, dispatches them to the appropriate backend service, and then returns the result of the service call to the client. To install the Remote API servlet, add the following to your web.xml:

<servlet>
  <display-name>Remote API Servlet</display-name>
  <servlet-name>RemoteApiServlet</servlet-name>
  <servlet-class>com.google.apphosting.utils.remoteapi.RemoteApiServlet</servlet-class>
  <load-on-startup>1</load-on-startup>
</servlet>
<servlet-mapping>
  <servlet-name>RemoteApiServlet</servlet-name>
  <url-pattern>/remote_api</url-pattern>
</servlet-mapping>

The servlet returns an error if there is no authenticated user or the authenticated user is not an admin of your application, so there is no need to configure any additional security. Once you've deployed your app with these settings, any app with the Remote API client installed can use its services. This includes Python clients that are using the Python Remote API.

Configuring Remote API on a Standalone Client

To configure the client component of Remote API for use inside a Java application, add ${SDK_ROOT}/lib/impl/appengine-api.jar and ${SDK_ROOT}/lib/appengine-remote-api.jar to your classpath. Then, in your code, configure and install Remote API:

import com.google.appengine.tools.remoteapi.RemoteApiInstaller;
import com.google.appengine.tools.remoteapi.RemoteApiOptions;

// ...
RemoteApiOptions options = new RemoteApiOptions()
    .server("[APP_ID].[REGION_ID].r.appspot.com", 443)
    .useApplicationDefaultCredential();

RemoteApiInstaller installer = new RemoteApiInstaller();
installer.install(options);
// ... all API calls executed remotely
installer.uninstall();

The Remote API client will rely on Application Default Credentials that use OAuth 2.0.

In order to get a credential run:

gcloud auth application-default login

You can just as easily connect to an App Engine app running locally in the Development Server:

RemoteApiOptions options = new RemoteApiOptions()
    .server("localhost", 8888) // server name must equal "localhost"
    .useDevelopmentServerCredential();

Here's a complete Java application that, when run, inserts an Entity into the datastore:

public class RemoteApiExample {

  public static void main(String[] args) throws IOException {
    String serverString = args[0];
    RemoteApiOptions options;
    if (serverString.equals("localhost")) {
      options = new RemoteApiOptions().server(serverString,
        8080).useDevelopmentServerCredential();
    } else {
      options = new RemoteApiOptions().server(serverString,
        443).useApplicationDefaultCredential();
    }
    RemoteApiInstaller installer = new RemoteApiInstaller();
    installer.install(options);
    try {
      DatastoreService ds = DatastoreServiceFactory.getDatastoreService();
      System.out.println("Key of new entity is " + ds.put(new Entity("Hello Remote API!")));
    } finally {
      installer.uninstall();
    }
  }
}

Configuring Remote API on an App Engine Client

You can also use Remote API to access the services of one App Engine application from a different App Engine application. You need to add ${SDK_ROOT}/lib/appengine-remote-api.jar to your WEB-INF/lib directory and then, in your client App Engine app, configure and install Remote API just as you did in your standalone Java client.

Notice that RemoteApiInstaller only installs Remote API on the thread performing the installation, so be careful not to share instances of this class across threads.

Using Remote API with Maven

To use the Remote API feature in your Maven project, add the following dependencies to your project pom.xml file:

<dependency>
  <groupId>com.google.appengine</groupId>
  <artifactId>appengine-remote-api</artifactId>
  <version>1.9.64</version>
</dependency>
<dependency>
  <groupId>com.google.appengine</groupId>
  <artifactId>appengine-api-1.0-sdk</artifactId>
  <version>1.9.64</version>
</dependency>

Limitations and best practices

The remote_api module goes to great lengths to make sure that as far as possible, it behaves exactly like the native App Engine datastore. In some cases, this means doing things that are less efficient than they might otherwise be. When using remote_api, here's a few things to keep in mind:

Every datastore request requires a round-trip

Because you're accessing the datastore over HTTP, there's a bit more overhead and latency than when you access it locally. In order to speed things up and decrease load, try to limit the number of round-trips you do by batching gets and puts, and fetching batches of entities from queries. This is good advice not just for remote_api, but for using the datastore in general, because a batch operation is only considered to be a single Datastore operation.

Requests to remote_api use quota

Because the remote_api operates over HTTP, every datastore call you make incurs quota usage for HTTP requests, bytes in and out, as well as the usual datastore quota you would expect. Bear this in mind if you're using remote_api to do bulk updates.

1 MB API limits apply

As when running natively, the 1MB limit on API requests and responses still applies. If your entities are particularly large, you may need to limit the number you fetch or put at a time to keep below this limit. This conflicts with minimising round-trips, unfortunately, so the best advice is to use the largest batches you can without going over the request or response size limitations. For most entities, this is unlikely to be an issue, however.

Avoid iterating over queries

When you iterate over queries, the SDK fetches entities from the datastore in batches of 20, fetching a new batch whenever it uses up the existing ones. Because each batch has to be fetched in a separate request by remote_api, it's unable to do this as efficiently. Instead, remote_api executes an entirely new query for each batch, using the offset functionality to get further into the results.

If you know how many entities you need, you can do the whole fetch in one request by asking for the number you need.

If you don't know how many entities you will want, you can use cursors to efficiently iterate over large result sets. This also allows you to avoid the 1000 entity limit imposed on normal datastore queries.

Transactions are less efficient

In order to implement transactions via remote_api, it accumulates information on entities fetched inside the transaction, along with copies of entities that were put or deleted inside the transaction. When the transaction is committed, it sends all of this information off to the App Engine server, where it has to fetch all the entities that were used in the transaction again, verify that they have not been modified, then put and delete all the changes the transaction made and commit it. If there's a conflict, the server rolls back the transaction and notifies the client end, which then has to repeat the process all over again.

This approach works, and exactly duplicates the functionality provided by transactions on the local datastore, but is rather inefficient. By all means use transactions where they are necessary, but try to limit the number and complexity of the transactions you execute in the interest of efficiency.