Accessing App Engine with Remote API

The Google Cloud CLI includes a remote_api package that lets you transparently access App Engine services from any Go application. For example, you can use the Remote API to access Datastore from an app running on your local machine or from another one of your App Engine apps.

To view the contents of the remote_api package, see the remote_api package reference.

Enabling Remote API

First, add the remote_api url handler to your app.yaml. For example:

- url: /_ah/remote_api
  script: _go_app

This maps the URL /_ah/remote_api to your Go app. Access to this URL is restricted to administrators for the application by the Remote API handler.

Then you import the remote_api package in one of your project's packages by adding the following line to any of your .go source files:

import _ "google.golang.org/appengine/remote_api"

During program initialization, the remote_api package registers a handler for the /_ah/remote_api path. The underscore in the import declaration means "import this package, but we won't use it directly." Without the underscore, you would receive an "imported but not used" error message on compilation.

Finally, you deploy the update to App Engine. For example:

gcloud app deploy app.yaml

Using the Remote API in a Local Client

The Remote API can be used to write local applications that use App Engine services and access datastore. It is important to note that using the Remote API will incur quota usage on the application you are accessing.

Before beginning, make sure the Remote API is enabled in your App Engine application. The local application can use the Remote API by creating a context with remote_api.NewRemoteContext, and using that in place of the regular App Engine context in all API calls.

// Copyright 2011 Google Inc. All rights reserved.
// Use of this source code is governed by the Apache 2.0
// license that can be found in the LICENSE file.

package main

import (
	"log"
	"time"

	"golang.org/x/net/context"
	"golang.org/x/oauth2/google"

	"google.golang.org/appengine/datastore"
	"google.golang.org/appengine/remote_api"
)

func main() {
	const host = "<your-app-id>.appspot.com"

	ctx := context.Background()

	hc, err := google.DefaultClient(ctx,
		"https://www.googleapis.com/auth/appengine.apis",
		"https://www.googleapis.com/auth/userinfo.email",
		"https://www.googleapis.com/auth/cloud-platform",
	)
	if err != nil {
		log.Fatal(err)
	}

	remoteCtx, err := remote_api.NewRemoteContext(host, hc)
	if err != nil {
		log.Fatal(err)
	}

	q := datastore.NewQuery("Greeting").
		Filter("Date >=", time.Now().AddDate(0, 0, -7))

	log.Println(q.Count(remoteCtx))
}

To run this code, you need to fetch these packages:

$ go get google.golang.org/appengine/...
$ go get golang.org/x/oauth2/...

You need to provide the hostname of your server and a http.Client in the call to NewRemoteContext. The provided http.Client is responsible for passing the required authentication information in each request.

In the sample above, the DefaultClient from the golang.org/x/oauth2/google package provides OAuth 2 credentials via Application Default Credentials.

Limitations and best practices

The remote_api module goes to great lengths to make sure that as far as possible, it behaves exactly like the native App Engine datastore. In some cases, this means doing things that are less efficient than they might otherwise be. When using remote_api, here's a few things to keep in mind:

Every datastore request requires a round-trip

Because you're accessing the datastore over HTTP, there's a bit more overhead and latency than when you access it locally. In order to speed things up and decrease load, try to limit the number of round-trips you do by batching gets and puts, and fetching batches of entities from queries. This is good advice not just for remote_api, but for using the datastore in general, because a batch operation is only considered to be a single Datastore operation.

Requests to remote_api use quota

Because the remote_api operates over HTTP, every datastore call you make incurs quota usage for HTTP requests, bytes in and out, as well as the usual datastore quota you would expect. Bear this in mind if you're using remote_api to do bulk updates.

1 MB API limits apply

As when running natively, the 1MB limit on API requests and responses still applies. If your entities are particularly large, you may need to limit the number you fetch or put at a time to keep below this limit. This conflicts with minimising round-trips, unfortunately, so the best advice is to use the largest batches you can without going over the request or response size limitations. For most entities, this is unlikely to be an issue, however.

Avoid iterating over queries

When you iterate over queries, the SDK fetches entities from the datastore in batches of 20, fetching a new batch whenever it uses up the existing ones. Because each batch has to be fetched in a separate request by remote_api, it's unable to do this as efficiently. Instead, remote_api executes an entirely new query for each batch, using the offset functionality to get further into the results.

If you know how many entities you need, you can do the whole fetch in one request by asking for the number you need.

If you don't know how many entities you will want, you can use cursors to efficiently iterate over large result sets. This also allows you to avoid the 1000 entity limit imposed on normal datastore queries.

Transactions are less efficient

In order to implement transactions via remote_api, it accumulates information on entities fetched inside the transaction, along with copies of entities that were put or deleted inside the transaction. When the transaction is committed, it sends all of this information off to the App Engine server, where it has to fetch all the entities that were used in the transaction again, verify that they have not been modified, then put and delete all the changes the transaction made and commit it. If there's a conflict, the server rolls back the transaction and notifies the client end, which then has to repeat the process all over again.

This approach works, and exactly duplicates the functionality provided by transactions on the local datastore, but is rather inefficient. By all means use transactions where they are necessary, but try to limit the number and complexity of the transactions you execute in the interest of efficiency.