The Google Cloud CLI includes a remote_api
package that lets you transparently
access App Engine services from any Go application. For example, you can use the
Remote API to access Datastore from an app running on your local
machine or from another one of your App Engine apps.
To view the contents of the remote_api
package, see the
remote_api
package
reference.
Enabling Remote API
First, add the remote_api
url
handler to your app.yaml
. For example:
This maps the URL /_ah/remote_api
to your Go app. Access to this URL is
restricted to administrators for the application by the Remote API handler.
Then you import the remote_api
package in one of your project's
packages by adding the following line to any of your .go
source files:
During program initialization, the remote_api
package registers a handler
for the /_ah/remote_api
path. The underscore in the import
declaration means "import this package, but we won't use it directly." Without
the underscore, you would receive an "imported but not used" error message on
compilation.
Finally, you deploy the update to App Engine. For example:
Using the Remote API in a Local Client
The Remote API can be used to write local applications that use App Engine services and access datastore. It is important to note that using the Remote API will incur quota usage on the application you are accessing.
Before beginning, make sure the Remote API is enabled in your App Engine
application. The local application can use the Remote API by creating a context
with remote_api.NewRemoteContext
, and using that in place of the regular App
Engine context in all API calls.
To run this code, you need to fetch these packages:
$ go get google.golang.org/appengine/...
$ go get golang.org/x/oauth2/...
You need to provide the hostname of your server and a http.Client
in the call
to NewRemoteContext
. The provided http.Client
is responsible for passing
the required authentication information in each request.
In the sample above, the DefaultClient
from the golang.org/x/oauth2/google
package provides OAuth 2 credentials via Application Default Credentials.
Limitations and best practices
The remote_api module goes to great lengths to make sure that as far as possible, it behaves exactly like the native App Engine datastore. In some cases, this means doing things that are less efficient than they might otherwise be. When using remote_api, here's a few things to keep in mind:
Every datastore request requires a round-trip
Because you're accessing the datastore over HTTP, there's a bit more overhead and latency than when you access it locally. In order to speed things up and decrease load, try to limit the number of round-trips you do by batching gets and puts, and fetching batches of entities from queries. This is good advice not just for remote_api, but for using the datastore in general, because a batch operation is only considered to be a single Datastore operation.
Requests to remote_api use quota
Because the remote_api operates over HTTP, every datastore call you make incurs quota usage for HTTP requests, bytes in and out, as well as the usual datastore quota you would expect. Bear this in mind if you're using remote_api to do bulk updates.
1 MB API limits apply
As when running natively, the 1MB limit on API requests and responses still applies. If your entities are particularly large, you may need to limit the number you fetch or put at a time to keep below this limit. This conflicts with minimising round-trips, unfortunately, so the best advice is to use the largest batches you can without going over the request or response size limitations. For most entities, this is unlikely to be an issue, however.
Avoid iterating over queries
When you iterate over queries, the SDK fetches entities from the datastore in batches of 20, fetching a new batch whenever it uses up the existing ones. Because each batch has to be fetched in a separate request by remote_api, it's unable to do this as efficiently. Instead, remote_api executes an entirely new query for each batch, using the offset functionality to get further into the results.
If you know how many entities you need, you can do the whole fetch in one request by asking for the number you need.
If you don't know how many entities you will want, you can use cursors to efficiently iterate over large result sets. This also allows you to avoid the 1000 entity limit imposed on normal datastore queries.
Transactions are less efficient
In order to implement transactions via remote_api, it accumulates information on entities fetched inside the transaction, along with copies of entities that were put or deleted inside the transaction. When the transaction is committed, it sends all of this information off to the App Engine server, where it has to fetch all the entities that were used in the transaction again, verify that they have not been modified, then put and delete all the changes the transaction made and commit it. If there's a conflict, the server rolls back the transaction and notifies the client end, which then has to repeat the process all over again.
This approach works, and exactly duplicates the functionality provided by transactions on the local datastore, but is rather inefficient. By all means use transactions where they are necessary, but try to limit the number and complexity of the transactions you execute in the interest of efficiency.