Google Cloud Platform

Asynchronous Requests

A Python app can make an asynchronous request to the URL Fetch service to fetch a URL in the background, while the application code does other things.

This API is provided by the google.appengine.api.urlfetch

Making Asynchronous Requests

To use the URL Fetch service, your application needs to first call the create_rpc() function to create an RPC object that represents the asynchronous call. It then calls the make_fetch_call() function, passing the RPC object as an argument, to initiate the fetch. The app calls methods on the RPC object to wait for the call to complete and retrieve the results.

The make_fetch_call() function initiates the URL fetch then returns immediately, allowing the app to do other things while the service fetches the URL. When the app is ready for the results, it calls the get_result() method of the RPC object. If the service has not completed the fetch when the app calls get_result(), the method waits until the request is complete (or has reached the deadline, or an error occurs). The method returns the result object, or raises an exception if an error occurred during the fetch.

from google.appengine.api import urlfetch

rpc = urlfetch.create_rpc()
urlfetch.make_fetch_call(rpc, "")

# ... do other things ...

    result = rpc.get_result()
    if result.status_code == 200:
        text = result.content
        # ...
except urlfetch.DownloadError:
    # Request timed out or failed.
    # ...

You can set a maximum amount of time the service will wait for the remote host to respond when you create the RPC object. The timer starts when make_fetch_call() is called.

The RPC object can be created with an optional callback function. This function is called when the application calls a method on the RPC object that causes it to wait for the request to complete. The callback does not occur in the background; the application must call a method (wait(), check_success() or get_result()) to invoke the callback function. You can assign the callback function to the RPC object after the object has been created, so the callback can refer to the RPC and call check_success() or get_result():

def handle_result(rpc):
    result = rpc.get_result()
    # ... Do something with result...

# Use a helper function to define the scope of the callback.
def create_callback(rpc):
    return lambda: handle_result(rpc)

rpcs = []
for url in urls:
    rpc = urlfetch.create_rpc()
    rpc.callback = create_callback(rpc)
    urlfetch.make_fetch_call(rpc, url)

# ...

# Finish all RPCs, and let callbacks process the results.
for rpc in rpcs:

Due to how RPC waiting is implemented, if you have multiple simultaneous asynchronous requests with callbacks, calling the wait() method of one RPC object may cause the callbacks of other RPC objects to trigger if they complete while the app is waiting. Calling wait() on an RPC whose callback has already been called will not trigger the callback again.

The synchronous fetch() function is equivalent to starting an asynchronous call, then immediately calling get_result().

An RPC object can only represent one URL fetch call in its lifetime. To make multiple asynchronous calls, you must create an RPC object for each call.

The development server does not make asynchronous URL fetch calls in the background. Instead, it performs the URL fetch synchronously when the RPC object's wait() method is called. Asynchronous URL fetch calls will have different performance characteristics on App Engine than on the development server. Such calls may complete out of order on App Engine, but will always complete in order on the development server.