Issuing HTTP(S) Requests

Python |Java |PHP |Go

This page describes how to issue HTTP(S) requests from your App Engine app.

App Engine uses the URL Fetch service to issue outbound HTTP(S) requests. For details about how the URL Fetch service is implemented and which headers are sent in a URL Fetch request, see Outbound Requests.

  1. Issuing an HTTP request
    1. Setting a request timeout
  2. Issuing an HTTPS request
  3. Issuing an asynchronous request
    1. Setting a request timeout
    2. Using a callback function
  4. Issuing a request to another App Engine app

Issuing an HTTP request

To issue an outbound HTTP request, use the urlfetch.fetch method. For improved code portability, you can also use the Python standard libraries urllib, urllib2, or httplib to issue HTTP requests. When you use these libraries in App Engine, they perform HTTP requests using App Engine's URL Fetch service. You can also use the third-party requests library as long as you configure it to use URLFetch.

urlfetch

The following snippets demonstrate how to perform a basic HTTP GET request using urlfetch. First, import the urlfetch library from the App Engine SDK:

from google.appengine.api import urlfetch

Next, use urlfetch to perform the GET request:

url = 'http://www.google.com/humans.txt'
try:
    result = urlfetch.fetch(url)
    if result.status_code == 200:
        self.response.write(result.content)
    else:
        self.response.status_code = result.status_code
except urlfetch.Error:
    logging.exception('Caught exception fetching url')

The following snippet demonstrates how to perform a more advanced request, submitting data from a web form via a HTTP POST request using urlfetch:

try:
    form_data = urllib.urlencode(UrlPostHandler.form_fields)
    headers = {'Content-Type': 'application/x-www-form-urlencoded'}
    result = urlfetch.fetch(
        url='http://localhost:8080/submit_form',
        payload=form_data,
        method=urlfetch.POST,
        headers=headers)
    self.response.write(result.content)
except urlfetch.Error:
    logging.exception('Caught exception fetching url')

urllib2

The following snippets demonstrate how to perform a basic HTTP GET request using urllib2. First, import the urllib2 library:

import urllib2

Next, use urllib2 to perform the GET request:

url = 'http://www.google.com/humans.txt'
try:
    result = urllib2.urlopen(url)
    self.response.write(result.read())
except urllib2.URLError:
    logging.exception('Caught exception fetching url')

requests

To use requests, you'll need to install both requests and requests-toolbelt using the vendoring instructions.

Once installed, use the requests_toolbelt.adapters.appengine module to configure requests to use URLFetch:

import requests
import requests_toolbelt.adapters.appengine

# Use the App Engine Requests adapter. This makes sure that Requests uses
# URLFetch.
requests_toolbelt.adapters.appengine.monkeypatch()

Once configured, you can use requests normally:

url = 'http://www.google.com/humans.txt'
response = requests.get(url)
response.raise_for_status()
return response.text

For more information about requests' support for Google App Engine, see the documentation for urllib3.contrib.appengine and requests_toolbelt.appengine

Setting a request timeout

You can adjust the default deadline by using the urlfetch.set_default_fetch_deadline() function. This function stores the new default deadline on a thread-local variable, so it must be set for each request, for example, in a custom middleware.

Issuing an HTTPS request

To issue an HTTPS request, set the validate_certificate parameter to true when calling the urlfetch.fetch() method.

Issuing an asynchronous request

HTTP(S) requests are synchronous by default. To issue an asynchronous request, your application must:

  1. Create a new RPC object using urlfetch.create_rpc(). This object represents your asynchronous call in subsequent method calls.
  2. Call urlfetch.make_fetch_call() to make the request. This method takes your RPC object and the request target's URL as parameters.
  3. Call the RPC object's get_result() method. This method returns the result object if the request is successful, and raises an exception if an error occurred during the request.

The following snippets demonstrate how to make a basic asynchronous request from a Python application. First, import the urlfetch library from the App Engine SDK:

from google.appengine.api import urlfetch

Next, use urlfetch to make the asynchronous request:

rpc = urlfetch.create_rpc()
urlfetch.make_fetch_call(rpc, "http://www.google.com/")

# ... do other things ...
try:
    result = rpc.get_result()
    if result.status_code == 200:
        text = result.content
        self.response.write(text)
    else:
        self.response.status_code = result.status_code
        logging.error("Error making RPC request")
except urlfetch.DownloadError:
    logging.error("Error fetching URL0")

Setting a request timeout

To set a timeout for your request, set the urlfetch.create_rpc() method's deadline parameter when you create your RPC object.

Using a callback function

You can define a callback function for your RPC object. The function will be called when your application calls a method on the object—such as wait(), checksuccess(), or get_result()—that causes the object to wait for the request to complete.

To use a callback function to handle the result of your fetch call:

  1. Create a helper function to define the scope of the callback.
  2. Create a handler function to handle the result of your fetch call.
  3. Set your RPC object's callback attribute to the helper function.

The following snippet demonstrates how to invoke a callback function:

def handle_result(rpc):
    result = rpc.get_result()
    self.response.write(result.content)
    logging.info("Handling RPC in callback: result {}".format(result))

urls = ['http://www.google.com',
        'http://www.github.com',
        'http://www.travis-ci.org']
rpcs = []
for url in urls:
    rpc = urlfetch.create_rpc()
    rpc.callback = functools.partial(handle_result, rpc)
    urlfetch.make_fetch_call(rpc, url)
    rpcs.append(rpc)

# ... do other things ...

# Finish all RPCs, and let callbacks process the results.

for rpc in rpcs:
    rpc.wait()

logging.info("Done waiting for RPCs")

Issuing a request to another App Engine app

When issuing a request to another App Engine app, your App Engine app must assert its identity by adding the header X-Appengine-Inbound-Appid to the request. If you instruct the URL Fetch service to not follow redirects, App Engine will add this header to requests automatically.

To instruct the URL Fetch service to not follow redirects, set the fetch follow_redirects parameter to False.

What's next

Learn about the URL Fetch service, such as the headers that are sent in a URL Fetch request in Outbound Requests.

Send feedback about...

App Engine standard environment for Python