Issuing HTTP(S) Requests

Region ID

The REGION_ID is an abbreviated code that Google assigns based on the region you select when you create your app. The code does not correspond to a country or province, even though some region IDs may appear similar to commonly used country and province codes. For apps created after February 2020, REGION_ID.r is included in App Engine URLs. For existing apps created before this date, the region ID is optional in the URL.

Learn more about region IDs.

This page describes how to issue HTTP(S) requests from your App Engine app.

By default, App Engine uses the URL Fetch service to issue outbound HTTP(S) requests.

For details on request size limits and which headers are sent in a URL Fetch request, see Outbound Requests.

If you have set up Serverless VPC Access or if you use the Sockets API, you need to stop URL Fetch from handling requests. URL Fetch causes requests to your VPC network or to the Sockets API to fail. After you disable URL Fetch, the standard Python library will handle HTTP requests. If you need the features provided by URL Fetch for specific requests, you can use the urlfetch library directly for those specific requests.

Issuing an HTTP request

To issue an outbound HTTP request, use the urlfetch.fetch method. For improved code portability, you can also use the Python standard libraries urllib, urllib2, or httplib to issue HTTP requests. When you use these libraries in App Engine, they perform HTTP requests using App Engine's URL Fetch service. You can also use the third-party requests library as long as you configure it to use URLFetch.

urlfetch

The following snippets demonstrate how to perform a basic HTTP GET request using urlfetch. First, import the urlfetch library from the App Engine SDK:

from google.appengine.api import urlfetch

Next, use urlfetch to perform the GET request:

url = 'http://www.google.com/humans.txt'
try:
    result = urlfetch.fetch(url)
    if result.status_code == 200:
        self.response.write(result.content)
    else:
        self.response.status_code = result.status_code
except urlfetch.Error:
    logging.exception('Caught exception fetching url')

The following snippet demonstrates how to perform a more advanced request, submitting data from a web form via a HTTP POST request using urlfetch:

try:
    form_data = urllib.urlencode(UrlPostHandler.form_fields)
    headers = {'Content-Type': 'application/x-www-form-urlencoded'}
    result = urlfetch.fetch(
        url='http://localhost:8080/submit_form',
        payload=form_data,
        method=urlfetch.POST,
        headers=headers)
    self.response.write(result.content)
except urlfetch.Error:
    logging.exception('Caught exception fetching url')

urllib2

The following snippets demonstrate how to perform a basic HTTP GET request using urllib2. First, import the urllib2 library:

import urllib2

Next, use urllib2 to perform the GET request:

url = 'http://www.google.com/humans.txt'
try:
    result = urllib2.urlopen(url)
    self.response.write(result.read())
except urllib2.URLError:
    logging.exception('Caught exception fetching url')

requests

To use requests, you'll need to install both requests and requests-toolbelt using the vendoring instructions.

Once installed, use the requests_toolbelt.adapters.appengine module to configure requests to use URLFetch:

import requests
import requests_toolbelt.adapters.appengine

# Use the App Engine Requests adapter. This makes sure that Requests uses
# URLFetch.
requests_toolbelt.adapters.appengine.monkeypatch()

Once configured, you can use requests normally:

url = 'http://www.google.com/humans.txt'
response = requests.get(url)
response.raise_for_status()
return response.text

For more information about requests' support for Google App Engine, see the documentation for urllib3.contrib.appengine and requests_toolbelt.appengine

Setting a request timeout

You can adjust the default deadline by using the urlfetch.set_default_fetch_deadline() function. This function stores the new default deadline on a thread-local variable, so it must be set for each request, for example, in a custom middleware.

Disabling redirects

If you are using URL Fetch, the underlying URL Fetch service follows up to five redirects by default. These redirects could forward sensitive information, such as authorization headers, to the redirected destination. If your app does not require HTTP redirects, it is recommended that you disable the redirects.

To instruct the URL Fetch service to not follow redirects, set the fetch method's follow_redirects parameter to False.

Issuing an HTTPS request

To issue an HTTPS request, set the validate_certificate parameter to true when calling the urlfetch.fetch() method.

Issuing an asynchronous request

HTTP(S) requests are synchronous by default. To issue an asynchronous request, your application must:

  1. Create a new RPC object using urlfetch.create_rpc(). This object represents your asynchronous call in subsequent method calls.
  2. Call urlfetch.make_fetch_call() to make the request. This method takes your RPC object and the request target's URL as parameters.
  3. Call the RPC object's get_result() method. This method returns the result object if the request is successful, and raises an exception if an error occurred during the request.

The following snippets demonstrate how to make a basic asynchronous request from a Python application. First, import the urlfetch library from the App Engine SDK:

from google.appengine.api import urlfetch

Next, use urlfetch to make the asynchronous request:

rpc = urlfetch.create_rpc()
urlfetch.make_fetch_call(rpc, 'http://www.google.com/')

# ... do other things ...
try:
    result = rpc.get_result()
    if result.status_code == 200:
        text = result.content
        self.response.write(text)
    else:
        self.response.status_int = result.status_code
        self.response.write('URL returned status code {}'.format(
            result.status_code))
except urlfetch.DownloadError:
    self.response.status_int = 500
    self.response.write('Error fetching URL')

Setting a request timeout

To set a timeout for your request, set the urlfetch.create_rpc() method's deadline parameter when you create your RPC object.

Using a callback function

You can define a callback function for your RPC object. The function will be called when your application calls a method on the object—such as wait(), checksuccess(), or get_result()—that causes the object to wait for the request to complete.

To use a callback function to handle the result of your fetch call:

  1. Create a helper function to define the scope of the callback.
  2. Create a handler function to handle the result of your fetch call.
  3. Set your RPC object's callback attribute to the helper function.

The following snippet demonstrates how to invoke a callback function:

def handle_result(rpc):
    result = rpc.get_result()
    self.response.write(result.content)
    logging.info('Handling RPC in callback: result {}'.format(result))

urls = ['http://www.google.com',
        'http://www.github.com',
        'http://www.travis-ci.org']
rpcs = []
for url in urls:
    rpc = urlfetch.create_rpc()
    rpc.callback = functools.partial(handle_result, rpc)
    urlfetch.make_fetch_call(rpc, url)
    rpcs.append(rpc)

# ... do other things ...

# Finish all RPCs, and let callbacks process the results.

for rpc in rpcs:
    rpc.wait()

logging.info('Done waiting for RPCs')

Issuing a request to another App Engine app

When issuing a request to another App Engine app, your App Engine app must assert its identity by adding the header X-Appengine-Inbound-Appid to the request. If you instruct the URL Fetch service to not follow redirects, App Engine will add this header to requests automatically.

See Disabling redirects for guidance on disabling redirects.

What's next

Learn about the URL Fetch service, such as the headers that are sent in a URL Fetch request in Outbound Requests.