Migrating Outbound Requests

By default, the Python 2.7 runtime uses the URL Fetch service to handle outbound HTTP(S) requests, even if you use the urllib, urllib2, or httplib Python libraries to issue those requests. URL Fetch does not handle requests from the requests library unless you explicitly enable it.

The Python 3 runtime doesn't need an intermediary service to handle outbound requests, so URL Fetch is not available in the Python 3 runtime. If your app uses standard Python libraries to issue requests, the absence of URL Fetch in the Python 3 runtime should not affect your app's ability to make requests. However, we recommend that you test your app in an environment that doesn't use URL Fetch to make sure.

If your app uses URL Fetch APIs directly, for example to make asynchronous requests, you need to migrate those requests to use a standard Python library such as requests.

Key differences between URL Fetch and standard Python libraries

The size limit and quotas for requests that are handled by URL Fetch are different from the size limit and quotas for requests that are not handled by URL Fetch.
In Python 2, when your app sends a request to another App Engine app, URL Fetch adds the X-Appengine-Inbound-Appid request header to assert the app's identity. The app that receives the request can use the identity to determine if it should process the request.

This header is only available in requests that are sent from a Python 2 app. App Engine removes the header if you or anyone else adds it to a request.

For information about asserting and verifying identity without using URL Fetch, see Migrating App Identity to OIDC ID Tokens.
You could use URL Fetch to set a default timeout for all requests. Most Python 3 libraries such as requests and urllib set the default timeout to None, so you should update each request your code makes to specify a timeout.

Overview of the migration process

If your app uses URL Fetch APIs to make requests, update your code to use a standard Python library instead. We recommend that you specify a timeout for each request.
Test your outbound requests in the local development server.
Configure your Python 2 app to bypass URL Fetch when running in App Engine.
Deploy your app.

Replacing URL Fetch APIs with a Python library

If you aren't already using a standard Python library to issue outgoing requests, choose a library and add it to your app's dependencies.

For example, to use the Requests library create a requirements.txt file in the same folder as your app.yaml file and add the following line:
```
requests==2.24.0
```
We recommend you pin the requests library to version 2.24.0, because it is known to work with Python 2 apps. When you deploy your app, App Engine will download all of the dependencies that are defined in the requirements.txt file.

For local development, we recommend that you install dependencies in a virtual environment such as venv.
Search your code for any use of the google.appengine.api.urlfetch module, and update the code to use your Python library.

Making simple HTTPS requests

The following example shows how to make a standard HTTPS request using the requests library:

appengine/standard/migration/urlfetch/requests/main.py

View on GitHub

# Copyright 2020 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

import logging

from flask import Flask

import requests

app = Flask(__name__)


@app.route('/')
def index():
    url = 'http://www.google.com/humans.txt'
    response = requests.get(url)
    response.raise_for_status()
    return response.text


@app.errorhandler(500)
def server_error(e):
    logging.exception('An error occurred during a request.')
    return """
    An internal error occurred: <pre>{}</pre>
    See logs for full stacktrace.
    """.format(e), 500


if __name__ == '__main__':
    # This is used when running locally.
    app.run(host='127.0.0.1', port=8080, debug=True)

Making asynchronous HTTPS requests

The following example shows how to make an asynchronous HTTPS request using the requests library:

appengine/standard/migration/urlfetch/async/main.py

View on GitHub

# Copyright 2020 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

import logging

from flask import Flask, make_response

from requests_futures.sessions import FuturesSession
from time import sleep


TIMEOUT = 10    # Wait this many seconds for background calls to finish
app = Flask(__name__)


@app.route('/')     # Fetch and return remote page asynchronously
def get_async():
    session = FuturesSession()
    url = 'http://www.google.com/humans.txt'

    rpc = session.get(url)

    # ... do other things ...

    resp = make_response(rpc.result().text)
    resp.headers['Content-type'] = 'text/plain'
    return resp


@app.route('/callback')     # Fetch and return remote pages using callback
def get_callback():
    global response_text
    global counter

    response_text = ''
    counter = 0

    def cb(resp, *args, **kwargs):
        global response_text
        global counter

        if 300 <= resp.status_code < 400:
            return  # ignore intermediate redirection responses

        counter += 1
        response_text += 'Response number {} is {} bytes from {}\n'.format(
            counter, len(resp.text), resp.url)

    session = FuturesSession()
    urls = [
        'https://google.com/',
        'https://www.google.com/humans.txt',
        'https://www.github.com',
        'https://www.travis-ci.org'
    ]

    futures = [session.get(url, hooks={'response': cb}) for url in urls]

    # No wait functionality in requests_futures, so check every second to
    # see if all callbacks are done, up to TIMEOUT seconds
    for elapsed_time in range(TIMEOUT+1):
        all_done = True
        for future in futures:
            if not future.done():
                all_done = False
                break
        if all_done:
            break
        sleep(1)

    resp = make_response(response_text)
    resp.headers['Content-type'] = 'text/plain'
    return resp


@app.errorhandler(500)
def server_error(e):
    logging.exception('An error occurred during a request.')
    return """
    An internal error occurred: <pre>{}</pre>
    See logs for full stacktrace.
    """.format(e), 500

Testing locally

If you updated any of your outbound requests, run your Python 2 app in the local development server and confirm that the requests succeed.

Bypassing URL Fetch

To stop URL Fetch from handling requests when you deploy your app to App Engine:

In your app.yaml file, set the GAE_USE_SOCKETS_HTTPLIB environment variable to any value. The value can be any value including an empty string. For example:
```
env_variables:
  GAE_USE_SOCKETS_HTTPLIB : ''
```
If you enabled URL Fetch to handle requests sent from the requests library, you can remove the Requests AppEngineAdapter from your app.

For example, remove requests_toolbelt.adapters.appengine from your appengine_config.py file and requests_toolbelt.adapters.appengine.monkeypatch() from your Python files.

Note that even if you bypass URL Fetch as described in the previous steps, your Python 2 app can still use the URL Fetch API directly. Be sure to remove any use of the google.appengine.api.urlfetch module before you run your app in a Python 3 environment.

Deploying your app

Once you are ready to deploy your app, you should:

Test the app on App Engine.

View the App Engine Quotas page in the Google Cloud Console to confirm that your app isn't making Url Fetch API Calls.

View URL Fetch API Calls
If the app runs without errors, use traffic splitting to slowly ramp up traffic for your updated app. Monitor the app closely for any issues before routing more traffic to the updated app.