By default, the Python 2.7 runtime
uses the URL Fetch service
to handle outbound HTTP(S) requests, even if you use the urllib
, urllib2
,
or httplib
Python libraries to issue those requests. URL Fetch does not handle
requests from the requests
library unless you explicitly
enable it.
The Python 3 runtime doesn't need an intermediary service to handle outbound
requests. If you want to migrate away from using URL Fetch APIs
but still need similar functionality, you should migrate those requests to use
a standard Python library such as the
requests
library.
Key differences between URL Fetch and standard Python libraries
The size limit and quotas for requests that are handled by URL Fetch are different from the size limit and quotas for requests that are not handled by URL Fetch.
With URL Fetch, when your app sends a request to another App Engine app, URL Fetch adds the
X-Appengine-Inbound-Appid
request header to assert the app's identity. The app that receives the request can use the identity to determine if it should process the request.This header is only available in requests that are sent from your app if it uses URL Fetch. App Engine removes the header if you or a third-party adds it to a request.
For information about asserting and verifying identity without using URL Fetch, see Migrating App Identity to OIDC ID Tokens.
For an example of how to use the request header to verify the calling app's identity when requests are sent between App Engine apps, see the App Engine to App Engine Request Sample.
You could use URL Fetch to set a default timeout for all requests. Most Python 3 libraries such as
requests
andurllib
set the default timeout toNone
, so you should update each request your code makes to specify a timeout.
Overview of the migration process
If your app uses URL Fetch APIs to make requests, update your code to use a standard Python library instead. We recommend that you specify a timeout for each request.
Test your outbound requests in the local development server.
Configure your app to bypass URL Fetch when running in App Engine.
Deploy your app.
Replacing URL Fetch APIs with a Python library
If you aren't already using a standard Python library to issue outgoing requests, choose a library and add it to your app's dependencies.
For example, to use the Requests library create a
requirements.txt
file in the same folder as yourapp.yaml
file and add the following line:requests==2.24.0
For compatibility with Python 2, we recommend you pin the
requests
library to version 2.24.0. When you deploy your app, App Engine will download all of the dependencies that are defined in therequirements.txt
file.For local development, we recommend that you install dependencies in a virtual environment such as venv.
Search your code for any use of the
google.appengine.api.urlfetch
module, and update the code to use your Python library.
Making simple HTTPS requests
The following example shows how to make a standard HTTPS request using the
requests
library:
Making asynchronous HTTPS requests
The following example shows how to make an asynchronous HTTPS request using the
requests
library:
Testing locally
If you updated any of your outbound requests, run your app in the local development server and confirm that the requests succeed.
Bypassing URL Fetch
To stop URL Fetch from handling requests when you deploy your app to App Engine:
In your
app.yaml
file, set theGAE_USE_SOCKETS_HTTPLIB
environment variable to any value. The value can be any value including an empty string. For example:env_variables: GAE_USE_SOCKETS_HTTPLIB : ''
If you enabled URL Fetch to handle requests sent from the
requests
library, you can remove the RequestsAppEngineAdapter
from your app.For example, remove
requests_toolbelt.adapters.appengine
from yourappengine_config.py
file andrequests_toolbelt.adapters.appengine.monkeypatch()
from your Python files.
Note that even if you bypass URL Fetch as described in the previous steps, your app can still use the URL Fetch API directly.
Deploying your app
Once you are ready to deploy your app, you should:
-
View the App Engine Quotas page in the Google Cloud console to confirm that your app isn't making Url Fetch API Calls.
If the app runs without errors, use traffic splitting to slowly ramp up traffic for your updated app. Monitor the app closely for any issues before routing more traffic to the updated app.