URL Fetch for legacy bundled services

This page describes how App Engine applications use the URL Fetch service to issue HTTP and HTTPS requests and receive responses. To see code samples demonstrating how to issue HTTP and HTTPS requests from your App Engine application, see Issuing HTTP(S) Requests. To view the contents of the urlfetch package, see the urlfetch package reference.

This API is supported for first-generation runtimes and can be used when upgrading to corresponding second-generation runtimes. If you are updating to the App Engine Go 1.12+ runtime, refer to the migration guide to learn about your migration options for legacy bundled services.

Requests

App Engine uses the URL Fetch service to issue outbound requests. In Go, the urlfetch package provides urlfetch.Transport, an implementation of the http.RoundTripper interface that makes requests using App Engine's infrastructure.

Request protocols

An application can fetch a URL using either HTTP or HTTPS. The protocol that should be used is inferred by looking at the protocol in the target URL.

The URL to be fetched can use any port number in the following ranges:

• 80-90
• 440-450
• 1024-65535.

If the port is not mentioned in the URL, the port is implied by the protocol. HTTP requests occur on port 80, and HTTPS requests occur on port 443.

Request methods

If you issue requests through the URL Fetch service, you can use any of the following HTTP methods:

• GET
• POST
• PUT
• HEAD
• DELETE
• PATCH

A request can include HTTP headers and, for POST, PUT, and PATCH requests, a payload.

Request proxying

Note that the URL Fetch service uses an HTTP/1.1 compliant proxy to fetch the result.

To prevent an application from causing an endless recursion of requests, a request handler is not allowed to fetch its own URL. It is still possible to cause an endless recursion with other means, so exercise caution if your application can be made to fetch requests for URLs supplied by the user.

Request headers

Your application can set HTTP headers for the outgoing request.

When sending an HTTP POST request, if a Content-Type header is not set explicitly, the header is set to x-www-form-urlencoded. This is the content type used by web forms.

For security reasons, the following headers cannot be modified by the application:

• Content-Length
• Host
• Vary
• Via
• X-Appengine-Inbound-Appid
• X-Forwarded-For
• X-ProxyUser-IP

These headers are set to accurate values by App Engine as appropriate. For example, App Engine calculates the Content-Length header from the request data and adds it to the request prior to sending.

The following headers indicate the application ID of the requesting app:

• User-Agent. This header can be modified, but App Engine will append an identifier string to allow servers to identify App Engine requests. The appended string has the format "AppEngine-Google; (+http://code.google.com/appengine; appid: APPID)", where APPID is your app's identifier.
• X-Appengine-Inbound-Appid. This header cannot be modified, and is added automatically if the request is sent via the URL Fetch service.

Request timeouts

You can set a deadline, or timeout, for a request. By default, the timeout for a request is 60 seconds. The maximum deadline is 60 seconds for HTTP(S) requests and 60 seconds for Task Queue and cron job requests.

Secure connections and HTTPS

Your application can fetch a URL securely by using HTTPS to connect to secure servers. Request and response data are transmitted over the network in encrypted form.

In the Go API, the URL Fetch proxy validates the host it is contacting by default. This behavior allows the API to detect man-in-the-middle attacks between App Engine and the remote host when using HTTPS.

Responses

If you use the URL Fetch API, note that the URL Fetch service returns all response data, including the response, code, headers, and body.

By default, if the URL Fetch service receives a response with a redirect code, the service will follow the redirect. The service will follow up to five redirect responses, then return the final resource.

Note: The URL Fetch API imposes response size limits. If the incoming response exceeds the maximum response size limit, the URL Fetch service raises an exception. See the Quotas and limits section for details.

You can instruct the URL Fetch API to truncate the response instead of raising an exception.

Using URL Fetch on the development server

When your application is running on the App Engine development server on your computer, calls to the URL Fetch service are handled locally. The development server fetches URLs by contacting remote hosts directly from your computer, using whatever network configuration your computer is using to access the Internet.

When testing the features of your application that fetch URLs, make sure that your computer can access the remote hosts.

Quotas and limits for URL Fetch

For information about URL Fetch service quotas, see Quotas. To see the current quota usage of your application, go to the Quota Details page in the Google Cloud console.

Go to the Quota Details page

In addition, the following limits apply to the use of the URL Fetch service:

Limit	Amount
Request size	10 megabytes
Request header size	16 KB (Note that this limits the maximum length of the URL that can be specified in the header)
Response size	32 megabytes
Maximum deadline (request handler)	60 seconds
Maximum deadline (Task Queue and cron job handler)	60 seconds

What's next

Run code samples and get guidance on how to issue requests from your application in Issuing HTTP(S) Requests.