Troubleshooting HTTP(S) Load Balancing

This guide describes how to troubleshoot configuration issues for a Google Cloud external HTTP(S) load balancer.

Overview

The types of issues discussed in this guide include the following:

General connectivity issues
Troubleshooting issues with HTTP/2 to the backends
Custom origin and internet NEG issues (beta)

Before you begin

Before investigating issues, familiarize yourself with the following pages.

For general connectivity:

For custom origins and internet NEGs:

Troubleshooting general connectivity issues

Load balanced traffic does not have the source address of the original client

Traffic from the load balancer to your instances has an IP address in the ranges of 130.211.0.0/22 and 35.191.0.0/16. When viewing logs on your load balanced instances, you will not see the source address of the original client. Instead, you will see source addresses from this range.

Getting a permission error when trying to view an object in my Cloud Storage bucket

In order to serve objects through load balancing, the Cloud Storage objects must be publicly accessible. Make sure to update the permissions of the objects being served so they are publicly readable.

URL doesn't serve expected Cloud Storage object

The Cloud Storage object to serve is determined based on your URL map and the URL that you request. If the request path maps to a backend bucket in your URL map, the Cloud Storage object is determined by appending the full request path onto the Cloud Storage bucket that the URL map specifies.

For example, if you map /static/* to gs://[EXAMPLE_BUCKET], the request to https://<GCLB IP or Host>/static/path/to/content.jpg will try to serve gs://[EXAMPLE_BUCKET]/static/path/to/content.jpg. If that object doesn't exist, you will get the following error message instead of the object:


NoSuchKey
The specified key does not exist.

Compression isn't working

HTTP(S) Load Balancing does not compress or decompress responses itself, but it can serve responses generated by your backend service that are compressed by using tools such as gzip or DEFLATE.

If responses served by HTTP(S) Load Balancing are not compressed but should be, check to be sure that the web server software running on your instances is configured to compress responses. By default, some web server software automatically disables compression for requests that include a Via header, which indicates that the request was forwarded by a proxy. Because it is a proxy, HTTP(S) Load Balancing adds a Via header to each request as required by the HTTP specification. To enable compression, you may have to override your web server's default configuration to tell it to compress responses even if the request had a Via header.

If you are using the nginx web server software, modify the nginx.conf configuration file to enable compression. The location of this file depends on where nginx is installed. In many Linux distributions, the file is stored at /etc/nginx/nginx.conf. To allow nginx compression to work with HTTP(S) Load Balancing, add the following two lines to the http section of nginx.conf:

gzip_proxied any;
gzip_vary on;

The first line enables compression even for requests forwarded by a proxy like HTTP(S) Load Balancing. The second line adds a Vary: Accept-Encoding header to responses. Vary: Accept-Encoding notifies caching proxies such as Cloud CDN that they should maintain separate cache entries for compressed and non-compressed variants of compressible resources.

After modifying nginx.conf, you need to restart nginx before it will use the new configuration. In many Linux distributions, nginx can be restarted by running sudo service nginx restart or /etc/init.d/nginx restart.

Troubleshooting issues with HTTP/2 to the backends

Invalid value for field `resource.loadBalancingScheme`: 'EXTERNAL'

Backend-service-based Network Load Balancing is not yet supported.

This could happen if you create a backend service without selecting the global option. When you issue a gcloud command as follows, you are prompted to designate a region or designate the load balancer as global:

gcloud beta compute backend-services create service-test \
    --health-checks=hc-test \
    --project=test1 \
    --protocol=http2

For the following backend service:

- [service-test] choose a region or global:
[1] global
[2] region: [REGION_A_NAME]
[3] region: [REGION_B_NAME]
....
Please enter your numeric choice:

For the HTTP(S) load balancer, the backend services must be global, so you must choose option 1 or issue the gcloud command with the --global option:

gcloud beta compute backend-services create service-test \
    --health-checks=hc-test \
    --project=test \
    --protocol=http2 \
    --global

Unexplained 502 errors

Make sure that your backend instance is healthy and supports HTTP/2 protocol. You can verify this by testing connectivity to the backend instance using HTTP/2. Ensure that the VM uses HTTP/2 spec-compliant cipher suites. For example, certain TLS 1.2 cipher suites are disallowed by HTTP/2. Refer to the TLS 1.2 Cipher Suite Black List.

After you verify that the VM uses the HTTP/2 protocol, make sure your firewall setup allows the health checker and load balancer to pass through.

If there are no problems with the firewall setup, ensure that the load balancer is configured to talk to the correct port on the VM.

Troubleshooting custom origin and internet NEG issues

Traffic does not reach the endpoints

After you configure a service, the new endpoint becomes reachable through the external HTTP(S) load balancer when:

The endpoint is attached to the internet NEG.
The associated FQDN can be DNS resolved successfully (if you are using FQDN endpoint type).
The endpoint is accessible over the internet.

If traffic cannot reach the endpoint, which results in a 502 error code for HTTP(s), check the following:

Query the _cloud-eoips.googleusercontent.com DNS TXT record using a tool like dig or nslookup. Note the CIDRs (following ip4:) and ensure these ranges are allowed by your firewall or cloud Access Control List (ACL).

After configuring a custom origin, requests to custom origin failed with a 5xx error

Check Logging.
Verify that the network endpoint group is configured with the correct IP:Port or FQDN:Port for your custom origin.
If you are using FQDN, make sure that it is resolvable through Google Public DNS. You can verify that the FQDN is resolvable through Google Public DNS using these steps or the web interface directly.
If you are accessing the HTTP(S) load balancer on its external IP only, and your origin web-server is expecting a hostname, ensure that you have sending a valid HTTP Host header to your backend by configuring a user-defined request header.
If communicating to a backend over HTTPS or HTTP2 (as set in the protocol field of the backend service) configured as an INTERNET_FQDN_PORT custom origin endpoint, ensure that your origin is presenting a valid TLS (SSL) certificate and the configured FQDN matches a SAN (Subject Alternative Name) in the certificates' list of SANs. A valid certificate is defined as one signed by a public Certificate Authority and that has not expired.
When using INTERNET_FQDN_PORT custom origin endpoints, self-signed certificates are not accepted by the HTTPS load balancer, and are rejected.
When using HTTPS or HTTP/2 with INTERNET_IP_PORT type endpoints, no SSL certificate validation/SAN check is performed. This means one can use self signed certificates. When using SSL, our recommendation is to use INTERNET_FQDN_PORT endpoints to make sure server certificates and SANs can be validated.

Responses from my custom origin are not cached by Cloud CDN

Ensure that:

You have enabled Cloud CDN on the backend service containing the NEG that points to your custom origin by setting enableCDN to true.
Responses served by your custom origin meet Cloud CDN caching requirements. For example, you are sending Cache-Control: public, max-age=3600 response headers from the origin.