HTTP(S) Load Balancing with Google Cloud Storage

This guide demonstrates how to distribute traffic to different instances and Cloud Storage buckets based on the incoming URL. You will configure the load balancing service to send requests for static content (CSS, images) to a Cloud Storage bucket, and requests for all other content to virtual machine instances.

Using a Cloud Storage bucket as a load balancer backend

This guide shows how to modify an HTTP(S) load balancing configuration to route requests for static content to a Cloud Storage bucket and requests for dynamic content to instances. Specifically, requests to URL paths that begin with /static will be sent to your storage bucket, and all other requests will be sent to your instances.

Distributing traffic to Instances and Cloud Storage buckets with HTTP(S) LB (click to enlarge)
Distributing traffic to Instances and Cloud Storage buckets with HTTP(S) LB (click to enlarge)

Before you begin

  1. Complete the Before you begin portion of the Cloud Storage Quickstart: Using the gsutil Tool. This installs the Cloud SDK, which includes the gcloud and gsutil command-line tools you need.
  2. Set a default project.

    gcloud config set project [PROJECT_ID]
    

    or

    gsutil config set project [PROJECT_ID]
    
    • [PROJECT_ID] - the project you will use for this guide
  3. Complete the guide for content-based load balancing. The material here builds on the resources you'll build using that guide.

Configuring a Cloud Storage bucket and the load balancing service

The following instructions assume you are uploading and setting permissions for a single object. If you would like to use wildcards (globs) to upload and set permissions for multiple objects, see Wildcard Names.

Configuring a bucket

Create a bucket and upload an object to it. Bucket names have to be globally unique.

  1. Create a bucket.

    gsutil mb gs://[EXAMPLE_BUCKET]/
    
    • [EXAMPLE_BUCKET] - a globally unique bucket name
  2. Upload an object from your local directory. If the file is in another directory, provide the complete path.

    gsutil cp [OBJECT_NAME] gs://[EXAMPLE_BUCKET]/static/[OBJECT_NAME]
    
    • [EXAMPLE_BUCKET] - the bucket you already created
    • [OBJECT_NAME] - the filename of an object to upload
  3. Make the object publicly readable so it can be served through load balancing.

    gsutil acl ch -u AllUsers:R gs://[EXAMPLE_BUCKET]/static/[OBJECT_NAME]
    
    • [EXAMPLE_BUCKET] - the bucket you already created
    • [OBJECT_NAME] - the filename of the object you uploaded

Creating a backend bucket for your Cloud Storage bucket

Instead of creating a backend service for static content like you did in the content-based load balancing example, create a new backend bucket using the backend-buckets command that points to the Cloud Storage bucket you created above. Replace [EXAMPLE_BUCKET] with the name of your bucket.

gcloud beta compute backend-buckets create static-bucket \
    --description "A Bucket" \
    --gcs-bucket-name [EXAMPLE_BUCKET]
  • [EXAMPLE_BUCKET] - the bucket you created

Creating a URL map

  1. Take a look at the previous URL map you created.

    gcloud compute url-maps list
    
  2. Create a new URL map where the default path points to the same default service you used in the previous URL map (web-map-backend-service for the purposes of this guide).

    gcloud compute url-maps create www-map2 \
        --default-service web-map-backend-service
    
  3. Add a rule that maps /static/* to the backend bucket created above. This routes all requests that have /static/* to your Cloud Storage bucket. Unlike your www-map URL map, this URL map does not have a separate path rule for /video.

    gcloud beta compute url-maps add-path-matcher www-map2 \
        --default-service web-map-backend-service \
        --path-matcher-name pathmap \
        --backend-bucket-path-rules=/static/*=static-bucket
    

Updating your HTTP or HTTPS proxy to use your new URL map

When you ran through the content-based load balancing example, you had the choice of proxying HTTP or HTTPS traffic. In each of the steps in this section, use the target-http-proxies or target-https-proxies, depending on which one you used in the prior example.

  1. Locate the proxy you set up in the content-based load balancing example.

    gcloud compute target-http-proxies list
    

    or

    gcloud compute target-https-proxies list
    
  2. Update your proxy to use the www-map2 URL map.

    gcloud compute target-http-proxies update [PROXY_NAME] \
        --url-map www-map2
    

    or

    gcloud compute target-https-proxies update [PROXY_NAME] \
        --url-map www-map2
    

Fetching resources from your Cloud Storage bucket

  1. Find the IP address of your global forwarding rule.

    gcloud compute forwarding-rules list
    
  2. The address you want is the address of the rule you created when you ran through the content-based load balancing example. Test that your Cloud Storage bucket is receiving traffic using curl or by visiting the URL in your browser.

    curl http://[IP_ADDRESS]/static/[OBJECT_NAME]
    
    • [IP_ADDRESS] - the IP address of your global forwarding rule
    • [OBJECT_NAME] - use the name of the example object you uploaded

This should respond with [OBJECT_NAME] from your [EXAMPLE_BUCKET] Cloud Storage bucket.

If you store an object in the bucket with name of static/path/to/[OBJECT_NAME], you can retrieve it with a URL of http://[IP_ADDRESS]/static/path/to/[OBJECT_NAME].

Using Cloud CDN with Cloud Storage buckets

Google Cloud CDN allows you to cache your content at the edge of Google's network, lowering latency and improving performance for your users. You can use Cloud CDN when you give users access to Cloud Storage buckets via HTTP(S) load balancing.

Distributing traffic to Instances and Cloud Storage buckets with Cloud CDN and HTTP(S) LB (click to enlarge)
Distributing traffic to Instances and Cloud Storage buckets with Cloud CDN and HTTP(S) LB (click to enlarge)

Storage buckets can be connected to a load balancer via a backend bucket load balancer resource. A backend bucket resource functions in much the same way as a backend service resource, except that it points to a storage bucket intead of an instance group.

Only content that is marked as cachable can be cached by Cloud CDN. The Cache-control section of Working With Object Metadata explains how to mark your storage bucket objects as cachable. Content can be served from cache if all of the Cloud CDN Cacheability requirements are fulfilled.

You enable or disable caching for each bucket individually. You can enable Cloud CDN when creating a backend bucket or when updating an existing one. You can also turn off Cloud CDN caching by updating an existing backend bucket.

Activating Cloud CDN when creating a backend bucket

Add the --enable-cdn parameter to the normal backend-buckets create command to turn on Cloud CDN for that bucket.

Replace [EXAMPLE_BUCKET] with the name of your Cloud Storage bucket.

gcloud beta compute backend-buckets create static-bucket \
    --description "A Bucket" \
    --gcs-bucket-name [EXAMPLE_BUCKET] \
    --enable-cdn

* `[EXAMPLE_BUCKET]` - the bucket you created

Activating Cloud CDN for an existing backend bucket

Add the --enable-cdn parameter to the normal backend-buckets update command to turn on Cloud CDN for that bucket.

gcloud beta compute backend-buckets update static-bucket \
    --gcs-bucket-name [EXAMPLE_BUCKET] \
    --enable-cdn

* `[EXAMPLE_BUCKET]` - the bucket you created

Deactivating Cloud CDN for an existing backend bucket

Add the --no-enable-cdn parameter to the normal backend-buckets update command to turn off Cloud CDN for that bucket.

This command deactivates CDN for a particular backend bucket. Caches will no longer serve content cached from that bucket. Cached content is not actively purged. Instead, it is allowed to expire normally.

gcloud beta compute backend-buckets update static-bucket \
    --gcs-bucket-name [EXAMPLE_BUCKET] \
    --no-enable-cdn

* `[EXAMPLE_BUCKET]` - the bucket you created

Invalidating cached content

Sometimes you need to stop caches from serving objects they have cached. Cache invalidation causes the caching system to stop serving the invalidated content. See Invalidating Cached Content for instructions on invalidating one or more files in the caches.

Cleaning up

Revert your proxy back to its previous state, then clean up your backend bucket.

  1. Update your proxy to point back to the initial URL map.

    gcloud compute target-http-proxies update [PROXY_NAME] \
        --url-map www-map
    

    or

    gcloud compute target-https-proxies update [PROXY_NAME] \
        --url-map www-map
    
    • [PROXY_NAME] - the name of the proxy you updated before
  2. Delete the URL map you created.

    gcloud compute url-maps delete www-map2
    
  3. Delete your backend bucket. This will not delete your Cloud Storage bucket.

    gcloud beta compute backend-buckets delete static-bucket
    

Troubleshooting

When trying to view an object in my Cloud Storage bucket, I get a permissions error.

In order to serve objects through load balancing, the Cloud Storage objects must be publicly accessible. Make sure to update the permissions of the objects being served so they are publicly readable.

The URL I enter doesn’t serve the object I expect.

The Cloud Storage object to serve is determined based on your URL map and the URL that you request. If the request path maps to a backend bucket in your URL map, the Cloud Storage object will be determined by appending the full request path onto the Cloud Storage bucket that the URL map specifies.

For example, if you map /static/* to gs://[EXAMPLE_BUCKET], the request to https://<GCLB IP or Host>/static/path/to/content.jpg will try to serve gs://[EXAMPLE_BUCKET]/static/path/to/content.jpg. If that object doesn’t exist, a 404 will be served.

Next Steps

You can also configure a custom index page and a custom error page that will be served if the requested object doesn’t exist. This can be done by adding a Website Configuration to your Cloud Storage bucket. With a Website Configuration, you could serve a static webpage directly out of a Cloud Storage bucket from your own domain.

Send feedback about...

Compute Engine Documentation