Setting up Cloud CDN with a backend bucket

Cloud CDN leverages Google Cloud global external HTTP(S) load balancers to provide routing, health checking, and Anycast IP support. Because global external HTTP(S) load balancers can have multiple backend instance types— Compute Engine VM instances, Google Kubernetes Engine Pods, Cloud Storage buckets, or external origins outside of Google Cloud—you can choose which backends (origins) to enable Cloud CDN for.

This setup guide shows you how to create a simple external HTTP(S) load balancer with Cloud CDN enabled. The example uses the following resources:

  • The default Virtual Private Cloud (VPC) network
  • A default URL map
  • A reserved external IP address
  • A Cloud Storage bucket as the backend
  • A single load balancer backend bucket that acts as a wrapper around the Cloud Storage bucket

A backend bucket supports the following:

  • Cloud Storage buckets of any storage class, including multi-region buckets
  • Cloud CDN policies for caching content at Google's global edge

To learn how Cloud CDN works, see the Cloud CDN overview.

Load balancer backends

An external HTTP(S) load balancer uses a URL map to direct traffic from specified URLs to specified services. The following table summarizes the types of backends where you can host content and services.

Load balancer backend configuration Typical content type Backend types
Backend service Dynamic (such as data)
  • Unmanaged instance groups
  • Managed instance groups
  • Network endpoint groups internal to Google Cloud
  • Network endpoint groups external to Google Cloud
Backend bucket Static (such as images)
  • Cloud Storage buckets (discussed on this page)

Before you begin

  • If you are using the gcloud or gsutil utilities, see Quickstart: Using the gsutil tool to install them.
  • Set a default project.

    Console

    1. In the Google Cloud Console, go to the Home page.

      Go to the Google Cloud home page

    2. To the right of Google Cloud, select a project from the pull-down menu.

    gcloud or gsutil

     gcloud config set project PROJECT_ID
    

    or

     gsutil config set project PROJECT_ID
    
    • Replace PROJECT_ID with the project you will use for this guide.

Creating a Cloud Storage bucket

If you have an existing Cloud Storage bucket that isn't already assigned to a load balancer, you can skip to the next step.

When you create a Cloud Storage bucket to use as the backend for an external HTTP(S) load balancer with Cloud CDN, we recommend that you choose a multi-region bucket, which automatically replicates objects across multiple Google Cloud regions. This can improve the availability of your content and improve failure tolerance across your application.

Console

  1. In the Google Cloud Console, open the Cloud Storage browser.

    Open the Storage browser

  2. Click Create bucket.
  3. Specify values for the fields in the following table, leaving all others at their defaults.

    Property Value (type the value or select an option as specified)
    Name For each bucket, enter a globally unique name. If the name you enter is not unique, you see a message to try another name.
    Location type Multi-region
    Location Select a region, such as us (multiple regions in United States).
    Default storage class Standard
    Access control Uniform
  4. Click Create.

  5. Note the name of the newly-created Cloud Storage bucket for the next step.

gsutil

gsutil mb -p PROJECT_ID -c standard -l us-east1 -b on gs://BUCKET_NAME

Copying a graphic file into your Cloud Storage bucket

To enable you to test the setup, copy a graphic file from a public Cloud Storage bucket to your own Cloud Storage bucket.

  1. Run the following command in Cloud Shell. Replace BUCKET_NAME with your unique Cloud Storage bucket name:

    gsutil cp gs://gcp-external-http-lb-with-bucket/three-cats.jpg gs://BUCKET_NAME/never-fetch/
    
  2. In the Cloud Console, click Refresh to verify that the graphic file is copied.

Making your Cloud Storage bucket public

This example makes your Cloud Storage bucket publicly readable. This is the recommended approach for public content. With this setting, anyone on the internet can view and list your objects and their metadata, excluding ACLs. The recommended practice is to dedicate specific Cloud Storage buckets for public objects. For more information, see Recommended bucket architecture.

The following are alternatives to making your Cloud Storage buckets public:

The following procedure grants all users access to view objects in your Cloud Storage bucket, making the bucket publicly readable.

Console

  1. In the Google Cloud Console, open the Cloud Storage browser.

    Open the Storage browser

  2. Navigate to the bucket and click the Permissions tab.
  3. Click Add members.
  4. In New members, enter allUsers.
  5. For the role, select Cloud Storage > Storage Object Viewer.
  6. Click Save.

gsutil

gsutil iam ch allUsers:objectViewer gs://BUCKET_NAME

Reserving an external IP address

Now that your Cloud Storage bucket is up and running, set up a global static external IP address that your customers use to reach your load balancer.

This step is optional, but recommended, because a static external IP address provides a single address to point your domain at.

Console

  1. In the Google Cloud Console, go to the External IP addresses page.

    Go to the External IP addresses page

  2. To reserve an IPv4 address, click Reserve static address.
  3. Assign a Name of example-ip.
  4. Set the Network Service Tier to Premium.
  5. Set the IP version to IPv4.
  6. Set the Type to Global.
  7. Click Reserve.

gcloud

gcloud compute addresses create example-ip \
    --network-tier=PREMIUM \
    --ip-version=IPV4 \
    --global

Note the IPv4 address that was reserved:

gcloud compute addresses describe example-ip \
    --format="get(address)" \
    --global

Creating the external HTTP(S) load balancer

In this procedure, you create the load balancer's backend bucket, which serves as a wrapper for your Cloud Storage bucket. When creating or editing a backend bucket, you can enable Cloud CDN.

Console

Start the external HTTP(S) load balancer configuration process

  1. In the Google Cloud Console, go to the Load balancing page.

    Go to the Load balancing page

  2. Under HTTP(S) load balancing, click Start configuration.
  3. Select From Internet to my VMs, and then click Continue.
  4. Set Name to http-lb, and then go to the next step.

Configure the backend and enable Cloud CDN

Create the load balancer's backend bucket, which serves as a wrapper for your Cloud Storage bucket. When creating or editing a backend bucket, you can enable Cloud CDN.

  1. Click Backend configuration.
  2. Under Backend services & backend buckets, click Create or select backend services & backend buckets, and then click Backend buckets > Create a backend bucket.
  3. Set the Name to cat-backend-bucket. This name doesn't need to be globally unique and can be different from the name of the actual Cloud Storage bucket.
  4. Under Cloud Storage bucket, click Browse.
  5. Select the Cloud Storage globally unique BUCKET_NAME that you created, and then click Select.
  6. Click Enable Cloud CDN.

  7. Click Create.

Configure host rules and path matchers

Host rules and path matchers are configuration components of an external HTTP(S) load balancer's URL map.

In Host and path rules, you can retain the default settings.

For a customized setup example, see Adding backend buckets to load balancers.

To learn more about host rules and path matchers, see URL maps overview.

Configure the frontend

  1. Click Frontend configuration.
  2. Verify that the options are configured with these values.

    Property Value (type a value or select an option as specified)
    Protocol HTTP
    Network Service Tier Premium
    IP version IPv4
    IP address example-ip
    Port 80

    If you want to create an HTTPS load balancer instead of an HTTP load balancer, you must have an SSL certificate (gcloud compute ssl-certificates list), and you must fill in the fields as follows.

    Property Value (type a value or select an option as specified)
    Protocol HTTPS
    Network Service Tier Premium
    IP version IPv4
    IP address example-ip
    Port 443
    Certificate Select a certificate or Create a new certificate
  3. Click Done.

Review the configuration

  1. Click Review and finalize.
  2. Review the Backend buckets, Host and path rules, and Frontend sections.
  3. Click Create.
  4. Wait for the load balancer to be created.
  5. Click the name of the load balancer (http-lb).
  6. Note the IP address of the load balancer for the next task. It's referred to as IP_ADDRESS.

gcloud

Configure the backend

gcloud compute backend-buckets create cat-backend-bucket \
    --gcs-bucket-name=BUCKET_NAME \
    --enable-cdn

Configure the URL map

gcloud compute url-maps create http-lb \
    --default-backend-bucket=cat-backend-bucket

Configure the target proxy

gcloud compute target-http-proxies create http-lb-proxy \
    --url-map=http-lb

Configure the forwarding rule

gcloud compute forwarding-rules create http-lb-forwarding-rule \
    --address=example-ip \
    --global \
    --target-http-proxy=http-lb-proxy \
    --ports=80

Sending traffic to your backend bucket

After creating the global forwarding rule, it can take several minutes for your configuration to propagate worldwide. After several minutes have passed, you can start sending traffic to the load balancer's IP address.

Console

  1. In the Google Cloud Console, go to the Load balancing page.

    Go to the Load balancing page

  2. Click http-lb to expand the load balancer that you just created.

    In the Backend section, confirm that the backend bucket is healthy. There should be a green checkmark next to your backend bucket. If you see otherwise, first try reloading the page. It can take a few moments for the Cloud Console to indicate that backends are healthy.

  3. After the Cloud Console shows that the backend bucket is healthy, you can test your load balancer using a web browser by going to http://IP_ADDRESS/never-fetch/three-cats.jpg. Replace IP_ADDRESS with the load balancer's IP address. Your browser should render a page with content showing the graphic file.

gcloud

Use the curl command to test the response from the URL. Replace IP_ADDRESS with the load balancer's IPv4 address.

Note the IPv4 address that was reserved:

gcloud compute addresses describe example-ip \
    --format="get(address)" \
    --global

Send a curl request:

curl http://IP_ADDRESS/never-fetch/three-cats.jpg

Verifying that Cloud CDN is working

If you reload the http://ip-address/never-fetch/three-cats.jpg page several times in quick succession, there should be several cache hits.

The following log entry shows a cache hit. You can view cache hits by opening the Logs Viewer in the Google Cloud Console and filtering by the forwarding rule name.

Open the Logs Viewer

Logs Viewer

{
    insertId: "1oek5rg3l3fxj7"
    jsonPayload: {
        @type: "type.googleapis.com/google.cloud.loadbalancing.type.LoadBalancerLogEntry"
        cacheId: "SFO-fbae48ad"
        statusDetails: "response_from_cache"
    }
    httpRequest: {
        requestMethod: "GET"
        requestUrl: "http://LOAD_BALANCER_IP_ADDRESS/never-fetch/three-cats.jpg"
        requestSize: "577"
        status: 254
        responseSize: "157"
        userAgent: "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36"
        remoteIp: "CLIENT_IP_ADDRESS"
        cacheHit: true
        cacheLookup: true
    }
    resource: {
        type: "http_load_balancer"
        labels: {
            zone: "global"
            url_map_name: "URL_MAP_NAME"
            forwarding_rule_name: "FORWARDING_RULE_NAME"
            target_proxy_name: "TARGET_PROXY_NAME"
            backend_service_name: ""
            project_id: "PROJECT_ID"
        }
    }
    timestamp: "2020-06-08T23:41:25.078651Z"
    severity: "INFO"
    logName: "projects/PROJECT_ID/logs/requests"
    trace: "projects/PROJECT_ID/traces/241d69833e64b3bf83fabac8c873d992"
    receiveTimestamp: "2020-06-08T23:41:25.588272510Z"
    spanId: "7b6537d3672e08e1"
}

Console

  1. In the Google Cloud Console, go to the Load balancing page.

    Go to the Load balancing page

  2. Click http-lb to expand the load balancer that you just created.

    In the Backend section, confirm that the backend bucket is healthy. There should be a green checkmark next to your backend bucket. If you see otherwise, first try reloading the page. It can take a few moments for the Cloud Console to indicate that backends are healthy.

  3. After the Cloud Console shows that the backend bucket is healthy, you can test your load balancer using a web browser by going to http://IP_ADDRESS/never-fetch/three-cats.jpg. Replace IP_ADDRESS with the load balancer's IP address. Your browser should render a page with content showing the graphic file.

gcloud

Use the curl command to test the response from the URL. Replace IP_ADDRESS with the load balancer's IPv4 address.

Note the IPv4 address that was reserved:

gcloud compute addresses describe example-ip \
    --format="get(address)" \
    --global

Send a curl request:

curl http://IP_ADDRESS/never-fetch/three-cats.jpg

Content is fetched from Cloud Storage, cached by Cloud CDN, and then validated and re-fetched when it expires or is otherwise evicted from the cache. Content that needs to be refreshed earlier than the TTL can be invalidated and re-fetched from Cloud Storage.

Using cache modes and TTLs

To automatically cache static responses from your origin, you can use the CACHE_ALL_STATIC cache mode setting (Beta).

To control cacheability for each response by using HTTP cache directives, set the cache mode to use origin headers (USE_ORIGIN_HEADERS). For information about the cache directives that Cloud CDN understands and what's not cached by Cloud CDN, see Cacheable content and Non-cacheable content.

If your origin isn't serving any per-user dynamic content, you might want to cache all responses from the origin. To do this, use the FORCE_CACHE_ALL mode. This mode caches all responses, regardless of content type or cache directives.

If you don't explicitly select a cache mode when you enable Cloud CDN on a backend, the following defaults apply:

  • Beta. The API and the gcloud command-line tool default to USE_ORIGIN_HEADERS if you enable Cloud CDN but don't specify a cache mode. The Cloud Console (when supported) defaults to CACHE_ALL_STATIC when you enable Cloud CDN on a backend.

  • GA. All newly Cloud CDN-enabled backends default to CACHE_ALL_STATIC, regardless of your configuration method (Cloud Console, API, or the gcloud tool).

New backend buckets with Cloud CDN enabled automatically default to the CACHE_ALL_STATIC mode.

  1. Enable Cloud CDN on your backend bucket.
  2. Set the cache mode to automatically cache all static content.
  3. Set an appropriate default TTL (for example, one day) to have Cloud CDN cache it.

Disabling Cloud CDN

Console

Disable Cloud CDN for a single backend bucket

  1. In the Google Cloud Console, go to the Cloud CDN page.

    Go to the Cloud CDN page

  2. On the right side of the origin row, click Menu and then select Edit.
  3. Clear the checkboxes of any backend buckets that you want to stop from using Cloud CDN.
  4. Click Update.

Remove Cloud CDN for all backend buckets for an origin

  1. In the Cloud Console, go to the Cloud CDN page.

    Go to the Cloud CDN page

  2. On the right side of the origin row, click Menu and then select Remove.
  3. To confirm, click Remove.

gcloud

gcloud compute backend-buckets update BACKEND_BUCKET_NAME \
    --no-enable-cdn

Disabling Cloud CDN does not invalidate or purge caches. If you turn Cloud CDN off and back on again, most or all of your cached content might still be cached. To prevent content from being used by the caches, you must invalidate that content.

What's next