Setting up Cloud CDN with a backend bucket

Cloud CDN leverages Google Cloud global external HTTP(S) load balancers to provide routing, health checking, and Anycast IP support. Because global external HTTP(S) load balancers can have multiple backend instance types— Compute Engine VM instances, Google Kubernetes Engine Pods, Cloud Storage buckets, or external origins outside of Google Cloud—you can choose which backends (origins) to enable Cloud CDN for.

This setup guide shows you how to create a simple external HTTP(S) load balancer with Cloud CDN enabled. The example uses the following resources:

• The default Virtual Private Cloud (VPC) network
• A default URL map
• A reserved external IP address
• A Cloud Storage bucket as the backend
• A single load balancer backend bucket that acts as a wrapper around the Cloud Storage bucket

A backend bucket supports the following:

• Cloud Storage buckets of any storage class, including multi-region buckets
• Cloud CDN policies for caching content at Google's global edge

To learn how Cloud CDN works, see the Cloud CDN overview.

Load balancer backends

An external HTTP(S) load balancer uses a URL map to direct traffic from specified URLs to specified services. The following table summarizes the types of backends where you can host content and services.

Load balancer backend configuration	Typical content type	Backend types
Backend service	Dynamic (such as data)	Unmanaged instance groups Managed instance groups Network endpoint groups internal to Google Cloud Network endpoint groups external to Google Cloud
Backend bucket	Static (such as images)	Cloud Storage buckets (discussed on this page)

Before you begin

• If you are using the gcloud or gsutil utilities, see Quickstart: Using the gsutil tool to install them.

• Set a default project.

  Console

  1. In the Google Cloud Console, go to the Home page.

     Go to the Google Cloud home page

  2. To the right of Google Cloud, select a project from the pull-down menu.

  gcloud or gsutil

   gcloud config set project PROJECT_ID
  

  or

   gsutil config set project PROJECT_ID
  

  • Replace PROJECT_ID with the project you will use for this guide.

Creating a Cloud Storage bucket

If you have an existing Cloud Storage bucket that isn't already assigned to a load balancer, you can skip to the next step.

When you create a Cloud Storage bucket to use as the backend for an external HTTP(S) load balancer with Cloud CDN, we recommend that you choose a multi-region bucket, which automatically replicates objects across multiple Google Cloud regions. This can improve the availability of your content and improve failure tolerance across your application.

Console

1. In the Google Cloud Console, open the Cloud Storage browser.

   Open the Storage browser

2. Click Create bucket.

3. Specify values for the fields in the following table, leaving all others at their defaults.

   Property	Value (type the value or select an option as specified)
   Name	For each bucket, enter a globally unique name. If the name you enter is not unique, you see a message to try another name.
   Location type	Multi-region
   Location	Select a region, such as us (multiple regions in United States).
   Default storage class	Standard
   Access control	Uniform

4. Click Create.

5. Note the name of the newly-created Cloud Storage bucket for the next step.

gsutil

gsutil mb -p PROJECT_ID -c standard -l us-east1 -b on gs://BUCKET_NAME

Copying a graphic file into your Cloud Storage bucket

To enable you to test the setup, copy a graphic file from a public Cloud Storage bucket to your own Cloud Storage bucket.

1. Run the following command in Cloud Shell. Replace BUCKET_NAME with your unique Cloud Storage bucket name:

   gsutil cp gs://gcp-external-http-lb-with-bucket/three-cats.jpg gs://BUCKET_NAME/never-fetch/
   

2. In the Cloud Console, click Refresh to verify that the graphic file is copied.

Making your Cloud Storage bucket public

This example makes your Cloud Storage bucket publicly readable. This is the recommended approach for public content. With this setting, anyone on the internet can view and list your objects and their metadata, excluding ACLs. The recommended practice is to dedicate specific Cloud Storage buckets for public objects. For more information, see Recommended bucket architecture.

Important: When objects are served from a public Cloud Storage bucket, by default they have a Cache-Control: public, max-age=3600 response header applied. This allows the objects to be cached when Cloud CDN is enabled.

The following are alternatives to making your Cloud Storage buckets public:

• Make the individual Cloud Storage bucket objects publicly readable. We don't recommend this approach.

• Use signed URLs.

The following procedure grants all users access to view objects in your Cloud Storage bucket, making the bucket publicly readable.

Console

1. In the Google Cloud Console, open the Cloud Storage browser.

   Open the Storage browser

2. Navigate to the bucket and click the Permissions tab.
3. Click Add members.
4. In New members, enter allUsers.
5. For the role, select Cloud Storage > Storage Object Viewer.
6. Click Save.

gsutil

gsutil iam ch allUsers:objectViewer gs://BUCKET_NAME

Reserving an external IP address

Now that your Cloud Storage bucket is up and running, set up a global static external IP address that your customers use to reach your load balancer.

This step is optional, but recommended, because a static external IP address provides a single address to point your domain at.

Note: You can skip this step and have Google Cloud associate an ephemeral IP address with your load balancer's forwarding rule. An ephemeral IP address remains constant while the forwarding rule exists. If you need to delete the forwarding rule and re-add it, the forwarding rule might receive a new IP address.

Console

1. In the Google Cloud Console, go to the External IP addresses page.

   Go to the External IP addresses page

2. To reserve an IPv4 address, click Reserve static address.
3. Assign a Name of example-ip.
4. Set the Network Service Tier to Premium.
5. Set the IP version to IPv4.
6. Set the Type to Global.
7. Click Reserve.

gcloud

gcloud compute addresses create example-ip \
    --network-tier=PREMIUM \
    --ip-version=IPV4 \
    --global

Note the IPv4 address that was reserved:

gcloud compute addresses describe example-ip \
    --format="get(address)" \
    --global

Creating the external HTTP(S) load balancer

In this procedure, you create the load balancer's backend bucket, which serves as a wrapper for your Cloud Storage bucket. When creating or editing a backend bucket, you can enable Cloud CDN.

Console

Start the external HTTP(S) load balancer configuration process

1. In the Google Cloud Console, go to the Load balancing page.

   Go to the Load balancing page

2. Under HTTP(S) load balancing, click Start configuration.
3. Select From Internet to my VMs, and then click Continue.
4. Set Name to http-lb, and then go to the next step.

Configure the backend and enable Cloud CDN

Create the load balancer's backend bucket, which serves as a wrapper for your Cloud Storage bucket. When creating or editing a backend bucket, you can enable Cloud CDN.

1. Click Backend configuration.
2. Under Backend services & backend buckets, click Create or select backend services & backend buckets, and then click Backend buckets > Create a backend bucket.
3. Set the Name to cat-backend-bucket. This name doesn't need to be globally unique and can be different from the name of the actual Cloud Storage bucket.
4. Under Cloud Storage bucket, click Browse.
5. Select the Cloud Storage globally unique BUCKET_NAME that you created, and then click Select.

6. Click Enable Cloud CDN.

   Important: Make sure that you select Enable Cloud CDN at this step. Forgetting to do this causes objects to be served directly from the Cloud Storage bucket and not from Google's global CDN caches.

7. Click Create.

Configure host rules and path matchers

Host rules and path matchers are configuration components of an external HTTP(S) load balancer's URL map.

In Host and path rules, you can retain the default settings.

For a customized setup example, see Adding backend buckets to load balancers.

To learn more about host rules and path matchers, see URL maps overview.

Configure the frontend

1. Click Frontend configuration.

2. Verify that the options are configured with these values.

   Property	Value (type a value or select an option as specified)
   Protocol	HTTP
   Network Service Tier	Premium
   IP version	IPv4
   IP address	example-ip
   Port	80

   If you want to create an HTTPS load balancer instead of an HTTP load balancer, you must have an SSL certificate (gcloud compute ssl-certificates list), and you must fill in the fields as follows.

   Property	Value (type a value or select an option as specified)
   Protocol	HTTPS
   Network Service Tier	Premium
   IP version	IPv4
   IP address	example-ip
   Port	443
   Certificate	Select a certificate or Create a new certificate

3. Click Done.

Review the configuration

1. Click Review and finalize.
2. Review the Backend buckets, Host and path rules, and Frontend sections.
3. Click Create.
4. Wait for the load balancer to be created.
5. Click the name of the load balancer (http-lb).
6. Note the IP address of the load balancer for the next task. It's referred to as IP_ADDRESS.

gcloud

Configure the backend

gcloud compute backend-buckets create cat-backend-bucket \
    --gcs-bucket-name=BUCKET_NAME \
    --enable-cdn

Configure the URL map

gcloud compute url-maps create http-lb \
    --default-backend-bucket=cat-backend-bucket

Configure the target proxy

gcloud compute target-http-proxies create http-lb-proxy \
    --url-map=http-lb

Configure the forwarding rule

gcloud compute forwarding-rules create http-lb-forwarding-rule \
    --address=example-ip \
    --global \
    --target-http-proxy=http-lb-proxy \
    --ports=80

Sending traffic to your backend bucket

After creating the global forwarding rule, it can take several minutes for your configuration to propagate worldwide. After several minutes have passed, you can start sending traffic to the load balancer's IP address.

Console

1. In the Google Cloud Console, go to the Load balancing page.

   Go to the Load balancing page

2. Click http-lb to expand the load balancer that you just created.

   In the Backend section, confirm that the backend bucket is healthy. There should be a green checkmark next to your backend bucket. If you see otherwise, first try reloading the page. It can take a few moments for the Cloud Console to indicate that backends are healthy.

3. After the Cloud Console shows that the backend bucket is healthy, you can test your load balancer using a web browser by going to http://IP_ADDRESS/never-fetch/three-cats.jpg. Replace IP_ADDRESS with the load balancer's IP address. Your browser should render a page with content showing the graphic file.

gcloud

Use the curl command to test the response from the URL. Replace IP_ADDRESS with the load balancer's IPv4 address.

Note the IPv4 address that was reserved:

gcloud compute addresses describe example-ip \
    --format="get(address)" \
    --global

Send a curl request:

curl http://IP_ADDRESS/never-fetch/three-cats.jpg

Verifying that Cloud CDN is working

If you reload the http://ip-address/never-fetch/three-cats.jpg page several times in quick succession, there should be several cache hits.

The following log entry shows a cache hit. You can view cache hits by opening the Logs Viewer in the Google Cloud Console and filtering by the forwarding rule name.

Open the Logs Viewer

Note: If you don't specify a forwarding rule name when you set up your load balancer, the forwarding rule name is the load balancer name plus -forwarding-rule. In this case, http-lb-forwarding-rule.

Logs Viewer

{
    insertId: "1oek5rg3l3fxj7"
    jsonPayload: {
        @type: "type.googleapis.com/google.cloud.loadbalancing.type.LoadBalancerLogEntry"
        cacheId: "SFO-fbae48ad"
        statusDetails: "response_from_cache"
    }
    httpRequest: {
        requestMethod: "GET"
        requestUrl: "http://LOAD_BALANCER_IP_ADDRESS/never-fetch/three-cats.jpg"
        requestSize: "577"
        status: 254
        responseSize: "157"
        userAgent: "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36"
        remoteIp: "CLIENT_IP_ADDRESS"
        cacheHit: true
        cacheLookup: true
    }
    resource: {
        type: "http_load_balancer"
        labels: {
            zone: "global"
            url_map_name: "URL_MAP_NAME"
            forwarding_rule_name: "FORWARDING_RULE_NAME"
            target_proxy_name: "TARGET_PROXY_NAME"
            backend_service_name: ""
            project_id: "PROJECT_ID"
        }
    }
    timestamp: "2020-06-08T23:41:25.078651Z"
    severity: "INFO"
    logName: "projects/PROJECT_ID/logs/requests"
    trace: "projects/PROJECT_ID/traces/241d69833e64b3bf83fabac8c873d992"
    receiveTimestamp: "2020-06-08T23:41:25.588272510Z"
    spanId: "7b6537d3672e08e1"
}

Console

1. In the Google Cloud Console, go to the Load balancing page.

   Go to the Load balancing page

2. Click http-lb to expand the load balancer that you just created.

   In the Backend section, confirm that the backend bucket is healthy. There should be a green checkmark next to your backend bucket. If you see otherwise, first try reloading the page. It can take a few moments for the Cloud Console to indicate that backends are healthy.

3. After the Cloud Console shows that the backend bucket is healthy, you can test your load balancer using a web browser by going to http://IP_ADDRESS/never-fetch/three-cats.jpg. Replace IP_ADDRESS with the load balancer's IP address. Your browser should render a page with content showing the graphic file.

gcloud

Use the curl command to test the response from the URL. Replace IP_ADDRESS with the load balancer's IPv4 address.

Note the IPv4 address that was reserved:

gcloud compute addresses describe example-ip \
    --format="get(address)" \
    --global

Send a curl request:

curl http://IP_ADDRESS/never-fetch/three-cats.jpg

Content is fetched from Cloud Storage, cached by Cloud CDN, and then validated and re-fetched when it expires or is otherwise evicted from the cache. Content that needs to be refreshed earlier than the TTL can be invalidated and re-fetched from Cloud Storage.

Using cache modes and TTLs

Beta

This product or feature is covered by the Pre-GA Offerings Terms of the Google Cloud Platform Terms of Service. Pre-GA products and features may have limited support, and changes to pre-GA products and features may not be compatible with other pre-GA versions. For more information, see the launch stage descriptions.

To automatically cache static responses from your origin, you can use the CACHE_ALL_STATIC cache mode setting.

To control cacheability for each response by using HTTP cache directives, set the cache mode to use origin headers (USE_ORIGIN_HEADERS). For information about the cache directives that Cloud CDN understands and what's not cached by Cloud CDN, see Cacheable content and Non-cacheable content.

If your origin isn't serving any per-user dynamic content, you might want to cache all responses from the origin. To do this, use the FORCE_CACHE_ALL mode. This mode caches all responses, regardless of content type or cache directives.

If you don't explicitly select a cache mode when you enable Cloud CDN on a backend, the API and the gcloud command-line tool default to USE_ORIGIN_HEADERS, and the Cloud Console defaults to CACHE_ALL_STATIC.

New backend buckets with Cloud CDN enabled automatically default to the CACHE_ALL_STATIC mode.

1. Enable Cloud CDN on your backend bucket.
2. Set the cache mode to automatically cache all static content.
3. Set an appropriate default TTL (for example, one day) to have Cloud CDN cache it.

Disabling Cloud CDN

Console

Disable Cloud CDN for a single backend bucket

1. In the Google Cloud Console, go to the Cloud CDN page.

   Go to the Cloud CDN page

2. On the right side of the origin row, click Menu and then select Edit.
3. Clear the checkboxes of any backend buckets that you want to stop from using Cloud CDN.
4. Click Update.

Remove Cloud CDN for all backend buckets for an origin

1. In the Cloud Console, go to the Cloud CDN page.

   Go to the Cloud CDN page

2. On the right side of the origin row, click Menu and then select Remove.
3. To confirm, click Remove.

gcloud

gcloud compute backend-buckets update BACKEND_BUCKET_NAME \
    --no-enable-cdn

Disabling Cloud CDN does not invalidate or purge caches. If you turn Cloud CDN off and back on again, most or all of your cached content might still be cached. To prevent content from being used by the caches, you must invalidate that content.

What's next

• To learn about what content is cached, see the Caching overview.
• To use Cloud CDN in GKE, see Ingress features.
• To check whether Cloud CDN is serving responses from cache, see Viewing logs.
• To find information about common problems and solutions, see Troubleshooting.