Use a private Amazon S3-compatible bucket as an origin

This page describes how to connect Media CDN to private Amazon S3-compatible buckets by using Amazon Simple Storage Service (Amazon S3) and AWS Signature Version 4. Media CDN does not support AWS Signature Version 4a.

Media CDN supports AWS Signature Version 4 to authenticate origin requests. You can use this feature to connect Media CDN to your private Amazon S3-compatible buckets to ensure that your content is shared only with Media CDN. You can also enable client authentication for greater defense in depth. For more information, see Use signed requests.

Before you begin

Verify that you have the following resources:

A private Amazon S3 bucket that follows the naming rules.
A dedicated AWS IAM user account to access the preceding private Amazon S3 bucket. Ensure that the bucket grants the s3:getObject permission to the AWS IAM user account. For more information about configuring access to your Amazon S3 bucket, see Identity and access management in Amazon S3.
The AWS access key ID and the AWS secret access key for the dedicated AWS IAM user account. For more information, see Managing access keys for IAM users.
If you haven't used Secret Manager before, configure Secret Manager.

Create a service agent for Media CDN

To create a service agent for Media CDN, use the gcloud beta services identity create command.

gcloud

gcloud beta services identity create \
    --project PROJECT_ID \
    --service=networkservices.googleapis.com

Replace PROJECT_ID with your project ID.

For more information, see Trigger service agent creation.

Store access keys in Secret Manager

Follow these steps to create a secret in Secret Manager.

Console

In the Google Cloud console, go to the Secret Manager page.

Go to Secret Manager
On the Secret Manager page, click Create secret.
In the Secret details section, complete the following:
1. In the Name field, enter a name for the secret—for example aws-access-key-id.
2. In the Secret value section, skip the Upload file field.
3. In the Secret value field, enter the AWS secret access key.
Skip the remaining sections.
Click Create secret.

gcloud

Ensure that you're using version 402.0.0 or later of the Google Cloud CLI. On Compute Engine or Google Kubernetes Engine (GKE), you must authenticate with the cloud-platform scope.

gcloud secrets create SECRET_NAME \
    --replication-policy="automatic" \
    --data-file="PATH_TO_AWS_SECRET_ACCESS_KEY"

Replace the following:

SECRET_NAME: the name for your secret, such as aws-access-key-id
PATH_TO_AWS_SECRET_ACCESS_KEY: the path to your AWS secret access key value

For information about adding secret versions, see Add a secret version.

Grant the Secret Manager Access role

Follow these steps to grant the Secret Manager Secret Accessor role (roles/secretmanager.secretAccessor) to the Media CDN service account.

Console

In the Google Cloud console, go to the Secret Manager page.

Go to Secret Manager
Select the secret.
Select Permissions, and then select Grant access. The Grant access dialog appears.
In the Add principals section, enter the Media CDN service account in the New principals field as follows:
service-PROJECT_NUMBER@gcp-sa-mediaedgefill.iam.gserviceaccount.com

Replace PROJECT_NUMBER with your project number.
In the Assign roles section, for Select a role, select Secret Manager, and then select Secret Manager Secret Accessor.
Click Save.

gcloud

gcloud secrets add-iam-policy-binding \
projects/PROJECT_NUMBER/secrets/SECRET_NAME \
    --member="serviceAccount:service-PROJECT_NUMBER@gcp-sa-mediaedgefill.iam.gserviceaccount.com" \
    --role="roles/secretmanager.secretAccessor"

Replace the following:

PROJECT_NUMBER: your project number
SECRET_NAME: the name for your secret

Configure Media CDN to authenticate to your external origin bucket

Console

In the Google Cloud console, go to the Media CDN page.

Go to Media CDN
Click the Origins tab.
Click the name of the origin that you want to configure.

The Edge Cache origin details page opens.
Click the Edit button.
For Origin address, choose Specify an FQDN or IP address, and then enter the FQDN or IP address.
In the Private origin authentication section, select Authenticate Media CDN's request to this origin with AWS signature version 4.
Specify the following information:
- Key ID: your AWS access key ID with read permissions to your origin
- Reference a secret: the ID of your access key in Secret Manager and its version. You can select an existing ID, enter an ID manually, or create a new ID and select it.
Note: If the service account does not have permissions to access the selected secret, a prompt appears. Click Grant to grant the Secret Manager Secret Accessor role to the service account.
- Region: the region that your Amazon S3 bucket is located in, such as us-east-1
For Protocol, select HTTPS.
Click Update origin.

gcloud

To export the current configuration of your service to a YAML file, run the gcloud edge-cache services export command:
```
gcloud edge-cache services export SERVICE_NAME \
    --destination=FILENAME.yaml
```
Replace the following:
- SERVICE_NAME: the name of the Media CDN service
- FILENAME: the name of the YAML file
In Cloud Shell, use a text editor to edit the YAML file.
Update the YAML file so that it has the following lines:
```
name: ORIGIN_NAME
originAddress: "S3_BUCKET_NAME.s3.S3_REGION.amazonaws.com"
protocol: HTTPS
awsV4Authentication:
  accessKeyId: "AWS_ACCESS_KEY_ID"
  secretAccessKeyVersion: "projects/PROJECT_NUMBER/secrets/SECRET_NAME/versions/SECRET_VERSION"
  originRegion: "S3_REGION"
```
Replace the following:
- ORIGIN_NAME: a name that you set for this origin
- S3_BUCKET_NAME: the name of your Amazon S3 bucket
- S3_REGION: the region that your Amazon S3 bucket is located in, such as us-east-1
- AWS_ACCESS_KEY_ID: your AWS access key ID with read permissions to your origin
- PROJECT_NUMBER: your project number
- SECRET_NAME: the ID of your access key in Secret Manager
- SECRET_VERSION: the secret version to use
Configure host and path rewrites for requests to your origin. For example, to map all requests with a path prefix of /vod/, configure a hostRewrite that corresponds to the originAddress listed in your EdgeCacheService's YAML file. The following code sample shows how to use hostRewrite:
```
name: SERVICE_NAME
routeRules:
  - priority: 1
  - description: ROUTE_DESCRIPTION
    origin: ORIGIN_NAME
    matchRules:
    - prefixMatch: "/vod/"
    routeAction:
      urlRewrite:
        hostRewrite: "S3_BUCKET_NAME.s3.S3_REGION.amazonaws.com"
        pathPrefixRewrite: "/"
      cdnPolicy:
        cacheMode: CACHE_ALL_STATIC
        defaultTtl: 3600s
```
Replace SERVICE_NAME with the name of your EdgeCacheService.

For more information, see Use third-party storage backends.
Save the YAML file.

To update your configuration, import the YAML file.

gcloud edge-cache origins import ORIGIN_NAME \
    --source=FILENAME.yaml

After you configure Media CDN to authenticate to your external bucket, Media CDN generates an HTTP authorization header for all requests to your header. All query parameters are removed from the signature calculation and request to your origin.

Your origin might append additional headers to your responses. To remove the response headers before serving to clients, see Set custom headers. For more information about configuring origins, see Origin connectivity and shielding.