Migrating from Amazon S3 to Cloud Storage

This page describes how to migrate from Amazon Simple Storage Service (Amazon S3) to Cloud Storage for users sending requests using an API.

If you are new to Cloud Storage and will not be using the API directly, consider using the Google Cloud Console to set and manage transfers. The Google Cloud Console provides a graphical interface to Cloud Storage that enables you to accomplish many of your storage tasks using just a browser, including migration of your data from Amazon S3 to Cloud Storage.

Overview of migration

If you are an Amazon S3 user, you can easily migrate your applications that use Amazon S3 to use Cloud Storage. You have two migration options:

  • Simple Migration: This is this easiest way to get started with Cloud Storage if you are coming from Amazon S3 because it requires just a few simple changes to the tools and libraries you currently use with Amazon S3.

  • Full Migration: A full migration from Amazon S3 to Cloud Storage requires a few extra steps compared to a simple migration, but the benefit is that you can use all the features of Cloud Storage, including multiple projects and OAuth 2.0 for authentication.

Simple migration

In a simple migration from Amazon S3 to Cloud Storage, you use your existing tools and libraries for generating authenticated REST requests to Amazon S3 to also send authenticated requests to Cloud Storage. The only steps you need to take to make requests to Cloud Storage are:

  • Set a default Google project.
  • Get an HMAC key.

  • In your existing tools or libraries, make the following changes:

    • Change the request endpoint to use the Cloud Storage XML API request endpoint.
    • Replace the Amazon Web Services (AWS) access and secret key with the corresponding Cloud Storage access ID and secret (collectively called your Cloud Storage HMAC key).

    • Make sure your x-amz- headers use supported Cloud Storage values. For example, x-amz-storage-class should use one of the available Cloud Storage storage classes.

      When you use the Cloud Storage XML API in a simple migration scenario, specifying the AWS signature identifier in the Authorization header lets Cloud Storage know to expect x-amz-* headers and Amazon S3 ACL XML syntax in your request. Cloud Storage processes x-amz-* headers that have an x-goog-* equivalent, such as those listed in the headers table.

With these changes you can start using your existing tools and libraries to send keyed-hash message authentication code (HMAC) requests to Cloud Storage.

For example, the following samples demonstrate how to list Cloud Storage buckets using the Amazon S3 SDK:

Go

For more information, see the Cloud Storage Go API reference documentation. 

import (
	"context"
	"fmt"
	"io"
	"time"

	"github.com/aws/aws-sdk-go/aws"
	"github.com/aws/aws-sdk-go/aws/credentials"
	"github.com/aws/aws-sdk-go/aws/session"
	"github.com/aws/aws-sdk-go/service/s3"
)

func listGCSBuckets(w io.Writer, googleAccessKeyID string, googleAccessKeySecret string) error {
	// googleAccessKeyID := "Your Google Access Key ID"
	// googleAccessKeySecret := "Your Google Access Key Secret"

	// Create a new client and do the following:
	// 1. Change the endpoint URL to use the Google Cloud Storage XML API endpoint.
	// 2. Use Cloud Storage HMAC Credentials.
	sess := session.Must(session.NewSession(&aws.Config{
		Region:      aws.String("auto"),
		Endpoint:    aws.String("https://storage.googleapis.com"),
		Credentials: credentials.NewStaticCredentials(googleAccessKeyID, googleAccessKeySecret, ""),
	}))

	client := s3.New(sess)
	ctx := context.Background()

	ctx, cancel := context.WithTimeout(ctx, time.Second*10)
	defer cancel()
	result, err := client.ListBucketsWithContext(ctx, &s3.ListBucketsInput{})
	if err != nil {
		return fmt.Errorf("ListBucketsWithContext: %v", err)
	}

	fmt.Fprintf(w, "Buckets:")
	for _, b := range result.Buckets {
		fmt.Fprintf(w, "%s\n", aws.StringValue(b.Name))
	}

	return nil
}

Java

For more information, see the Cloud Storage Java API reference documentation. 

import com.amazonaws.auth.AWSStaticCredentialsProvider;
import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.client.builder.AwsClientBuilder;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3ClientBuilder;
import com.amazonaws.services.s3.model.Bucket;
import java.util.List;

public class ListGcsBuckets {
  public static void listGcsBuckets(String googleAccessKeyId, String googleAccessKeySecret) {

    // String googleAccessKeyId = "your-google-access-key-id";
    // String googleAccessKeySecret = "your-google-access-key-secret";

    // Create a BasicAWSCredentials using Cloud Storage HMAC credentials.
    BasicAWSCredentials googleCreds =
        new BasicAWSCredentials(googleAccessKeyId, googleAccessKeySecret);

    // Create a new client and do the following:
    // 1. Change the endpoint URL to use the Google Cloud Storage XML API endpoint.
    // 2. Use Cloud Storage HMAC Credentials.
    AmazonS3 interopClient =
        AmazonS3ClientBuilder.standard()
            .withEndpointConfiguration(
                new AwsClientBuilder.EndpointConfiguration(
                    "https://storage.googleapis.com", "auto"))
            .withCredentials(new AWSStaticCredentialsProvider(googleCreds))
            .build();

    // Call GCS to list current buckets
    List<Bucket> buckets = interopClient.listBuckets();

    // Print bucket names
    System.out.println("Buckets:");
    for (Bucket bucket : buckets) {
      System.out.println(bucket.getName());
    }

    // Explicitly clean up client resources.
    interopClient.shutdown();
  }

Python

For more information, see the Cloud Storage Python API reference documentation. 

import boto3


def list_gcs_buckets(google_access_key_id, google_access_key_secret):
    """Lists all GCS buckets using boto3 SDK"""
    # Create a new client and do the following:
    # 1. Change the endpoint URL to use the
    #    Google Cloud Storage XML API endpoint.
    # 2. Use Cloud Storage HMAC Credentials.
    client = boto3.client(
        "s3",
        region_name="auto",
        endpoint_url="https://storage.googleapis.com",
        aws_access_key_id=google_access_key_id,
        aws_secret_access_key=google_access_key_secret,
    )

    # Call GCS to list current buckets
    response = client.list_buckets()

    # Print bucket names
    print("Buckets:")
    for bucket in response["Buckets"]:
        print(bucket["Name"])

Note that while most actions are available using the Amazon S3 V2 SDK, listing objects can only be performed using the Amazon S3 V1 list objects method. The following samples demonstrate such an object listing:

Go

For more information, see the Cloud Storage Go API reference documentation. 

import (
	"context"
	"fmt"
	"io"

	"github.com/aws/aws-sdk-go/aws"
	"github.com/aws/aws-sdk-go/aws/credentials"
	"github.com/aws/aws-sdk-go/aws/session"
	"github.com/aws/aws-sdk-go/service/s3"
)

func listGCSObjects(w io.Writer, bucketName string, googleAccessKeyID string, googleAccessKeySecret string) error {
	// bucketName := "your-gcs-bucket-name"
	// googleAccessKeyID := "Your Google Access Key ID"
	// googleAccessKeySecret := "Your Google Access Key Secret"

	// Create a new client and do the following:
	// 1. Change the endpoint URL to use the Google Cloud Storage XML API endpoint.
	// 2. Use Cloud Storage HMAC Credentials.
	sess := session.Must(session.NewSession(&aws.Config{
		Region:      aws.String("auto"),
		Endpoint:    aws.String("https://storage.googleapis.com"),
		Credentials: credentials.NewStaticCredentials(googleAccessKeyID, googleAccessKeySecret, ""),
	}))

	client := s3.New(sess)
	ctx := context.Background()

	result, err := client.ListObjectsWithContext(ctx, &s3.ListObjectsInput{
		Bucket: aws.String(bucketName),
	})
	if err != nil {
		return fmt.Errorf("ListObjectsWithContext: %v", err)
	}

	fmt.Fprintf(w, "Objects:")
	for _, o := range result.Contents {
		fmt.Fprintf(w, "%s\n", aws.StringValue(o.Key))
	}

	return nil
}

Java

For more information, see the Cloud Storage Java API reference documentation. 

import com.amazonaws.auth.AWSStaticCredentialsProvider;
import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.client.builder.AwsClientBuilder;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3ClientBuilder;
import com.amazonaws.services.s3.model.ObjectListing;
import com.amazonaws.services.s3.model.S3ObjectSummary;

public class ListGcsObjects {
  public static void listGcsObjects(
      String googleAccessKeyId, String googleAccessKeySecret, String bucketName) {

    // String googleAccessKeyId = "your-google-access-key-id";
    // String googleAccessKeySecret = "your-google-access-key-secret";
    // String bucketName = "bucket-name";

    // Create a BasicAWSCredentials using Cloud Storage HMAC credentials.
    BasicAWSCredentials googleCreds =
        new BasicAWSCredentials(googleAccessKeyId, googleAccessKeySecret);

    // Create a new client and do the following:
    // 1. Change the endpoint URL to use the Google Cloud Storage XML API endpoint.
    // 2. Use Cloud Storage HMAC Credentials.
    AmazonS3 interopClient =
        AmazonS3ClientBuilder.standard()
            .withEndpointConfiguration(
                new AwsClientBuilder.EndpointConfiguration(
                    "https://storage.googleapis.com", "auto"))
            .withCredentials(new AWSStaticCredentialsProvider(googleCreds))
            .build();

    // Call GCS to list current objects
    ObjectListing objects = interopClient.listObjects(bucketName);

    // Print objects names
    System.out.println("Objects:");
    for (S3ObjectSummary object : objects.getObjectSummaries()) {
      System.out.println(object.getKey());
    }

    // Explicitly clean up client resources.
    interopClient.shutdown();
  }
}

Python

For more information, see the Cloud Storage Python API reference documentation. 

import boto3


def list_gcs_objects(google_access_key_id, google_access_key_secret, bucket_name):
    """Lists GCS objects using boto3 SDK"""
    # Create a new client and do the following:
    # 1. Change the endpoint URL to use the
    #    Google Cloud Storage XML API endpoint.
    # 2. Use Cloud Storage HMAC Credentials.

    client = boto3.client(
        "s3",
        region_name="auto",
        endpoint_url="https://storage.googleapis.com",
        aws_access_key_id=google_access_key_id,
        aws_secret_access_key=google_access_key_secret,
    )

    # Call GCS to list objects in bucket_name
    response = client.list_objects(Bucket=bucket_name)

    # Print object names
    print("Objects:")
    for blob in response["Contents"]:
        print(blob["Key"])

Setting a default project

To use the Cloud Storage in a simple migration scenario, we recommend setting a default project. When you set a default project, you are telling Cloud Storage the project to use for operations like GET service or PUT bucket. If you don't set a default project, you must specify a project header in certain requests.

To set a default project:

  1. Open the Cloud Storage Settings page in the Google Cloud Console .
  2. Select the Interoperability tab.

  3. Click Set PROJECT-ID as default project, located in the section Default project for interoperable access.

    If the project is already the default project, you see PROJECT-ID is your default project for interoperable access.

This project is now your default project. You can change your default project at any time by choosing a different project and following these steps.

Alternative option: Specifying a project header

Instead of, or in addition to, setting a default project, you can use the x-amz-project-id header in individual requests that require you to specify a project.

  • A request that uses x-amz-project-id uses the project specified in the header, even if you've set a default project.

Using the x-amz-project-id header is useful if:

  • You're working with multiple projects.
  • Your requests are made by a service account associated with a different project, because service accounts use their parent project as the default project.

Note that Amazon S3 does not have projects, so depending on the tools or client libraries you use, specifying a x-amz-project-id header may not be an option. In this case, you should set a default project.

Using HMAC keys for a simple migration

To use the Cloud Storage XML API in a simple migration scenario, use Cloud Storage hash-based message authentication code (HMAC) keys for the credentials. Typically, you should create an HMAC key that is associated with a service account, however you can alternatively use one associated with a user account.

Authenticating in a simple migration scenario

Authorization header

For operations in a simple migration scenario that require authentication, you include an Authorization request header just like you do for requests to Amazon S3. The Authorization header syntax for an Amazon S3 request is:

Authorization: AWS4-HMAC-SHA256 Credential=AWS-ACCESS-KEY/CREDENTIAL_SCOPE, SignedHeaders=SIGNED_HEADERS, Signature=SIGNATURE

In a simple migration scenario, you only change the header to use your Google HMAC access ID and make sure the Signature you attach is calculated with your Google HMAC secret key:

Authorization: ALGORITHM Credential=GOOG-ACCESS-ID/CREDENTIAL_SCOPE, SignedHeaders=SIGNED_HEADERS, Signature=SIGNATURE

The parts of the Authorization header are:

  • ALGORITHM: The signature algorithm and version that you are using. Using AWS4-HMAC-SHA256 indicates that you are using an HMAC V4 signature and you intend to send x-amz-* headers. You can also use GOOG4-HMAC-SHA256, which indicates that you are using an HMAC V4 signature and you intend to send x-goog-* headers.

  • GOOG-ACCESS-ID: The access ID identifies the entity that is making and signing the request. In a simple migration, replace the Amazon Web Service (AWS) access key ID you use to access Amazon S3 with your Google HMAC access ID. Your Google HMAC access ID starts with GOOG.

  • CREDENTIAL_SCOPE: The credential scope, as defined in the signature. In a simple migration, you don't need to change credential scope if you are using AWS4-HMAC-SHA256 for your ALGORITHM value.

  • SIGNED_HEADERS: A semicolon-separated list of names of headers that must be included to sign this request. All headers should be lowercase and sorted by character code.

    An example of an Amazon S3-style signed header string looks like:

    content-type;host;x-amz-date

    In a simple migration, you don't need to make any changes to the signed header string.

  • SIGNATURE: The signature that allows the request to be authenticated. In a simple migration, replace the AWS access key information with the equivalent Google HMAC key information.

Sample authentication request

The following examples upload an object named /europe/france/paris.jpg to a bucket named my-travel-maps, apply the predefined ACL public-read, and define a custom metadata header for reviewers. Here is the request to a bucket in Amazon S3:

PUT europe/france/paris.jpg HTTP/1.1
Host: my-travel-maps.s3.amazonaws.com
Date: Mon, 11 Mar 2019 23:46:19 GMT
Content-Length: 888814
Content-Type: image/jpg
x-amz-acl: public-read
x-amz-date:20190311T192918Z
x-amz-meta-reviewer: joe,jane
Authorization: AWS4-HMAC-SHA256 Credential=AWS-ACCESS-KEY/20190311/us-east-1/s3/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-acl;x-amz-date;x-amz-meta-reviewer, Signature=SIGNATURE

Here is the request for a bucket in Cloud Storage:

PUT europe/france/paris.jpg HTTP/1.1
Host: my-travel-maps.storage.googleapis.com
Date: Mon, 11 Mar 2019 23:46:19 GMT
Content-Length: 888814
Content-Type: image/jpg
x-amz-acl: public-read
x-amz-date:20190311T192918Z
x-amz-meta-reviewer: joe,jane
Authorization: AWS4-HMAC-SHA256 Credential=GOOG-ACCESS-ID/20190311/us-east-1/s3/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-acl;x-amz-date;x-amz-meta-reviewer, Signature=SIGNATURE

Here is the corresponding canonical request that was created for this request:

PUT
/europe/france/paris.jpg

content-length:888814
content-type:image/jpg
host:my-travel-maps.storage.googleapis.com
x-amz-acl:public-read
x-amz-date:20190311T192918Z
x-amz-meta-reviewer:joe,jane

content-length,content-type,host,x-amz-acl,x-amz-date,x-amz-meta-reviewer
82e3da8b3f35989512e8d428add7eca73ab0e5f36586e66fbad8e1051343cbd2

Here is the corresponding string-to-sign that was created for this request:

AWS4-HMAC-SHA256
20190311T192918Z
20190311/us-east-1/s3/aws4_request
73918a5ff373d7a03e406fbf9ea35675396b06fca2af76c27a5c451fa783ef65

This request did not provide a Content-MD5 header, so an empty string is shown in the second line of the message.

Access control in a simple migration scenario

To support simple migrations, Cloud Storage accepts ACLs produced by Amazon S3. In a simple migration scenario, you use AWS as your signature identifier, which tells Cloud Storage to expect ACL syntax using Amazon S3 ACL XML syntax. You should ensure that the Amazon S3 ACLs you use map to the Cloud Storage ACL model. For example, if your tools and libraries use Amazon S3's ACL syntax to grant bucket WRITE permission, then they must also grant bucket READ permission because Cloud Storage permissions are concentric. You do not need to specify both WRITE and READ permission when you grant WRITE permission using the Cloud Storage syntax.

Cloud Storage supports Amazon S3 ACL syntax in the following scenarios:

  • In a request to Cloud Storage to retrieve ACLs (for example, a GET Object or GET Bucket request), Cloud Storage returns Amazon S3 ACL syntax.
  • In a request to Cloud Storage to apply ACLs (for example, a PUT Object or PUT Bucket request), Cloud Storage expects to receive Amazon S3 ACL syntax.

The Authorization header in a simple migration scenario uses AWS for the signature identifier, but with your Google access ID.

Authorization: AWS4-HMAC-SHA256 Credential=GOOG-ACCESS-ID/CREDENTIAL_SCOPE, SignedHeaders=SIGNED_HEADERS, Signature=SIGNATURE

The following example shows a GET request to Cloud Storage to return the ACLs for an object.

GET europe/france/paris.jpg?acl HTTP/1.1
Host: my-travel-maps.storage.googleapis.com
Date: Thu, 21 Feb 2019 23:50:10 GMT
Content-Type: application/xml
X-Amz-Date: 20190221T235010Z
Authorization: AWS4-HMAC-SHA256 Credential=GOOGMC5PDPA5JLZYQMHQHRAX/20190221/region/s3/aws4_request, SignedHeaders=host;x-amz-date, Signature=29088b1d6dfeb2549f6ff67bc3744abb7e45475f0ad60400485805415bbfc534

The response to the request includes the ACL using Amazon S3 ACL syntax.

<?xml version='1.0' encoding='UTF-8'?>
<AccessControlPolicy>
    <Owner>
        <ID>00b4903a972faa8bcce9382686e9129676f1cd6e5def1f5663affc2ba4652490
        </ID>
        <DisplayName>OwnerName</DisplayName>
    </Owner>
    <AccessControlList>
        <Grant>
            <Grantee xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance'
                xsi:type='CanonicalUser'>
                <ID>00b4903a972faa8bcce9382686e9129676f1cd6e5def1f5663affc2ba4652490</ID>
                <DisplayName>UserName</DisplayName>
            </Grantee>
            <Permission>FULL_CONTROL</Permission>
        </Grant>
    </AccessControlList>
</AccessControlPolicy>

The following example shows a PUT request to Cloud Storage to set the ACLs for an object. The example shows a request body with Amazon S3 ACL syntax.

PUT europe/france/paris.jpg?acl HTTP/1.1
Host: my-travel-maps.storage.googleapis.com
Date: Thu, 21 Feb 2019 23:50:10 GMT
Content-Type: application/xml
Content-Length: 337
X-Amz-Date: 20190221T235010Z
Authorization: AWS4-HMAC-SHA256 Credential=GOOGMC5PDPA5JLZYQMHQHRAX/20190221/region/s3/aws4_request, SignedHeaders=host;x-amz-date, Signature=29088b1d6dfeb2549f6ff67bc3744abb7e45475f0ad60400485805415bbfc534

<?xml version='1.0' encoding='utf-8'?>
<AccessControlPolicy>
  <AccessControlList>
    <Grant>
      <Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="AmazonCustomerByEmail">
        <EmailAddress>jane@gmail.com</EmailAddress>
      </Grantee>
      <Permission>FULL_CONTROL</Permission>
    </Grant>
  </AccessControlList>
</AccessControlPolicy>

Finally, in a simple migration scenario, you can also use the GOOG1 signature identifier in the Authorization header. In this case, you must use the Cloud Storage ACL syntax and ensure that all of your headers are Google headers, x-goog-*. While this is possible, we recommend that you move to a full migration as described below in order to realize all the benefits of Cloud Storage.

Full migration

A full migration from Amazon S3 to Cloud Storage enables you to take advantage of all the features of Cloud Storage including OAuth 2.0 authentication. OAuth 2.0 relies on SSL for security instead of requiring your application to do cryptographic signing directly and is easier to implement. With OAuth, your application can request access to data associated with a user's Google Account, and access can be scoped to several levels, including read-only, read-write, and full-control. For more information, see OAuth 2.0 Authentication.

To migrate fully from Amazon S3 to Cloud Storage, make the following changes:

  • Change any existing x-amz-* headers to corresponding x-goog-* headers.
  • Change AWS Access Control List (ACL) XML to the corresponding Cloud Storage ACL XML (see Creating and managing access control lists).
  • Set the x-goog-project-id header in your requests.

  • Get set up to use OAuth 2.0 authentication as described in OAuth 2.0 Authentication. The first step is to register your application (where you will be issuing requests from) with Google. Using OAuth 2.0 means that your Authorization header looks like this:

    Authorization: Bearer OAUTH2_TOKEN

Access control in a full migration

This section shows a few examples of access control to help you migrate from Amazon S3 to Cloud Storage. For an overview of access control in Cloud Storage, see Access Control.

In Cloud Storage, there are several ways to apply ACLs to buckets and objects (see Creating and managing access control lists). Two of the ways you specify ACLs are analogous to what you do in Amazon S3:

  • The acl query string parameter to apply ACLs for specific scopes.
  • The x-goog-acl request header lets you apply predefined ACLs, which are sometimes known as canned ACLs.

Using the acl query string parameter

You can use the acl query string parameter for a Cloud Storage request exactly the same way you would use it for an Amazon S3 request. The acl parameter is used in conjunction with the PUT method to apply ACLs to the following: an existing object, an existing bucket, or a bucket you are creating. When you use the acl query string parameter in a PUT request, you must attach an XML document (using Cloud Storage ACL syntax) to the body of your request. The XML document contains the individual ACL entries that you want to apply to the bucket or object.

The following example shows a PUT request to Amazon S3 that uses the acl query string parameter. ACLs are defined in an XML document sent in the request body. The PUT request changes the ACLs on an object named europe/france/paris.jpg that is in a bucket named my-travel-maps. The ACL grants jane@gmail.com FULL_CONTROL permission.

PUT europe/france/paris.jpg?acl HTTP/1.1
Host: my-travel-maps.s3.amazonaws.com
Date: Wed, 06 Nov 2013 19:28:18 GMT
Content-Length: 598
Content-Type: application/xml
Authorization: AWS4-HMAC-SHA256 Credential=AWS-ACCESS-KEY/20131106/us-east-1/s3/aws4_request, SignedHeaders=content-length;content-type;date;host, Signature=4c45f25bb679fdab0de5a287625d6a143414728d93c9aeb9f4cc91c33a1c45fg

<?xml version='1.0' encoding='utf-8'?>
<AccessControlPolicy>
  <Owner>
    <ID>5a6557ba40f7c86496ffceae789fcd888abc1b62a7149873a0fe12c0f60a7d95</ID>
    <DisplayName>ownerEmail@example.com</DisplayName>
  </Owner>
  <AccessControlList>
    <Grant>
      <Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser">
        <ID>fd447671d60b979f78ee6fcec7b22afc80e6b26a4db16eed01afb8064047949b</ID>
        <DisplayName>jane@gmail.com</DisplayName>
      </Grantee>
      <Permission>FULL_CONTROL</Permission>
    </Grant>
  </AccessControlList>
</AccessControlPolicy>

Here is the same request to Cloud Storage:

PUT europe/france/paris.jpg?acl HTTP/1.1
Host: my-travel-maps.storage.googleapis.com
Date: Wed, 06 Nov 2013 19:37:33 GMT
Content-Length: 268
Content-Type: application/xml
Authorization: Bearer ya29.AHES6ZRVmB7fkLtd1XTmq6mo0S1wqZZi3-Lh_s-6Uw7p8vtgSwg

<?xml version='1.0' encoding='utf-8'?>
<AccessControlList>
  <Entries>
  <Entry>
    <Permission>FULL_CONTROL</Permission>
    <Scope type="UserByEmail">
      <EmailAddress>jane@gmail.com</EmailAddress>
    </Scope>
  </Entry>
  </Entries>
</AccessControlList>

Note that Cloud Storage does not require an <Owner/> element in the ACL XML document. For more information, see Bucket and object ownership.

You can also retrieve bucket and object ACLs by using the acl query string parameter with the GET method. The ACLs are described in an XML document, which is attached to the body of the response. You must have FULL_CONTROL permission to apply or retrieve ACLs on an object or bucket.

Applying ACLs with an extension request header

You can use the x-goog-acl header in a Cloud Storage request to apply predefined ACLs to buckets and objects exactly the same way you would use the x-amz-acl header in an Amazon S3 request. You typically use the x-goog-acl (x-amz-acl) header to apply a predefined ACL to a bucket or object when you are creating or uploading the bucket or object. The Cloud Storage predefined ACLs are similar to Amazon S3 Canned ACLs, including private, public-read, public-read-write, as well as others. For a list of Cloud Storage predefined ACLs, see Predefined ACLs.

The following example shows a PUT Object request that applies the public-read ACL to an object named europe/france/paris.jpg that is being uploaded into a bucket named my-travel-maps in Amazon S3.

PUT europe/france/paris.jpg HTTP/1.1
Host: my-travel-maps.s3.amazonaws.com
Date: Wed, 06 Nov 2013 20:48:42 GMT
Content-Length: 888814
Content-Type: image/jpg
x-amz-acl: public-read
Authorization: AWS4-HMAC-SHA256 Credential=AWS-ACCESS-KEY/20131106/us-east-1/s3/aws4_request, SignedHeaders=content-length;content-type;date;host, Signature=808150c37dbd1b425b2398421d6fc3dd6d4942dfaae9e519fd5835aa62fd62ab

<888814 bytes in entity body>

Here is the same request to Cloud Storage:

PUT europe/france/paris.jpg HTTP/1.1
Host: my-travel-maps.storage.googleapis.com
Date: Wed, 06 Nov 2013 20:49:57 GMT
Content-Length: 888814
Content-Type: image/jpg
x-goog-acl: public-read
Authorization: Bearer ya29.AHES6ZRVmB7fkLtd1XTmq6mo0S1wqZZi3-Lh_s-6Uw7p8vtgSwg

<888814 bytes in entity body>

You can also use the x-goog-acl header to apply a predefined ACL to an existing bucket or object. To do this, include the acl query string parameter in your request but do not include an XML document in your request. Applying a predefined ACL to an existing object or bucket is useful if you want to change from one predefined ACL to another, or you want to update custom ACLs to a predefined ACL. For example, the following PUT Object request applies the predefined ACL private to an object named europe/france/paris.jpg that is in a bucket named my-travel-maps.

PUT europe/france/paris.jpg?acl HTTP/1.1
Host: my-travel-maps.storage.googleapis.com
Date: Wed, 06 Nov 2013 00:26:36 GMT
Content-Length: 0
x-goog-acl: private
Authorization: Bearer ya29.AHES6ZRVmB7fkLtd1XTmq6mo0S1wqZZi3-Lh_s-6Uw7p8vtgSwg

<empty entity body>

For more information about managing ACLs, see Creating and managing access control lists.

Migrating from Amazon S3 to Cloud Storage Request Methods

Cloud Storage supports the same standard HTTP request methods for reading and writing data to your buckets as are supported in Amazon S3. Therefore, the majority of your tools and libraries that you currently use with Amazon S3, work as-is with Cloud Storage. Cloud Storage supports the following request methods:

  • Service request for GET.
  • Bucket requests, including PUT, GET, DELETE.
  • Object requests, including GET, POST, PUT, HEAD, and DELETE.

For more information, see XML API Reference Methods. Keep in mind that when you send requests to Cloud Storage, you need to change the request body, when applicable, to use the appropriate Cloud Storage syntax. For example, when you create a lifecycle configuration for a bucket, use the Cloud Storage lifecycle XML, which is different than the Amazon S3 lifecycle XML.

There are a few differences between Cloud Storage XML API and Amazon S3 which are summarized below, with suggested Cloud Storage alternatives:

Amazon S3 Functionality Cloud Storage XML API Functionality
Multipart upload.
POST /OBJECT_NAME, PUT /OBJECT_NAME

In the Cloud Storage XML API, you can upload a series of component objects, performing a separate upload for each component. Then you can compose the objects into a single composite object.

Note: While the JSON API offers a multipart upload feature, this feature is used for sending metadata along with object data. It is not equivalent to S3's multipart upload feature.

GET/POST bucket query string parameters:
  • "policy" - Working with Amazon S3 bucket policies.
  • "website" - Configuring bucket websites.
  • "tagging" - Tagging buckets for cost allocation purposes.
  • "notification" - Notifying for bucket events.
  • "requestPayment" - Configuring who pays for the request and the data download from a bucket.
 Alternatives:
  • "policy" - Cloud Storage ACLs, project team membership, and the ability to use multiple projects address many of the scenarios where bucket policies are used.
  • "website" - Use the gsutil web command to manage websites, or try the JSON API (see buckets resource).
  • "tagging" - Use multiple projects to track different cost centers. For more about projects, see Managing Projects.
  • "notification" - Use gsutil or JSON API Pub/Sub Notifications.
  • "requestPayment" - Use multiple projects with different billing profiles to manage who pays for requests and downloaded data from a bucket. For more about configuring billing, see Billing in the Google APIs Console Help documentation.
Multiple object delete.
POST /?delete

Use the gsutil rm command to easily remove multiple objects. The rm command supports the "-m" option to perform parallel (multi-threaded/multi-processing) deletes.

Alternatively, the JSON API supports sending batch requests to reduce the number of HTTP connections your client makes.

Migrating from Amazon S3 to Cloud Storage Headers

Cloud Storage uses several standard HTTP headers as well as several custom (extension) HTTP headers. If you are transitioning from Amazon S3 to Cloud Storage, you can convert your custom Amazon S3 headers to the equivalent Cloud Storage custom header or similar functionality as shown in the tables below.

For many Amazon S3 headers, you simply need to replace the x-amz prefix with x-goog:

Amazon S3 Header Cloud Storage Header
x-amz-storage-class x-goog-storage-class
x-amz-acl x-goog-acl
x-amz-date x-goog-date
x-amz-meta-* x-goog-meta-*
x-amz-copy-source x-goog-copy-source
x-amz-metadata-directive x-goog-metadata-directive
x-amz-copy-source-if-match x-goog-copy-source-if-match
x-amz-copy-source-if-none-match x-goog-copy-source-if-none-match
x-amz-copy-source-if-unmodified-since x-goog-copy-source-if-unmodified-since
x-amz-copy-source-if-modified-since x-goog-copy-source-if-modified-since

Several headers differ or do not apply in Cloud Storage:

Amazon S3 Header Cloud Storage Header
x-amz-server-side-encryption Not required. Cloud Storage automatically encrypts all data before it is written to disk. For more information, see Encryption.
x-amz-grant-* x-goog-acl with a predefined ACL value.
x-amz-mfa Use OAuth 2.0 Authentication.
x-amz-website-redirect-location, x-amz-copy-source-range n/a

See HTTP headers and query string parameters for XML API for a reference to Cloud Storage headers.

Support for XML API compatibility with Amazon S3

For discussions about XML API interoperability, see Stack Overflow using the tag google-cloud-storage. See the Getting support page for more information about discussion forums and subscribing to announcements.