Metadata preservation

This document describes metadata that is preserved when you use Storage Transfer Service and Transfer service for on-premises data to transfer data to Cloud Storage.

Overview

Storage Transfer Service and Transfer for on-premises preserve the following metadata during a transfer to Cloud Storage:

• User-created custom metadata for transfers that originate from Cloud Storage, Amazon Simple Storage Service (Amazon S3), or Microsoft Azure Blob Storage (Microsoft Azure Storage).

• File size and last modified time (mtime) for transfers that originate from POSIX file systems.

• Optionally, symlinks, numeric UID, numeric GID, and numeric MODE can be preserved for transfers from and to POSIX file systems.

Metadata fields that are not explicitly mentioned in this document are not preserved.

Object and file metadata

Each object in Cloud Storage has metadata associated with the object, stored as key:value pairs. The metadata identifies properties of the object and how the object is handled when it is accessed. For more information about object metadata in Cloud Storage, see Object metadata.

The following describes the mutability of metadata in Cloud Storage:

• Editable metadata: If you have sufficient privileges, you can edit these metadata values. The following are types of editable metadata available in Cloud Storage:

  • Fixed-key metadata: The values of fixed-key metadata can be changed, but the keys cannot. This type of metadata typically corresponds to HTTP headers. For more information, see Fixed-key metadata.

  • Custom metadata: Both the values and the keys of custom metadata can be changed. For more information, see Custom metadata.

• Non-editable metadata: This type of metadata cannot be directly edited. It is set when the object is created or rewritten in Cloud Storage. For more information, see Non-editable metadata.

Note: Cloud Storage fails transfers of objects that have non-UTF-8 strings as metadata.

File metadata contains information about files and directories. Examples of file metadata in POSIX file systems include mtime, file size, file modes, and file ownership.

Metadata preservation behavior

The following sections list metadata examples from different source storage systems and how Storage Transfer Service and Transfer for on-premises preserve metadata from each. For an exhaustive list of metadata, refer to the source storage system's documentation.

Amazon S3 to Cloud Storage

Metadata example	Preservation behavior
Amazon S3 fixed-key metadata fields, such as: Cache-Control, Content-Disposition, and Content-Type.	Preserved as fixed-key metadata.
Amazon S3 user-defined metadata, formatted as key:value pairs. For more information, see the User-defined object metadata section of Object key and metadata.	Preserved as a custom metadata field in destination Cloud Storage objects, which you can edit later or remove.
ETag	Preserved as a custom metadata field with the key x-goog-source-etag, which you can edit later or remove.
Object size.	Preserved as size.
Amazon S3 access control lists (ACLs). For a complete list, see the Condition Keys section of Access Control List (ACL) Overview.	Not preserved.
Amazon S3 object tags, defined by you as key-value pairs. For more information, see Object Tags.	Not preserved.
Amazon S3 system-defined metadata, except for ETag and object size. For a complete list, see the System-defined object metadata section of Object key and metadata.	Not preserved. Timestamp metadata from the source isn't preserved. Creation time, timeCreated, reflects the time that an object is created in Cloud Storage. Similarly, updated reflects the time that metadata for an object is modified in Cloud Storage.

Microsoft Azure Storage to Cloud Storage

Metadata example	Preservation behavior
Microsoft Azure Storage fixed-key metadata fields, such as: Cache-Control, Content-Disposition, and Content-Type.	Preserved as fixed-key metadata.
Microsoft Azure Storage user-defined metadata, formatted as key:value pairs. For more information, see Settings and retrieving properties and metadata for Blob service resources .	Preserved as a custom metadata field in destination Cloud Storage objects, which you can edit later or remove.
ETag	Preserved as a custom metadata field with the key x-goog-source-etag, which you can edit later or remove.
Object size.	Preserved as size.
POSIX file system permissions supported by Azure Data Lake Storage (ADLS) Gen 2.	Not preserved.
Microsoft Azure Storage access control, specifically x-ms-blob-public-access. For more information, see the Response Headers section of Get Container ACL .	Not preserved.
Microsoft Azure Storage index tags. For more information, see Manage and find Azure Blob data with blob index tags .	Not preserved.
Microsoft Azure Storage timestamp metadata, such as: Last-Modified, x-ms-creation-time, x-ms-version, x-ms-request-server-encrypted, and x-ms-encryption-scope. For more information, see Set Blob Metadata .	Not preserved. Timestamp metadata from the source isn't preserved. Creation time, timeCreated, reflects the time that an object is created in Cloud Storage. Similarly, updated reflects the time that metadata for an object is modified in Cloud Storage.

Transfers between Cloud Storage buckets

Metadata example	Preservation behavior
Cloud Storage fixed-key metadata fields, such as: Cache-Control, Content-Disposition, and Content-Type. For more information, see Object metadata	Preserved as fixed-key metadata. eventBasedHold might be overwritten if the destination bucket has a default event-based hold.
Cloud Storage user-defined metadata,formatted as key:value pairs. For more information, see Custom metadata.	Preserved as a custom metadata field in destination Cloud Storage objects, which you can edit later or remove.
Object size	Preserved as size.
Cloud Storage Access Control Lists (ACLs). For more information, see Predefined ACLs.	Not preserved.
Cloud Storage non-editable metadata, such as: generation, etag, componentCount, and storageClass.	Not Preserved.
Cloud Storage timestamp metadata, such as: timeCreated and updated.	Not Preserved. Timestamp metadata from the source isn't preserved. Creation time, timeCreated, reflects the time that an object is created in Cloud Storage. Similarly, updated, reflects the time that metadata for an object is modified in Cloud Storage.

For a list of metadata in Cloud Storage, see Objects.

URL list transfer to Cloud Storage

For more information about URL lists, see Creating a URL list.

Metadata example	Preservation behavior
Fixed-key metadata fields, such as: Cache-Control, Content-Disposition, and Content-Type.	Preserved as editable metadata.
Content-Length and MD5	Preserved as non-editable metadata. If the source doesn't provide an MD5 hash value, then we don't preserve a value. This preservation behavior is specific to Content-Length and MD5. Any other non-editable metadata not listed is not preserved.
Timestamp metadata, such as: Creation time, modified time, and other source-specific metadata.	Not preserved. Timestamp metadata from the source isn't preserved. Creation time, timeCreated, reflects the time that an object is created in Cloud Storage. Similarly, updated reflects the time that metadata for an object is modified in Cloud Storage.

POSIX file system transfers

When transferring files from POSIX filesystems, Transfer service for on-premises data can optionally preserve certain attributes as custom metadata. If these files are later written back to a filesystem, Transfer for on-premises can convert the preserved metadata back to POSIX attributes.

Metadata example	Preservation behavior
Modified time (mtime) and file size.	Preserved. mtime is preserved as custom metadata with the key goog-reserved-file-mtime. File size is preserved as size.
Folder permissions and hard links.	Not preserved. Storage Transfer Service and Transfer service for on-premises data don't create folder placeholder objects in Cloud Storage to represent folders.
Numeric UID, numeric GID, numeric MODE, and symbolic links.	Preservation behavior is specified with the metadataOptions object. See Preserving optional POSIX metadata for details. Default behavior is to not preserve any metadata.

Preserving optional POSIX metadata

Preview — Metadata preservation

This feature is covered by the Pre-GA Offerings Terms of the Google Cloud Terms of Service. Pre-GA features might have limited support, and changes to pre-GA features might not be compatible with other pre-GA versions. For more information, see the launch stage descriptions.

To preserve one or more of numeric UID, numeric GID, numeric MODE, and symbolic links, specify a metadataOptions object in the body of your transfer job.

These options apply to both POSIX-to-Cloud Storage transfers and Cloud Storage-to-POSIX transfers. For the latter, the metadata must have been preserved when files were initially transferred to Cloud Storage.

{
  "description": "metadata-example",
  "projectId": "example-project-id"
  "transferSpec": {
    ...
    "transferOptions": {
      "metadataOptions": {
        "gid":     "GID_NUMBER",       # Default is "GID_SKIP"
        "uid":     "UID_NUMBER",       # Default is "UID_SKIP"
        "mode":    "MODE_PRESERVE",    # Default is "MODE_SKIP"
        "symlink": "SYMLINK_PRESERVE"  # Default is "SYMLINK_SKIP"
      }
    }
  }
}

POSIX to Cloud Storage

Preserved metadata is stored in Cloud Storage as custom metadata key:value pairs.

• Numeric GID is stored as goog-reserved-posix-gid.
• Numeric UID is stored as goog-reserved-posix-uid.
• Numeric MODE is stored as goog-reserved-posix-mode.

For symbolic links, Transfer service for on-premises data preserves the target link as an object in Cloud Storage with the following qualities:

• Object key is composed of the destination prefix plus the path to the symlink, relative to the root_directory.
• Object metadata:
  • Any symlink metadata is preserved as Cloud Storage object metadata.
  • A custom metadata entry is made: goog-reserved-file-is-symlink:true.
• The object content is the target of the symlink. For example, for a symlink sym-> dir1/target, the object's content is "dir1/target".

Transfer for on-premises does not validate the link or copy the target file.

Cloud Storage to POSIX

If metadata is preserved when files are transferred to Cloud Storage, that metadata can be written back to the files when they are transferred back to a POSIX file system.

If a metadata option is set to preserve, Transfer for on-premises takes the following actions:

• Symbolic links: Transfer for on-premises creates a symlink file pointing to the target link. If the target file does not exist, the symlink will be broken.
• GID, UID, and MODE: the values stored in Cloud Storage metadata are written back to the file.