Object metadata

Manage

This page discusses the commonly-used metadata fields that are stored along with objects in Cloud Storage.

Introduction

Objects stored in Cloud Storage have metadata associated with them. Metadata identifies properties of the object, as well as specifies how the object should be handled when it's accessed. Metadata exists as key:value pairs. For example, the storage class of an object is represented by the metadata entry storageClass:STANDARD. storageClass is the key for the metadata, and all objects have such a key associated with them. STANDARD specifies the value this specific object has, and the value varies from object to object.

The mutability of metadata varies: some metadata you can edit at any time, some metadata you can only set at the time the object is created, and some metadata you can only view. For example, you can edit the value of the Cache-Control metadata at any time, but you can only assign the storageClass metadata when the object is created or rewritten, and you cannot directly edit the value for the generation metadata, though the generation value changes when the object is replaced.

Editable metadata

There are two categories of metadata that users can change for objects:

  • Fixed-key metadata: Metadata whose keys are set, but for which you can specify a value.

  • Custom metadata: Metadata that you add by specifying both a key and a value associated with the key.

When editing metadata, you should generally avoid non-ascii characters, because they are not permitted in HTTP headers, which the XML API uses.

Fixed-key metadata

You can edit the following metadata for objects, though you must have sufficient permission to do so:

Access control metadata

Cloud Storage uses Identity and Access Management (IAM) and Access Control Lists (ACLs) to control access to objects. Use these links to learn about these access control methods and associated metadata.

Cache-Control

The Cache-Control metadata can specify two different aspects of how data is served from Cloud Storage: whether the data can be cached and whether the data can be transformed.

Caching data

The Cache-Control metadata lets you control whether and for how long caches are allowed to cache your objects, which can then be served to satisfy future requests. Caches can include browser and Internet caches, as well as Cloud Storage built-in caching.

If an applicable object doesn't have a Cache-Control metadata entry, Cloud Storage uses the default value of:

If you allow caching, downloads might continue to receive earlier versions of an object, even after uploading a newer version. This is because such earlier versions remain "fresh" in caches for a period of time determined by max-age. Additionally, because objects can be cached at various places on the Internet, there is no way to force a cached object to expire globally. This means that if you revoke public access to an object, the object can still be served from a cache, depending on when it was last accessed and its Cache-Control setting. For example, if your object was served with a Cache-Control of public, max-age=3600, it can persist in a cache for an hour. If you want to prevent serving cached versions of publicly readable objects, set Cache-Control: no-store on the object.

If you need greater control over cache behavior, you can configure Cloud CDN in front of your bucket.

Transforming data

Cache-Control metadata also lets you serve objects as they are stored, without applying any transformations to the data, such as removing gzip content-encoding for incompatible clients. To serve an object as-is, set Cache-Control:no-transform.

Content-Disposition

The Content-Disposition metadata specifies presentation information about the data being transmitted. Setting Content-Disposition allows you to control presentation style of the content, for example determining whether an attachment should be automatically displayed or whether some form of action from the user should be required to open it. See https://datatracker.ietf.org/doc/html/rfc6266 for the Content-Disposition specification.

Content-Encoding

The Content-Encoding metadata can be used to indicate that an object is compressed, while still maintaining the object's underlying Content-Type. For example, a text file that is gzip compressed can have the fact that it's a text file indicated in Content-Type and the fact that it's gzip compressed indicated in Content-Encoding. You should ensure that files are, in fact, compressed using the specified Content-Encoding before uploading them, or else unexpected behavior can occur when attempting to download the objects. For more information, see the Transcoding page.

For compressible content, such as text, using Content-Encoding: gzip saves network and storage costs and improves content serving performance. However, for content that is already inherently compressed, such as archives and many media formats, applying another level of compression and marking it in the Content-Encoding metadata is typically detrimental to both object size and performance and should be avoided.

Content-Language

The Content-Language metadata indicates the language(s) that the object is intended for. Refer to ISO 639-1 language codes for typical values of this metadata.

Cloud Storage supports Content-Language values up to 100 characters in length.

Content-Type

The most commonly set metadata is Content-Type (also known as media type), which lets browsers render the object properly. All objects have a value specified in their Content-Type metadata, but this value does not have to match the underlying type of the object. For example, if the Content-Type is not specified by the uploader and cannot be determined, it is set to application/octet-stream or application/x-www-form-urlencoded, depending on how you uploaded the object. For a list of valid content types, see the IANA Media Types page.

Custom-Time

The Custom-Time metadata is a user-specified date and time represented in the RFC 3339 format YYYY-MM-DD'T'HH:MM:SS.SS'Z' or YYYY-MM-DD'T'HH:MM:SS'Z' when milliseconds are zero. This metadata is typically set in order to use the DaysSinceCustomTime condition in Object Lifecycle Management.

You cannot remove Custom-Time once it's been set on an object. Additionally, the value for Custom-Time cannot decrease. That is, you cannot set Custom-Time to be an earlier date/time than the existing Custom-Time. You can, however, effectively remove or reset the Custom-Time by rewriting the object.

Object holds

Use metadata flags to place object holds, which prevent objects from being deleted or replaced. For more information, see the Object holds page.

Retention configuration

When present, an object's retention configuration defines a date and time prior to which the object cannot be deleted or replaced. For more information, see Object Retention Lock.

Custom metadata

Custom metadata is metadata for which you define both the key and the value. To create custom metadata, you specify both a key and a value. After you have created a custom metadata key:value pair, you can delete the key or change the value.

Custom metadata is subject to a size limit and incurs storage costs.

The Viewing and Editing Metadata page includes information about setting custom metadata.

The x-goog-meta- prefix

The XML API sets and retrieves object metadata using request headers, and the JSON API allows setting custom metadata in the final request of a resumable upload by using request headers. In order to clearly distinguish custom metadata headers from standard request headers, both APIs prefix such custom metadata headers with x-goog-meta-.

Non-editable metadata

Some metadata cannot be edited directly. This metadata is set at the time of object creation or rewrite. As part of the object creation or rewrite, you can set some such metadata, such as the storage class of the object or customer-managed encryption keys. Other metadata is automatically added and can only be viewed, such the generation number of the object or the time of creation.

Generation and metageneration numbers

As part of its metadata, every Cloud Storage object has a numeric generation property and a numeric metageneration property that uniquely identifies the object:

Property Description
generation Identifies the version of an object, and exists for every object, regardless of whether a bucket uses Object Versioning.
  • An object version's generation value never changes. A new object with the same name can replace an existing object, but the new object always has a different generation assigned to it.
  • There is no guarantee that generation numbers increase for successive versions, only that each new version has a unique generation number.
  • There is no relationship between the generation numbers of unrelated objects, even if the objects are in the same bucket.
metageneration Identifies the metadata version, and increases every time the metadata of a given generation is updated.
  • metageneration begins at 1 for each new generation of an object.
  • The metageneration property has no meaning without the generation property and should be used only in conjunction with it; it's meaningless to compare metadata generations of two object versions.

The generation and metageneration properties are used in the following circumstances:

Checksums

Checksums are metadata calculated from the associated object's data. Checksums are used for validating that object data isn't corrupted. Cloud Storage objects have several checksum metadata fields.

CRC32C

All Cloud Storage objects have a CRC32C hash. Libraries for computing CRC32C include:

The Base64 encoded CRC32C is in big-endian byte order.

MD5

Cloud Storage objects have an MD5 hash if they meet the following criteria:

This hash only applies to a complete object, so it cannot be used to integrity check partial downloads caused by performing a range GET.

ETags

All Cloud Storage objects have an ETag. However, the same object can have a different ETag value when it's requested from the XML API compared to the JSON API. In most cases, users should make no assumptions about the value used in an ETag except that it changes whenever the underlying data or metadata changes, per the specification.

An object's ETag header returns the object's MD5 value if all the following are true:

Modification time

As part of its metadata, every Cloud Storage object has an updated property that indicates the last time the object's metadata was modified. The updated time is set initially to the object's creation time and then changes whenever any metadata of the object changes. This includes changes made by a requester, such as modifying custom metadata, as well as changes made by Cloud Storage on behalf of a requester, such as changing the storage class based on an Object Lifecycle Configuration.

What's next