Composite objects

Create

This page describes composite objects, which you create from existing objects without transferring additional object data. Composite objects are useful for making appends to an existing object, as well as for recreating objects that you uploaded as multiple components in parallel.

Compose operation

The compose operation concatenates the data in a given sequence of source objects to create a new object called a composite object. The source objects all must:

  • Have the same storage class.
  • Be stored in the same Cloud Storage bucket.

When you perform a composition:

  • The source objects are unaffected.
  • You can use between 1 and 32 source objects.
  • Source objects can themselves be composite objects.

The composite object that results from a composition:

  • Has the same storage class as the source objects.
  • Does not change if the source objects are subsequently replaced or deleted.

When using gcloud storage to perform object composition, the object that results has a Content-Type set to match the Content-Type of the first source object.

Composite object metadata

There are several differences between the metadata of a composite object and the metadata of other objects:

  • Composite objects do not have an MD5 hash metadata field.

  • The ETag value of a composite object is not based on an MD5 hash, and client code should make no assumptions about composite object ETags except that they change whenever the underlying object changes per the IETF specification for HTTP/1.1.

  • Each composite object has a component count metadata field, which counts the number of non-composite objects contained within the composite object.

    • If you rewrite a composite object to a different location or storage class, the result is a composite object with a component count of 1.

Integrity Checking Composite Objects

Cloud Storage uses CRC32C for integrity checking each source object at upload time and for allowing the caller to perform an integrity check of the resulting composite object when it is downloaded. CRC32C is an error detecting code that can be efficiently calculated from the CRC32C values of its components. Your application should use CRC32C as follows:

  • When uploading source objects, you should calculate the CRC32C for each object using a CRC32C library, such as one of those listed in the Object metadata page, and include that value in your request. Based on the values you provide, Cloud Storage validates each upload.
  • The compose operation automatically checks that the source objects are correctly assembled, and it ignores any CRC32C value you provide as part of the compose request. A CRC32C of the resulting composite object is returned in the response.
  • If your application could change source objects between the time of uploading and composing those objects, you should specify generation-specific names for the source objects to avoid race conditions. Alternatively, you can build a CRC32C value from the CRC32C values of the intended source objects and compare it to the CRC32C value returned by the compose operation.
  • At download time, you should calculate the CRC32C of the downloaded object and compare that with the value included in the response.

Limited Append and Edit

You can use the compose operation to perform limited object appends and edits.

You accomplish appending by uploading data to a temporary new object, composing the object you wish to append with this temporary object, optionally naming the output of the compose operation the same as the original object, and deleting the temporary object.

For example, in the gcloud CLI, the series of commands to append the string new data to an existing Cloud Storage object is the following:

$ echo 'new data' | gcloud storage cp - gs://bucket/temporary_object
$ gcloud storage objects compose gs://bucket/object_to_append gs://bucket/temporary_object \
    gs://bucket/object_to_append
$ gcloud storage rm gs://bucket/temporary_object

You can also use composition to support a basic flavor of object editing. For example, you could compose an object X from the sequence {Y1, Y2, Y3}, replace the contents of Y2, and recompose X from those same components. Note that this requires that Y1, Y2, and Y3 be left undeleted, so you will be billed for those components as well as for the composite.

What's next