A client for Google Cloud Storage offering asynchronous operations.
Optional Request Options
Most of the member functions in this class can receive optional request options. For example, the default when deleting an object is to delete the latest version:```cpp auto pending = gcs.DeleteObject("my-bucket", "my-object");
Some applications may want to delete a specific version. In this case just provide the `Generation` request option:
```cpp
auto pending = gcs.DeleteObject(
"my-bucket", "my-object", gcs::Generation(generation));
Each function documents the types accepted as optional request options. These parameters can be specified in any order. Specifying a request option that is not applicable to a member function results in a compile-time error.
All operations support the following common request options:
Fields: return a partial response, which includes only the desired fields.QuotaUser: attribute the request to this specific label for quota purposes.UserProject: change the request costs (if applicable) to this GCP project.CustomHeader: include a custom header with the request. These are typically used for testing, though they are sometimes helpful if environments where HTTPS traffic is mediated by a proxy.UserIp: attribute the request to this specific IP address for quota purpose. Not recommended, preferQuotaUserinstead.
Per-operation Overrides
In addition to the request options, which are passed on to the service to modify the request, you can specify options that override the local behavior of the library. For example, you can override the local retry policy:
auto pending = gcs.DeleteObject(
"my-bucket", "my-object",
google::cloud::Options{}
.set<gcs::RetryPolicyOption>(
gcs::LimitedErrorCountRetryPolicy(5).clone()));
Retry, Backoff, and Idempotency Policies
The library automatically retries requests that fail with transient errors, and follows the recommended practice to backoff between retries.
The default policies are to continue retrying for up to 15 minutes, and to use truncated (at 5 minutes) exponential backoff, doubling the maximum backoff period between retries. Likewise, the idempotency policy is configured to retry all operations.
The application can override these policies when constructing objects of this class. The documentation for the constructors shows examples of this in action.
Constructors
AsyncClient(Options)
Create a new client configured with options.
| Parameter | |
|---|---|
| Name | Description |
options |
Options
|
AsyncClient(std::shared_ptr< AsyncConnection >)
Create a new client using connection. This is often used for mocking.
| Parameter | |
|---|---|
| Name | Description |
connection |
std::shared_ptr< AsyncConnection >
|
Functions
InsertObject(std::string, std::string, Collection &&, Args &&...)
Creates an object given its name and contents.
Selecting an upload function
When choosing an upload method consider the following tradeoffs:
We recommend using InsertObject() for relatively small objects that fit in memory.
- Pro: Easy to use, a single function call uploads the object.
- Pro: Lowest latency for small objects. Use <= 4MiB as a rule of thumb. The precise threshold depends on your environment.
- Con: Recovery from transient errors requires resending all the data.
- Con: Multiple concurrent calls to
InsertObject()will consume as much memory as is needed to hold all the data.
We recommend using StartBufferedUpload() to upload data of unknown or arbitrary size.
- Pro: Relatively easy to use, the library can automatically resend data under most transient errors.
- Pro: The application can limit the amount of memory used by each upload, even if the full object is arbitrarily large.
- Pro: Can be used to upload "streaming" data sources where it is inefficient or impossible to go back and re-read data from an arbitrary point.
- Con: Throughput is limited as it needs to periodically wait for the service to flush the buffer to persistent storage.
- Con: Cannot automatically resume uploads after the application restarts.
We recommend using StartUnbufferedUpload() to upload data where the upload can efficiently resume from arbitrary points.
- Pro: Can achieve the maximum theoretical throughput for a single stream upload. It is possible to use Parallel Composite Uploads to achieve even higher throughput.
- Pro: It can resume uploads even after the application restarts.
- Con: Requires manually handling transient errors during the upload.
Idempotency
This operation is only idempotent if restricted by pre-conditions, in this case, IfGenerationMatch.
Example
namespace gcs_ex = google::cloud::storage_experimental;
[](gcs_ex::AsyncClient& client, std::string bucket_name,
std::string object_name) {
auto object =
client.InsertObject(std::move(bucket_name), std::move(object_name),
std::string("Hello World!\n"));
// Attach a callback, this is called when the upload completes.
auto done = object.then([](auto f) {
auto metadata = f.get();
if (!metadata) throw std::move(metadata).status();
std::cerr << "Object successfully inserted " << *metadata << "\n";
});
// To simplify example, block until the operation completes.
done.get();
}
| Parameters | |
|---|---|
| Name | Description |
bucket_name |
std::string
the name of the bucket that will contain the object. |
object_name |
std::string
the name of the object to be created. |
contents |
Collection &&
the contents (media) for the new object. |
args |
Args &&...
a list of optional query parameters and/or request headers. Valid types for this operation include |
typename Collection |
the type for the payload. This must be convertible to |
typename... |
|
| Returns | |
|---|---|
| Type | Description |
future< StatusOr< storage::ObjectMetadata > > | |
ReadObject(std::string, std::string, Args &&...)
Reads the contents of an object.
When satisfied, the returned future has a reader to asynchronously download the contents of the given object.
Idempotency
This is a read-only operation and is always idempotent.
Example
namespace gcs_ex = google::cloud::storage_experimental;
auto coro =
[](gcs_ex::AsyncClient& client, std::string bucket_name,
std::string object_name) -> google::cloud::future<std::uint64_t> {
auto [reader, token] = (co_await client.ReadObject(std::move(bucket_name),
std::move(object_name)))
.value();
std::uint64_t count = 0;
while (token.valid()) {
auto [payload, t] = (co_await reader.Read(std::move(token))).value();
token = std::move(t);
for (auto const& buffer : payload.contents()) {
count += std::count(buffer.begin(), buffer.end(), '\n');
}
}
co_return count;
};
| Parameters | |
|---|---|
| Name | Description |
bucket_name |
std::string
the name of the bucket that contains the object. |
object_name |
std::string
the name of the object to be read. |
args |
Args &&...
a list of optional query parameters and/or request headers. Valid types for this operation include |
typename... |
|
| Returns | |
|---|---|
| Type | Description |
future< StatusOr< std::pair< AsyncReader, AsyncToken > > > | |
ReadObjectRange(std::string, std::string, std::int64_t, std::int64_t, Args &&...)
Reads the contents of an object.
When satisfied, the returned future has the contents of the given object between offset and offset + limit (exclusive).
Be aware that this will accumulate all the bytes in memory, you need to consider whether limit is too large for your deployment environment.
Idempotency
This is a read-only operation and is always idempotent.
| Parameters | |
|---|---|
| Name | Description |
bucket_name |
std::string
the name of the bucket that contains the object. |
object_name |
std::string
the name of the object to be read. |
offset |
std::int64_t
where to begin reading from the object, results in an error if the offset is larger than the object |
limit |
std::int64_t
how much data to read starting at |
args |
Args &&...
a list of optional query parameters and/or request headers. Valid types for this operation include |
typename... |
|
| Returns | |
|---|---|
| Type | Description |
future< StatusOr< ReadPayload > > | |
StartUnbufferedUpload(std::string, std::string, Args &&...)
Uploads a new object without buffering.
We recommend using InsertObject() for relatively small objects that fit in memory.
- Pro: Easy to use, a single function call uploads the object.
- Pro: Lowest latency for small objects. Use <= 4MiB as a rule of thumb. The precise threshold depends on your environment.
- Con: Recovery from transient errors requires resending all the data.
- Con: Multiple concurrent calls to
InsertObject()will consume as much memory as is needed to hold all the data.We recommend usingStartBufferedUpload()to upload data of unknown or arbitrary size. - Pro: Relatively easy to use, the library can automatically resend data under most transient errors.
- Pro: The application can limit the amount of memory used by each upload, even if the full object is arbitrarily large.
- Pro: Can be used to upload "streaming" data sources where it is inefficient or impossible to go back and re-read data from an arbitrary point.
- Con: Throughput is limited as it needs to periodically wait for the service to flush the buffer to persistent storage.
- Con: Cannot automatically resume uploads after the application restarts.We recommend using
StartUnbufferedUpload()to upload data where the upload can efficiently resume from arbitrary points. - Pro: Can achieve the maximum theoretical throughput for a single stream upload. It is possible to use Parallel Composite Uploads to achieve even higher throughput.
- Pro: It can resume uploads even after the application restarts.
- Con: Requires manually handling transient errors during the upload.
This function always uses resumable uploads. The application can provide a #RestoreResumableUploadSession() option to resume a previously created upload. The returned object has accessors to query the session ID and the amount of data persisted by the service.
When resuming uploads it is the application's responsibility to save the session ID to restart the upload later. Likewise, it is the application's responsibility to query the amount of data already persisted and to send the remaining data without gaps or duplications.
If the application does not provide a #RestoreResumableUploadSession() option, or it provides the #NewResumableUploadSession() option, then this function will create a new resumable upload session.
Idempotency
The client library always retries (a) any RPCs to create a resumable upload session, and (b) any RPCs to query the status of a resumable upload session. The client library never retries functions to upload more data or to finalize the upload. The caller can retry these functions if it is safe to do so.
Example
namespace gcs = google::cloud::storage;
namespace gcs_ex = google::cloud::storage_experimental;
auto coro = [](gcs_ex::AsyncClient& client, std::string bucket_name,
std::string object_name, std::string const& filename)
-> google::cloud::future<gcs::ObjectMetadata> {
std::ifstream is(filename);
if (is.bad()) throw std::runtime_error("Cannot read " + filename);
auto [writer, token] = (co_await client.StartUnbufferedUpload(
std::move(bucket_name), std::move(object_name)))
.value();
is.seekg(0); // clear EOF bit
while (token.valid() && !is.eof()) {
std::vector<char> buffer(1024 * 1024);
is.read(buffer.data(), buffer.size());
buffer.resize(is.gcount());
token = (co_await writer.Write(std::move(token),
gcs_ex::WritePayload(std::move(buffer))))
.value();
}
co_return (co_await writer.Finalize(std::move(token))).value();
};
Example
namespace gcs = google::cloud::storage;
namespace gcs_ex = google::cloud::storage_experimental;
// Make one attempt to upload `is` using `writer`:
auto attempt =
[](gcs_ex::AsyncWriter& writer, gcs_ex::AsyncToken& token,
std::istream& is) -> google::cloud::future<google::cloud::Status> {
while (token.valid() && !is.eof()) {
std::vector<char> buffer(1024 * 1024);
is.read(buffer.data(), buffer.size());
buffer.resize(is.gcount());
auto w = co_await writer.Write(std::move(token),
gcs_ex::WritePayload(std::move(buffer)));
if (!w) co_return std::move(w).status();
token = *std::move(w);
}
co_return google::cloud::Status{};
};
// Make multiple attempts to upload the file contents, restarting from the
// last checkpoint on (partial) failures.
auto coro = [&attempt](
gcs_ex::AsyncClient& client, std::string const& bucket_name,
std::string const& object_name, std::string const& filename)
-> google::cloud::future<gcs::ObjectMetadata> {
std::ifstream is(filename);
if (is.bad()) throw std::runtime_error("Cannot read " + filename);
// The first attempt will create a resumable upload session.
auto upload_id = gcs::UseResumableUploadSession();
for (int i = 0; i != 3; ++i) {
// Start or resume the upload.
auto [writer, token] = (co_await client.StartUnbufferedUpload(
bucket_name, object_name, upload_id))
.value();
// If the upload already completed, there is nothing left to do.
auto state = writer.PersistedState();
if (absl::holds_alternative<gcs::ObjectMetadata>(state)) {
co_return absl::get<gcs::ObjectMetadata>(std::move(state));
}
// Refresh the upload id and reset the input source to the right offset.
upload_id = gcs::UseResumableUploadSession(writer.UploadId());
is.seekg(absl::get<std::int64_t>(std::move(state)));
auto status = co_await attempt(writer, token, is);
if (!status.ok()) continue; // Error in upload, try again.
auto f = co_await writer.Finalize(std::move(token));
if (f) co_return *std::move(f); // Return immediately on success.
}
throw std::runtime_error("Too many upload attempts");
};
See Also
Resumable Uploads for more information about resumable uploads.
| Parameters | |
|---|---|
| Name | Description |
bucket_name |
std::string
the name of the bucket that contains the object. |
object_name |
std::string
the name of the object to be read. |
args |
Args &&...
a list of optional query parameters and/or request headers. Valid types for this operation include |
typename... |
|
| Returns | |
|---|---|
| Type | Description |
future< StatusOr< std::pair< AsyncWriter, AsyncToken > > > | |
StartBufferedUpload(std::string, std::string, Args &&...)
Uploads a new object with buffering and automatic recovery from transient failures.
We recommend using InsertObject() for relatively small objects that fit in memory.
- Pro: Easy to use, a single function call uploads the object.
- Pro: Lowest latency for small objects. Use <= 4MiB as a rule of thumb. The precise threshold depends on your environment.
- Con: Recovery from transient errors requires resending all the data.
- Con: Multiple concurrent calls to
InsertObject()will consume as much memory as is needed to hold all the data.We recommend usingStartBufferedUpload()to upload data of unknown or arbitrary size. - Pro: Relatively easy to use, the library can automatically resend data under most transient errors.
- Pro: The application can limit the amount of memory used by each upload, even if the full object is arbitrarily large.
- Pro: Can be used to upload "streaming" data sources where it is inefficient or impossible to go back and re-read data from an arbitrary point.
- Con: Throughput is limited as it needs to periodically wait for the service to flush the buffer to persistent storage.
- Con: Cannot automatically resume uploads after the application restarts.We recommend using
StartUnbufferedUpload()to upload data where the upload can efficiently resume from arbitrary points. - Pro: Can achieve the maximum theoretical throughput for a single stream upload. It is possible to use Parallel Composite Uploads to achieve even higher throughput.
- Pro: It can resume uploads even after the application restarts.
- Con: Requires manually handling transient errors during the upload.
This function always uses resumable uploads. The application can provide a #RestoreResumableUploadSession() option to resume a previously created upload. The returned object has accessors to query the session ID and the amount of data persisted by the service.
When resuming uploads it is the application's responsibility to save the session ID to restart the upload later. Likewise, it is the application's responsibility to query the amount of data already persisted and to send the remaining data without gaps or duplications.
If the application does not provide a #RestoreResumableUploadSession() option, or it provides the #NewResumableUploadSession() option, then this function will create a new resumable upload session.
Idempotency
The client library always retries (a) any RPCs to create a resumable upload session, and (b) any RPCs to query the status of a resumable upload session.
See Also
Resumable Uploads for more information about resumable uploads.
| Parameters | |
|---|---|
| Name | Description |
bucket_name |
std::string
the name of the bucket that contains the object. |
object_name |
std::string
the name of the object to be read. |
args |
Args &&...
a list of optional query parameters and/or request headers. Valid types for this operation include |
typename... |
|
| Returns | |
|---|---|
| Type | Description |
future< StatusOr< std::pair< AsyncWriter, AsyncToken > > > | |
ComposeObject(std::string, std::vector< storage::ComposeSourceObject >, std::string, Args &&...)
Composes existing objects into a new object in the same bucket.
Idempotency
This operation is only idempotent if restricted by pre-conditions, in this case, IfGenerationMatch.
| Parameters | |
|---|---|
| Name | Description |
bucket_name |
std::string
the name of the bucket used for source object and destination object. |
source_objects |
std::vector< storage::ComposeSourceObject >
objects used to compose |
destination_object_name |
std::string
the composed object name. |
args |
Args &&...
a list of optional query parameters and/or request headers. Valid types for this operation include |
typename... |
|
| Returns | |
|---|---|
| Type | Description |
future< StatusOr< storage::ObjectMetadata > > | |
DeleteObject(std::string, std::string, Args &&...)
Deletes an object.
Idempotency
This operation is only idempotent if:
- restricted by pre-conditions, in this case,
IfGenerationMatch - or, if it applies to only one object version via
Generation.
Example
namespace g = google::cloud;
namespace gcs_ex = google::cloud::storage_experimental;
[](gcs_ex::AsyncClient& client, std::string bucket_name,
std::string object_name) {
client.DeleteObject(std::move(bucket_name), std::move(object_name))
.then([](auto f) {
auto status = f.get();
if (!status.ok()) throw g::Status(std::move(status));
std::cout << "Object successfully deleted\n";
})
.get();
}
| Parameters | |
|---|---|
| Name | Description |
bucket_name |
std::string
the name of the bucket that contains the object. |
object_name |
std::string
the name of the object to be deleted. |
args |
Args &&...
a list of optional query parameters and/or request headers. Valid types for this operation include |
typename... |
|
| Returns | |
|---|---|
| Type | Description |
future< Status > | |