A client for Google Cloud Storage offering asynchronous operations.
Optional Request Options
Most of the member functions in this class can receive optional request options. For example, the default when deleting an object is to delete the latest version:```cpp auto pending = gcs.DeleteObject("my-bucket", "my-object");
Some applications may want to delete a specific version. In this case just provide the `Generation` request option:
```cpp
auto pending = gcs.DeleteObject(
"my-bucket", "my-object", gcs::Generation(generation));
Each function documents the types accepted as optional request options. These parameters can be specified in any order. Specifying a request option that is not applicable to a member function results in a compile-time error.
All operations support the following common request options:
Fields: return a partial response, which includes only the desired fields.QuotaUser: attribute the request to this specific label for quota purposes.UserProject: change the request costs (if applicable) to this GCP project.CustomHeader: include a custom header with the request. These are typically used for testing, though they are sometimes helpful if environments where HTTPS traffic is mediated by a proxy.UserIp: attribute the request to this specific IP address for quota purpose. Not recommended, preferQuotaUserinstead.
Per-operation Overrides
In addition to the request options, which are passed on to the service to modify the request, you can specify options that override the local behavior of the library. For example, you can override the local retry policy:
auto pending = gcs.DeleteObject(
"my-bucket", "my-object",
google::cloud::Options{}
.set<gcs::RetryPolicyOption>(
gcs::LimitedErrorCountRetryPolicy(5).clone()));
Retry, Backoff, and Idempotency Policies
The library automatically retries requests that fail with transient errors, and follows the recommended practice to backoff between retries.
The default policies are to continue retrying for up to 15 minutes, and to use truncated (at 5 minutes) exponential backoff, doubling the maximum backoff period between retries. Likewise, the idempotency policy is configured to retry all operations.
The application can override these policies when constructing objects of this class. The documentation for the constructors shows examples of this in action.
Constructors
AsyncClient(Options)
Create a new client configured with options.
| Parameter | |
|---|---|
| Name | Description |
options |
Options
|
AsyncClient(std::shared_ptr< AsyncConnection >)
Create a new client using connection. This is often used for mocking.
| Parameter | |
|---|---|
| Name | Description |
connection |
std::shared_ptr< AsyncConnection >
|
Functions
InsertObject(std::string, std::string, Collection &&, Args &&...)
Creates an object given its name and contents.
Idempotency
This operation is only idempotent if restricted by pre-conditions, in this case, IfGenerationMatch.
Example
namespace gcs_ex = google::cloud::storage_experimental;
[](gcs_ex::AsyncClient& client, std::string bucket_name,
std::string object_name) {
auto object =
client.InsertObject(std::move(bucket_name), std::move(object_name),
std::string("Hello World!\n"));
// Attach a callback, this is called when the upload completes.
auto done = object.then([](auto f) {
auto metadata = f.get();
if (!metadata) throw std::move(metadata).status();
std::cerr << "Object successfully inserted " << *metadata << "\n";
});
// To simplify example, block until the operation completes.
done.get();
}
| Parameters | |
|---|---|
| Name | Description |
bucket_name |
std::string
the name of the bucket that will contain the object. |
object_name |
std::string
the name of the object to be created. |
contents |
Collection &&
the contents (media) for the new object. |
args |
Args &&...
a list of optional query parameters and/or request headers. Valid types for this operation include |
typename Collection |
the type for the payload. This must be convertible to |
typename... |
|
| Returns | |
|---|---|
| Type | Description |
future< StatusOr< storage::ObjectMetadata > > |
|
ReadObject(std::string, std::string, Args &&...)
Reads the contents of an object.
When satisfied, the returned future has a reader to asynchronously download the contents of the given object.
Idempotency
This is a read-only operation and is always idempotent.
Example
namespace gcs_ex = google::cloud::storage_experimental;
auto coro =
[](gcs_ex::AsyncClient& client, std::string bucket_name,
std::string object_name) -> google::cloud::future<std::uint64_t> {
auto [reader, token] = (co_await client.ReadObject(std::move(bucket_name),
std::move(object_name)))
.value();
std::uint64_t count = 0;
while (token.valid()) {
auto [payload, t] = (co_await reader.Read(std::move(token))).value();
token = std::move(t);
for (auto const& buffer : payload.contents()) {
count += std::count(buffer.begin(), buffer.end(), '\n');
}
}
co_return count;
};
| Parameters | |
|---|---|
| Name | Description |
bucket_name |
std::string
the name of the bucket that contains the object. |
object_name |
std::string
the name of the object to be read. |
args |
Args &&...
a list of optional query parameters and/or request headers. Valid types for this operation include |
typename... |
|
| Returns | |
|---|---|
| Type | Description |
future< StatusOr< std::pair< AsyncReader, AsyncToken > > > |
|
ReadObjectRange(std::string, std::string, std::int64_t, std::int64_t, Args &&...)
Reads the contents of an object.
When satisfied, the returned future has the contents of the given object between offset and offset + limit (exclusive).
Be aware that this will accumulate all the bytes in memory, you need to consider whether limit is too large for your deployment environment.
Idempotency
This is a read-only operation and is always idempotent.
| Parameters | |
|---|---|
| Name | Description |
bucket_name |
std::string
the name of the bucket that contains the object. |
object_name |
std::string
the name of the object to be read. |
offset |
std::int64_t
where to begin reading from the object, results in an error if the offset is larger than the object |
limit |
std::int64_t
how much data to read starting at |
args |
Args &&...
a list of optional query parameters and/or request headers. Valid types for this operation include |
typename... |
|
| Returns | |
|---|---|
| Type | Description |
future< StatusOr< ReadPayload > > |
|
WriteObject(std::string, std::string, Args &&...)
Uploads a new object without buffering.
Use this API to upload objects without any client-side buffering. The API permits resuming an upload after a transient error. If the application can checkpoint a simple string, the API can even be used to resume an upload, even after the application restarts.
In both cases this API assumes the application can start sending data from an arbitrary point in the data source. That makes this API best suited to upload persistent data sources, such as files, of arbitrary size.
To upload relatively small objects consider using InsertObject instead. To upload streaming data sources, where rewinding to an arbitrary point may be impossible, consider using StreamingUpload().
This function always uses resumable uploads. The application can provide a #RestoreResumableUploadSession() option to resume a previously created upload. The returned object has accessors to query the session ID and the next byte expected by GCS.
#RestoreResumableUploadSession() option, or it provides the #NewResumableUploadSession() option, then this function will create a new resumable upload session.
Idempotency
The client library always retries (a) any RPCs to create a resumable upload session, and (b) any RPCs to query the status of a resumable upload session. The client library never retries functions to upload more data or to finalize the upload. The caller can retry these functions if it is safe to do so.
Example
namespace gcs = google::cloud::storage;
namespace gcs_ex = google::cloud::storage_experimental;
auto coro = [](gcs_ex::AsyncClient& client, std::string bucket_name,
std::string object_name, std::string const& filename)
-> google::cloud::future<gcs::ObjectMetadata> {
std::ifstream is(filename);
if (is.bad()) throw std::runtime_error("Cannot read " + filename);
auto [writer, token] = (co_await client.WriteObject(std::move(bucket_name),
std::move(object_name)))
.value();
is.seekg(0); // clear EOF bit
while (token.valid() && !is.eof()) {
std::vector<char> buffer(1024 * 1024);
is.read(buffer.data(), buffer.size());
buffer.resize(is.gcount());
token = (co_await writer.Write(std::move(token),
gcs_ex::WritePayload(std::move(buffer))))
.value();
}
co_return (co_await writer.Finalize(std::move(token))).value();
};
Example
namespace gcs = google::cloud::storage;
namespace gcs_ex = google::cloud::storage_experimental;
// Make one attempt to upload `is` using `writer`:
auto attempt =
[](gcs_ex::AsyncWriter& writer, gcs_ex::AsyncToken& token,
std::istream& is) -> google::cloud::future<google::cloud::Status> {
while (token.valid() && !is.eof()) {
std::vector<char> buffer(1024 * 1024);
is.read(buffer.data(), buffer.size());
buffer.resize(is.gcount());
auto w = co_await writer.Write(std::move(token),
gcs_ex::WritePayload(std::move(buffer)));
if (!w) co_return std::move(w).status();
token = *std::move(w);
}
co_return google::cloud::Status{};
};
// Make multiple attempts to upload the file contents, restarting from the
// last checkpoint on (partial) failures.
auto coro = [&attempt](
gcs_ex::AsyncClient& client, std::string const& bucket_name,
std::string const& object_name, std::string const& filename)
-> google::cloud::future<gcs::ObjectMetadata> {
std::ifstream is(filename);
if (is.bad()) throw std::runtime_error("Cannot read " + filename);
// The first attempt will create a resumable upload session.
auto upload_id = gcs::UseResumableUploadSession();
for (int i = 0; i != 3; ++i) {
// Start or resume the upload.
auto [writer, token] =
(co_await client.WriteObject(bucket_name, object_name, upload_id))
.value();
// If the upload already completed, there is nothing left to do.
auto state = writer.PersistedState();
if (absl::holds_alternative<gcs::ObjectMetadata>(state)) {
co_return absl::get<gcs::ObjectMetadata>(std::move(state));
}
// Refresh the upload id and reset the input source to the right offset.
upload_id = gcs::UseResumableUploadSession(writer.UploadId());
is.seekg(absl::get<std::int64_t>(std::move(state)));
auto status = co_await attempt(writer, token, is);
if (!status.ok()) continue; // Error in upload, try again.
auto f = co_await writer.Finalize(std::move(token));
if (f) co_return *std::move(f); // Return immediately on success.
}
throw std::runtime_error("Too many upload attempts");
};
See Also
Resumable Uploads for more information about resumable uploads.
| Parameters | |
|---|---|
| Name | Description |
bucket_name |
std::string
the name of the bucket that contains the object. |
object_name |
std::string
the name of the object to be read. |
args |
Args &&...
a list of optional query parameters and/or request headers. Valid types for this operation include |
typename... |
|
| Returns | |
|---|---|
| Type | Description |
future< StatusOr< std::pair< AsyncWriter, AsyncToken > > > |
|
ComposeObject(std::string, std::vector< storage::ComposeSourceObject >, std::string, Args &&...)
Composes existing objects into a new object in the same bucket.
Idempotency
This operation is only idempotent if restricted by pre-conditions, in this case, IfGenerationMatch.
| Parameters | |
|---|---|
| Name | Description |
bucket_name |
std::string
the name of the bucket used for source object and destination object. |
source_objects |
std::vector< storage::ComposeSourceObject >
objects used to compose |
destination_object_name |
std::string
the composed object name. |
args |
Args &&...
a list of optional query parameters and/or request headers. Valid types for this operation include |
typename... |
|
| Returns | |
|---|---|
| Type | Description |
future< StatusOr< storage::ObjectMetadata > > |
|
DeleteObject(std::string, std::string, Args &&...)
Deletes an object.
Idempotency
This operation is only idempotent if:
- restricted by pre-conditions, in this case,
IfGenerationMatch - or, if it applies to only one object version via
Generation.
Example
namespace g = google::cloud;
namespace gcs_ex = google::cloud::storage_experimental;
[](gcs_ex::AsyncClient& client, std::string bucket_name,
std::string object_name) {
client.DeleteObject(std::move(bucket_name), std::move(object_name))
.then([](auto f) {
auto status = f.get();
if (!status.ok()) throw g::Status(std::move(status));
std::cout << "Object successfully deleted\n";
})
.get();
}
| Parameters | |
|---|---|
| Name | Description |
bucket_name |
std::string
the name of the bucket that contains the object. |
object_name |
std::string
the name of the object to be deleted. |
args |
Args &&...
a list of optional query parameters and/or request headers. Valid types for this operation include |
typename... |
|
| Returns | |
|---|---|
| Type | Description |
future< Status > |
|