API layers

Introduction

The Google Cloud client libraries have been designed to make simple operations as simple as possible, without sacrificing fine-grained tuning where necessary.

This article describes the different abstraction layers of the libraries, and how to take advantage of each of them at the appropriate time.

Background: gRPC and REST APIs

For a long time, most Google APIs - including those in Google Cloud - have used a REST-based approach, usually using JSON to represent the resources. These APIs have been self-describing using the Google API Discovery Service, which has been used to generate per-API packages under the umbrella of the "API Client Library for .NET".

In November 2014, Google introduced gRPC, a new RPC framework offering efficient multi-plexing, bidirectional streaming and many other features. New APIs are now being provided with access via either the REST-based HTTP 1.1 and JSON, or gRPC over HTTP 2.0 and Protocol Buffers.

The google-cloud-dotnet project provides client libraries for a mixture of these APIs. In particular, Google.Cloud.Storage.V1 and Google.Cloud.BigQuery.V2 are hand-written layers over the existing REST-based libraries for the Google Cloud Storage and BigQuery APIs, whereas other libraries target gRPC-based APIs. Where gRPC access to an API is available, our client libraries use it.

See the library types documentation for more details.

Layers of gRPC-based libraries

All gRPC-based libraries contain generated code, and some also contain hand-written code. The generated code comes from three sources:

To show the layering involved, we'll use the Google.Cloud.Datastore.V1 client library as an example. An application using this library will typically end up with the following assemblies:

  • Google.Cloud.Datastore.V1: the client library itself
  • Google.Api.Gax/Google.Api.Gax.Grpc: support libraries providing functionality required across multiple gRPC-based wrapper libraries, such as page streaming.
  • Google.Protobuf: Protocol Buffer support library
  • Grpc.Auth and Grpc.Core: gRPC support libraries
  • Google.Apis.Auth and Google.Apis.Auth.PlatformServices: support libraries used by both REST service libraries and gRPC libraries for authentication
  • Dependencies from the above libraries for HTTP, JSON serialization, cryptography, and asynchronous sequences

Within Google.Cloud.Datastore.V1 there are three abstraction layers, all within the Google.Cloud.Datastore.V1 namespace, with the following entry points:

  • DatastoreDb is hand-written code to provide the most idiomatic experience. For example, using DatastoreDb avoids the project ID from being specified on every call, and uses the IDisposable pattern to make DatastoreTransaction simple to work with.
  • DatastoreClient is generated by the Google API Tools. This is still a higher level of abstraction than the raw RPC, supporting features such as page streaming and automatic creation of requests from the constituent parameters, but it has less semantic awareness than the hand-written layer. An instance of DatastoreClient is available from a DatastoreDb via the DatastoreDb.Client property if necessary. Code which does not use DatastoreDb at all can simply call DatastoreClient.Create() instead.
  • Datastore.DatastoreClient is the gRPC-generated RPC access class. Very few applications will need to use this, but an instance can be retrieved from the higher level DatastoreClient class via the DatastoreClient.GrpcClient property if necessary.

Additional hand-written code exists in the form of extension methods and additions to the partial classes generated by the Protocol Buffer code generator. This does not act as a separate abstraction layer, but as a set of extensions to the generated code. Examples include the conversion operators on Value, the static factory methods in Filter, the constructors for PartitionId and Query, and extension methods for repeated fields. The extensions certainly don't have to be used, but they can lead to much simpler code.

Different libraries may include different amounts of wrapping. In Datastore, although the DatastoreDb abstraction layer helps simplify code by removing redundant specification of project IDs and allowing mutations to be built up in memory before being committed in a transaction, the objects used to describe the queries and entities are still the Protocol Buffer messages. In other APIs, the entities themselves may be wrapped in a hand-written layer to make them easier to work with. Some libraries may not provide any hand-written code at all: simple APIs are often very usable just with the code generated by the Google API Tools.

Importantly, these abstraction layers are transparent rather than opaque: if a higher level abstraction doesn't provide the level of control you need, you can always use the next level of abstraction down. This approach allows access to all the features of the API while higher level abstractions are built incrementally based on feedback.

Layers of REST-based libraries

The REST-based libraries wrap existing generated code provided in other NuGet packages. For example, Google.Cloud.Storage.V1 depends on Google.Apis.Storage.v1. The wrapper libraries exist to provide a more idiomatic way of working with the APIs than is exposed by the auto-generated code.

Where functionality hasn't been explicitly wrapped, application code can always access the underlying REST service.

To show the layering involved, we'll use the Google.Cloud.Storage.V1 client library as an example. An application using this library will typically end up with the following assemblies:

  • Google.Cloud.Storage.V1: the wrapper library
  • Google.Apis.Storage.v1: the REST service library that the wrapper library uses to call the REST API
  • Google.Api.Gax/Google.Api.Gax.Rest: support libraries providing functionality required across multiple REST-based wrapper libraries, such as page streaming.
  • Google.Apis: support libraries used by all REST service libraries to perform REST-based operations
  • Google.Apis.Auth and Google.Apis.Auth.PlatformServices: support libraries used by both REST service libraries and gRPC libraries for authentication
  • Dependencies from the above libraries for HTTP, logging, JSON serialization, cryptography, and asynchronous sequences

For simple use cases, you can simply use the Google.Cloud.Storage.V1.StorageClient class. If your application code requires functionality which isn't wrapped, you can use the StorageClient.Service property to obtain a reference to a Google.Apis.Storage.v1.StorageService which can be used for all other operations. This is the same StorageService object that the StorageClient uses for its API calls, with whatever authentication information you provided when creating the client.

The intent is that the most commonly-used functionality should be wrapped, however: if you find yourself frequently having to dig below the StorageClient abstraction, please file a feature request with details of what functionality isn't being wrapped at the moment.