Transcoding HTTP/JSON to gRPC

Google Cloud Endpoints supports protocol transcoding so that clients can access your gRPC API using HTTP/JSON. The Extensible Service Proxy (ESP) transcodes HTTP/JSON to gRPC.

This guide describes:

  • How to use annotations in your .proto file to specify data conversion from HTTP/JSON to gRPC
  • How to deploy your service in Cloud Endpoints to use this feature
  • Where to find more reference information about designing and implementing transcoding for gRPC services

It assumes that you have already completed our gRPC Tutorials and are familiar with basic Cloud Endpoints for gRPC APIs concepts.

Transcoding is currently supported only for gRPC services: the OpenAPI specification cannot be used to provide transcoding details.

How do I design a transcoding-friendly API?

Transcoding involves mapping HTTP/JSON requests and their parameters to gRPC methods and their parameters and return types (we'll look at exactly how you do this in the following sections). Because of this, while it's possible to map an HTTP/JSON request to any arbitrary API method, it's simplest and most intuitive to do so if the gRPC API itself is structured in a resource-oriented way, just like a traditional HTTP REST API. In other words, the API service should be designed so that it uses a small number of standard methods (corresponding to HTTP verbs like GET, PUT, and so on) that operate on the service's resources (and collections of resources, which are themselves a type of resource). These standard methods are List, Get, Create, Update, and Delete.

If necessary, the API can also have some non-standard custom methods, though these aren't quite as straightforward to map.

You can find out much more about resource oriented design and standard transcoding mappings in our API Design Guide: this is the design standard that's followed in Google when designing public APIs like Cloud APIs. While you don't need to follow this guide to use gRPC transcoding, we strongly recommend it. In particular, the following pages will help you understand these design principles and add useful transcoding mappings to your methods:

The following reference page may also be useful:

In the rest of this document, we're going to use the Bookstore example we used in our Tutorials, which already uses these principles: the Bookstore has "shelf" collections of "book" resources, which users can list, get, create, or delete. In general, we'll assume that when you're using transcoding, you've followed these design ideas when defining your API.

Where to configure transcoding

gRPC transcoding is enabled by default, and you can use it without any configuration at all. If you send an HTTP POST request to the URL path <GRPC_SERVICE_FULL_NAME>/<method_name> with the method's request message field values (if any) as JSON in the HTTP request body, ESP will send the request message to the appropriate gRPC method. For example, if you send a POST to the Bookstore's ListShelves URL as follows:

curl -XPOST http://mydomain/endpoints.example.bookstore.Bookstore/ListShelves

..you should get a current list of shelves back in JSON form.

However, it's strongly preferable in terms of HTTP interface design to explicitly configure mappings, as described in the rest of this document.

The gRPC API Configuration standard for service configuration lets you specify exactly how data should be translated from HTTP/JSON to gRPC. We support two mechanisms for doing this: direct annotations in your .proto file, and in YAML as part of your gRPC API Configuration file. We recommend using proto annotations for ease of reading and maintenance. For more information on YAML configuration and when you might need to use it, see Configuring transcoding in YAML below.

Here's an example using our recommended approach from the Bookstore:

// Returns a specific bookstore shelf.
rpc GetShelf(GetShelfRequest) returns (Shelf) {
  // Client example - returns the first shelf:
  //   curl http://DOMAIN_NAME/v1/shelves/1
  option (google.api.http) = { get: "/v1/shelves/{shelf}" };
}

...
// Request message for GetShelf method.
message GetShelfRequest {
  // The ID of the shelf resource to retrieve.
  int64 shelf = 1;
}

The annotation tells the ESP that making an HTTP GET request with the URL http://mydomain/v1/shelves/1 will call the gRPC server's GetShelf() method, with a GetShelfRequest containing the requested shelf ID shelf (in this case, 1).

Adding transcoding mappings

Now let's look at some mapping annotations in more detail: we'll use some standard methods from the Bookstore as our examples. We've provided you with two sample proto files in our Bookstore sample so that you can quickly deploy it both with or without the transcoding mappings, and compare the differences in the proto files:

For a more comprehensive guide to specifying transcoding mappings, see Standard Methods, Custom Methods, and the Http Rule Reference.

Mapping a List method

First let's look at a simple List method. Here's how it's defined in the .proto with its response type:

  // Returns a list of all shelves in the bookstore.
  rpc ListShelves(google.protobuf.Empty) returns (ListShelvesResponse) {
    // Define HTTP mapping.
    // Client example (Assuming your service is hosted at the given 'DOMAIN_NAME'):
    //   curl http://DOMAIN_NAME/v1/shelves
    option (google.api.http) = { get: "/v1/shelves" };
  }
...
message ListShelvesResponse {
  // Shelves in the bookstore.
  repeated Shelf shelves = 1;
}

The annotation in bold specifies the HTTP mapping for this method.

  • option (google.api.http): This specifies that this is a gRPC HTTP mapping annotation.
  • get: This specifies that this method is mapped to an HTTP GET request.
  • "/v1/shelves": This is the URL path template (appended to your service's domain) that the GET request uses to call this method, also known as the resource path as it typically specifies the "thing" or resource you want to use: in this case, all our Bookstore's shelf resources. As you can see, it specifies /v1/shelves.

So, for example, if a client calls this method by sending a GET to the URL http://mydomain/v1/shelves, the ESP calls the gRPC method ListShelves(). The gRPC backend then returns the shelves, which ESP converts to JSON format and returns to the client.

Mapping a Get method

Now let's take a closer look at the Bookstore's GetShelf method. Here's how it's defined in the .proto with its request and response types:

// Returns a specific bookstore shelf.
rpc GetShelf(GetShelfRequest) returns (Shelf) {
  // Client example - returns the first shelf:
  //   curl http://DOMAIN_NAME/v1/shelves/1
  option (google.api.http) = { get: "/v1/shelves/{shelf}" };
}

...
// Request message for GetShelf method.
message GetShelfRequest {
  // The ID of the shelf resource to retrieve.
  int64 shelf = 1;
}
...
// A shelf resource.
message Shelf {
  // A unique shelf id.
  int64 id = 1;
  // A theme of the shelf (fiction, poetry, etc).
  string theme = 2;
}

The annotation in bold specifies the HTTP mapping for this method.

  • option (google.api.http): This specifies that this is a gRPC HTTP mapping annotation.
  • get: This specifies that this method is mapped to an HTTP GET request.
  • "/v1/shelves/{shelf}": The tricky bit! This is the URL path for the request, as before, but as you can see, it specifies /v1/shelves/ and then {shelf}. This curly-brace notation tells the ESP that whatever is in {shelf} is the value it should provide for shelf in the method's GetShelfRequest parameter.

If a client calls this method by sending a GET to the URL http://mydomain/v1/shelves/4, the ESP creates a GetShelfRequest with a shelf value of 4 and then calls the gRPC method GetShelf() with it. The gRPC backend then returns the requested Shelf with the ID 4, which ESP converts to JSON format and returns to the client.

This method only requires a single request field value to be provided by the client, shelf, which we specify in the URL path template with our curly-brace "capture" notation. You can also capture multiple parts of the URL if necessary to identify the requested resource. For example, the GetBook method needs the client to specify both the shelf ID and the book ID in the URL:

// Returns a specific book.
rpc GetBook(GetBookRequest) returns (Book) {
  // Client example - get the first book from the second shelf:
  //   curl http://DOMAIN_NAME/v1/shelves/2/books/1
  option (google.api.http) = { get: "/v1/shelves/{shelf}/books/{book}" };
}
...
// Request message for GetBook method.
message GetBookRequest {
  // The ID of the shelf from which to retrieve a book.
  int64 shelf = 1;
  // The ID of the book to retrieve.
  int64 book = 2;
}

As well as literals and capture braces for field values, URL path templates can use wildcards to indicate that anything in this part of the URL should be captured (the {shelf} notation used above is actually a shortcut for {shelf=*}). You can find out more about the rules for path templates in the Http Rule Reference.

In the case of this method type, there's no specified HTTP request body: we'll look at a method type that uses this in the next section. You can find more guidelines for mapping Get methods, including using query parameters, in Standard Methods.

Mapping a Create method

Finally let's look at a Create method, which maps to HTTP POST. Here's the Bookstore's CreateShelf method:

  // Creates a new shelf in the bookstore.
  rpc CreateShelf(CreateShelfRequest) returns (Shelf) {
    // Client example:
    //   curl -d '{"theme":"Music"}' http://DOMAIN_NAME/v1/shelves
    option (google.api.http) = {
      post: "/v1/shelves"
      body: "shelf"
    };
  }
...
// Request message for CreateShelf method.
message CreateShelfRequest {
  // The shelf resource to create.
  Shelf shelf = 1;
}
...
// A shelf resource.
message Shelf {
  // A unique shelf id.
  int64 id = 1;
  // A theme of the shelf (fiction, poetry, etc).
  string theme = 2;
}
  • option (google.api.http): This specifies that this is a gRPC HTTP mapping annotation.
  • post: This specifies that this method is mapped to an HTTP POST request.
  • "/v1/shelves": The URL path for the request, as before.
  • body: "shelf": In this method type, you use the HTTP request body to specify the resource you want to add, in JSON format.

So, for example, if a client called this method like this:

curl -d '{"theme":"Music"}' http://DOMAIN_NAME/v1/shelves

ESP uses the JSON body to create a Shelf value with the theme "Music" for the CreateShelfRequest, then calls the gRPC CreateShelf() method. Note that the client does not provide the id value for the Shelf: the Bookstore's shelf IDs are provided by the service when creating a new shelf. This is the type of information you should provide to users of your service in API documentation.

Configuring transcoding in YAML

This is an alternative approach where you specify your HTTP to gRPC mappings in your gRPC API Configuration YAML file rather than in your .proto. You may need to configure transcoding in YAML if you have a single proto API definition that's used in multiple services, with different mappings specified for each service.

The rules in the http section of your YAML file specify how to map HTTP/JSON requests to gRPC methods:

http:
  rules:
  ...
  #
  # 'GetShelf' is available via the GET HTTP verb and '/shelves/{shelf}' URL
  # path, where {shelf} is the value of the 'shelf' field of 'GetShelfRequest'
  # protobuf message.
  #
  # Client example - returns the first shelf:
  #   curl http://DOMAIN_NAME/v1/shelves/1
  #
  - selector: endpoints.examples.bookstore.Bookstore.GetShelf
    get: /v1/shelves/{shelf}
  ...

You can see a more complete example using this approach for our example Bookstore service in api_config_http.yaml.

Deploying a service using transcoding

Deploying a gRPC service that uses transcoding is much the same as deploying any other gRPC service, with one major difference. In the Tutorials, our example just needed to accept gRPC requests from our sample client. However, if we want the Bookstore to accept HTTP requests as well, we need to do some extra configuration for ESP. Clients will use the HTTP1.1 protocol to send JSON/HTTP requests to the ESP, so the ESP either needs to be configured to use SSL (the SSL port can support both request types) or needs to have a special port enabled to accept these calls. Deployment is otherwise largely the same as in the tutorial for your chosen environment.

Ensure HTTP rules are deployed

If you've already downloaded our Bookstore example for our Tutorials, note that you need to download a slightly different version of the .proto with annotations, http_bookstore.proto. You'll also need to clone the googleapis repository from Github before running protoc, as you'll need annotations.proto in your include path.

    git clone https://github.com/googleapis/googleapis

    GOOGLEAPIS_DIR=<your-local-googleapis-folder>

Then create a new .pb descriptor from http_bookstore.proto when deploying your configuration to Endpoints:

     protoc -I${GOOGLEAPIS_DIR} -I. --include_imports --include_source_info http_bookstore.proto --descriptor_set_out out.pb

If you are using the alternative method of configuring HTTP mappings in your gRPC API Configuration YAML file, you also need to ensure that the relevant rules are deployed when deploying your configuration to Endpoints. To try this with the Bookstore service, its basic rules are in api_config.yaml and its HTTP rules are in api_config_http.yaml:

    gcloud service-management deploy out.pb api_config.yaml api_config_http.yaml

Using SSL

If SSL is enabled for communication between your clients and ESP, then clients can use the same port to make gRPC or HTTP1.1 calls. You can find out how to set up SSL for a Cloud Endpoints service in Enabling SSL.

Specify a port for the ESP to accept SSL calls using the -S flag in the Kubernetes configuration file (Container Engine) or docker run command (Compute Engine/Docker).

    args: [
      "--http_port", "8080",
      "-S", "443",  # enable SSL port at 443 to serve https requests
      "-a",  "grpc://127.0.0.1:8081",  # gRPC backend.
      "-s", "SERVICE_NAME",
      "-v", "SERVICE_VERSION",
    ]

Setting up an HTTP1.1 port

If you are not using SSL, you'll need to set up a separate port for HTTP1.1 requests: gRPC and HTTP1.1 can't share the same port without SSL. Use the --http_port flag in the Kubernetes configuration file (Container Engine) or docker run command (Compute Engine/Docker) to specify a port to accept HTTP1.1 calls. If you also want ESP to accept gRPC calls, you'll need to use the --http2_port flag as well to specify a gRPC port, as in our tutorials.

    args: [
      "--http_port", "8080",  # for HTTP 1.1
      "--http2_port", "8090",  # for gRPC
      "-a", "grpc://127.0.0.1:8081",  # gRPC backend.
      "-s", "SERVICE_NAME",
      "-v", "SERVICE_VERSION",
    ]

Calling a service using transcoding

Finally, let's look at how to make HTTP calls to the Bookstore service.

Service setup

This assumes that you've already completed the basic gRPC service Tutorials for your chosen environment and have a container cluster or Compute Engine instance to run the example.

  1. First ensure that you've deployed the HTTP-enabled Bookstore service configuration to Endpoints, as described in Ensure HTTP rules are deployed above.
  2. Deploy the backend and ESP as described in the tutorial for your chosen platform, using the --http_port flag to enable a port for HTTP1.1 requests:

Make HTTP calls to the service

  1. Get the ESP's external IP address and set it as $ESP_IP.
  2. Make the following HTTP request with curl

    curl http://$ESP_IP/v1/shelves
    

    (or use the same URL with https:// if you're using SSL). The server should respond with:

    {"shelves":[{"id":"1","theme":"Fiction"},{"id":"2","theme":"Fantasy"}]}
    

    If you see what looks like a binary response, check your port configuration; you might be hitting your HTTP2 port instead of your HTTP port.

  3. Now try a Create method. CreateShelf requires an API key, so you'll need to create a key for your project and set it as $KEY. Now call:

    curl -d '{"theme":"Music"}' http://$ESP_IP/v1/shelves?key=$KEY
    

    If you call GetShelves again you should see your new shelf.

Send feedback about...

Cloud Endpoints with gRPC