Connect to Cloud Storage using gRPC

gRPC is a high performance, open source universal RPC framework developed by Google that you can use to define your services using Protocol Buffers. You can use gRPC to interact with Cloud Storage. gRPC utilizes direct connectivity between Compute Engine and Cloud Storage buckets, bypassing Google Front Ends (GFEs).

Note: To achieve optimal performance with gRPC when connecting to Cloud Storage, ensure your Compute Engine VMs and regional buckets are located in the same region.

While directly using gRPC to connect to Cloud Storage is not supported, you can connect to Cloud Storage using gRPC through the following supported clients:

• Cloud Storage connector on Dataproc.
• Apache Beam I/O connector on Dataflow for read and write requests.
• Cloud Storage C++, Go, and Java client libraries. For instructions on enabling gRPC for these client libraries, see Enable gRPC on client library.

Enable gRPC on a client library

C++

Before you begin

1. Ensure you have the following versions installed:

   • gRPC version 1.65.1 or later.
   • C++ client library version v2.30.0 or later.
   • C++ version 14 or later.

   For installation instructions, see Setting up a C++ development environment.

2. Set up authentication.

Configure the C++ client library

1. Create a gRPC client using gcs::MakeGrpcClient():

     namespace gcs = google::cloud::storage;
     void App() {
      auto client = gcs::MakeGrpcClient();
   
     // application code
   
     }
   

   The C++ client library automatically uses direct connectivity when it detects that the application is running on Google Cloud.

2. To configure the C++ client library to use gRPC, enable the Cloud Storage gRPC client to update your build system configuration for CMake and Bazel.

   CMake

   1. Enable the Cloud Storage gRPC client plugin at compile-time

      cmake -DGOOGLE_CLOUD_CPP_ENABLE=storage_grpc [other options here]
      

   2. In your codebase, for the target_link_libraries() command replace google-cloud-cpp::storage with google-cloud-cpp::storage_grpc

      For example, the quickstart program for gRPC uses the following code:

      add_executable(quickstart_grpc quickstart_grpc.cc)
      target_link_libraries(quickstart_grpc google-cloud-cpp::storage_grpc)
      

   Bazel

   Replace the dependencies from @google_cloud_cpp//:storage to @google_cloud_cpp//:storage_grpc.

   For example, the quickstart program for gRPC uses the following code:

     cc_binary(
         name = "quickstart",
         srcs = [
             "quickstart.cc",
         ],
         deps = [
             "@com_github_googleapis_google_cloud_cpp//:storage_grpc",
         ],
     )
   

Java

Before you begin

1. Ensure you have the following versions installed:

   • Java client libraries:

     • com.google.cloud:google-cloud-storage:2.43.1 or later.
     • com.google.cloud:libraries-bom:26.48 or later.

   • Java 8 or later

   For installation instructions, see Setting up a Java development environment.

2. Set up authentication.

Update your project to use the BOM

To make sure that your projects have compatible versions of Google Cloud client libraries, use the versions specified in the Google Cloud libraries Bill of Materials (BOM). To update your project to use the BOM, use any of the following methods:

• Standalone Cloud Storage
• Cloud Storage with other Google Cloud libraries

Standalone Cloud Storage

If you are using the Cloud Storage client library independently (without other Google Cloud libraries), use the Cloud Storage client library-specific BOM.

Maven

Import the BOM in the dependencyManagement section of your pom.xml file.

The following example shows how you would import the BOM and include the google-cloud-storage artifact.

<dependencyManagement>
  <dependencies>
  <dependency>
   <groupId>com.google.cloud</groupId>
   <artifactId>google-cloud-storage-bom</artifactId>
   <version>2.43.1</version>
   <type>pom</type>
   <scope>import</scope>
  </dependency>
  </dependencies>
</dependencyManagement>

<dependencies>
  <dependency>
   <groupId>com.google.cloud</groupId>
   <artifactId>google-cloud-storage</artifactId>
  </dependency>
</dependencies>

Gradle

Add a platform dependency on com.google.cloud:google-cloud-storage-bom:

implementation platform('com.google.cloud:google-cloud-storage-bom:2.43.1')
implementation 'com.google.cloud:google-cloud-storage'

Cloud Storage with other Google Cloud libraries

If you are using the Cloud Storage client library along with other Google Cloud libraries, use the Google Cloud client libraries BOM.

Maven

Import the BOM in the dependencyManagement section of your pom.xml file.

The following example shows how to import the BOM and include the libraries-bom artifact.

<dependencyManagement>
  <dependencies>
  <dependency>
   <groupId>com.google.cloud</groupId>
   <artifactId>libraries-bom</artifactId>
   <version>26.48.0</version>
   <type>pom</type>
   <scope>import</scope>
  </dependency>
  </dependencies>
</dependencyManagement>

<dependencies>
  <dependency>
   <groupId>com.google.cloud</groupId>
   <artifactId>google-cloud-storage</artifactId>
  </dependency>
</dependencies>

Gradle

Add a platform dependency on com.google.cloud:libraries-bom:

implementation platform('com.google.cloud:libraries-bom:26.48.0')
implementation 'com.google.cloud:google-cloud-storage'

Create a gRPC client

The following sample uses a gRPC centric builder. The gRPC Java client automatically uses direct connectivity when it detects that the application is running on Google Cloud.

// Imports the Google Cloud client library
import com.google.cloud.storage.Bucket;
import com.google.cloud.storage.BucketInfo;
import com.google.cloud.storage.Storage;
import com.google.cloud.storage.StorageOptions;

public class QuickstartGrpcSample {
  public static void main(String... args) throws Exception {

    // Create an instance of options which will use the Google Cloud Storage gRPC API for all
    // operations
    StorageOptions options = StorageOptions.grpc().build();

    // Instantiates a client in a try-with-resource to automatically cleanup underlying resources
    try (Storage storage = options.getService()) {
      // The name for the new bucket
      String bucketName = args[0]; // "my-new-bucket";

      // Creates the new bucket using a request to the gRPC API
      Bucket bucket = storage.create(BucketInfo.of(bucketName));

      System.out.printf("Bucket %s created.%n", bucket.getName());
    }
  }
}

Go

Before you begin

1. Setup your development environment.

2. Ensure you use Cloud Storage Go client library version 1.46.0 or later.

3. Set up authentication.

Create a gRPC client

To use the client, you need to call the NewGRPCClient constructor in your application instead of NewClient.

// Sample storage-quickstart creates a Google Cloud Storage bucket using
// gRPC API.
package main

import (
	"context"
	"fmt"
	"log"
	"time"

	"cloud.google.com/go/storage"
)

func main() {
	ctx := context.Background()

	// Use your Google Cloud Platform project ID and Cloud Storage bucket
	projectID := "project-id"
	bucketName := "bucket-name"

	// Creates a gRPC enabled client.
	client, err := storage.NewGRPCClient(ctx)
	if err != nil {
		log.Fatalf("Failed to create client: %v", err)
	}
	defer client.Close()

	// Creates the new bucket.
	ctx, cancel := context.WithTimeout(ctx, time.Second*10)
	defer cancel()
	if err := client.Bucket(bucketName).Create(ctx, projectID, nil); err != nil {
		log.Fatalf("Failed to create bucket: %v", err)
	}

	fmt.Printf("Bucket %v created.\n", bucketName)
}

The Go client library automatically uses direct connectivity when it detects that the application is running on Google Cloud.

For information about how to use a gRPC client, see gRPC API.

Configure VPC Service Controls

If you are using Virtual Private Cloud with restricted virtual IP addresses ( VPC Service Controls) to enhance the security of your network, you need to update your firewall rules to enable direct connectivity for optimal performance with between the Compute Engine virtual machine instances and Cloud Storage.

To do so, add allowlist firewall rules to permit traffic on all ports for the following CIDR blocks:

• For IPv4 traffic: 34.126.0.0/18
• For IPv6 traffic: 2001:4860:8040::/42

In addition to the preceding rules, retain the existing allowlist rule for 199.36.153.4/30.

If you have restrictions on firewall rule modifications and you cannot update them, then you can force the traffic to avoid direct connectivity by using storage.googleapis.com as the Cloud Storage endpoint instead of google-c2p://storage.googleapis.com.

For example, for C++ use, .set<google::cloud::EndpointOption>(storage.googleapis.com) instead of google-c2p:///storage.googleapis.com.

Enable observability for gRPC related requests

You can configure Cloud Storage client libraries to to generate gRPC related metrics in Cloud Monitoring. The gRPC related metrics can help you to do the following:

• Monitor and optimize the performance of gRPC requests to Cloud Storage.

• Troubleshoot and debug issues.

• Gain insights into your application's usage and behavior.

For information about how to generate gRPC related metrics, see Use client-side metrics.

If gathering metrics isn't necessary for your use case, you can choose to opt-out of metrics collection. For instructions, see Opt-out of client-side metrics.

Limitations

• IPv6 requests cannot be sent over legacy networks.

• Using gRPC to connect to Cloud Storage is expected to improve the read performance for analytics workloads running on Google Cloud only when compute VMs and Cloud Storage buckets are located in the same region. We don't recommend using gRPC for buckets in multi-region and dual-regions.

• Direct connectivity is not supported when using the following GKE versions on IPv4-only clusters:

  • 1.28, 1.28.0-gke.100 or later until 1.28.5-gke.1199000
  • 1.27, 1.27.4-gke.1900 or later
  • 1.26, 1.26.10-gke.1238000 or later
  • 1.25, 1.25.15-gke.1045000 or later

• gRPC does not support notifications, hmacKeys, and serviceAccount methods.

• HTTP-specific client constructor options, such as WithHTTPClient, are not supported for Go client library.

What's next

• Connect to the Cloud Storage connector on Dataproc using gRPC.
• Enable gRPC on Apache Beam I/O connector on Dataflow.