Set up a gRPC application on Google Cloud with Microservices observability

---

Microservices observability tools provide you with the ability to instrument your applications to collect and present telemetry data in Cloud Monitoring, Cloud Logging, and Cloud Trace from gRPC workloads deployed on Google Cloud and elsewhere. Microservices observability works with any deployment that has been granted permission to access Monitoring, Logging, and Trace by enabling the Microservices API.

In this tutorial you learn how to use Microservices observability features by building a simple gRPC application on Google Cloud using Compute Engine and instrument your application with Microservices observability, and view them come up live on Monitoring and Logging.

Objectives

At a high level, you do the following.

• As a service developer, you can:

  • Create a gRPC application with the language of your choice (C++, Go, or Java).
  • Opt in your application and control the Microservices observability plugin.
  • Deploy the application on a Compute Engine VM.

• As a service operator, you consume the collected data in various ways:

  • View traces on Trace.
  • View metrics on the monitoring dashboard called Microservices (gRPC) Monitoring.
  • View metrics on Metrics Explorer.
  • Inspect log entries on Logs Explorer.

Costs

In this document, you use the following billable components of Google Cloud:

• Compute Engine

To generate a cost estimate based on your projected usage, use the pricing calculator. New Google Cloud users might be eligible for a free trial.

When you finish the tasks that are described in this document, you can avoid continued billing by deleting the resources that you created. For more information, see Clean up.

Before you begin

Console

1. Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.

2. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

   Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

   Go to project selector

3. Make sure that billing is enabled for your Google Cloud project.

4. Enable the Compute Engine, and Microservices API APIs.

   Enable the APIs

5. Create a service account:

   1. In the Google Cloud console, go to the Create service account page.

      Go to Create service account
   2. Select your project.

   3. In the Service account name field, enter a name. The Google Cloud console fills in the Service account ID field based on this name.

      In the Service account description field, enter a description. For example, Service account for quickstart.

   4. Click Create and continue.

   5. Grant the following roles to the service account: Logging > Logs Viewer > Logs Writer, Monitoring > Monitoring Editor > Metrics Writer, Trace > Trace Admin > Trace Agent.

      To grant a role, find the Select a role list, then select the role.

      To grant additional roles, click Add another role and add each additional role.

      Note: The Role field affects which resources the service account can access in your project. You can revoke these roles or grant additional roles later.
   6. Click Continue.

   7. In the Service account users role field, enter the identifier for the principal that will attach the service account to other resources, such as Compute Engine instances.

      This is typically the email address for a Google Account.

   8. Click Done to finish creating the service account.

6. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

   Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

   Go to project selector

7. Make sure that billing is enabled for your Google Cloud project.

8. Enable the Compute Engine, and Microservices API APIs.

   Enable the APIs

9. Create a service account:

   1. In the Google Cloud console, go to the Create service account page.

      Go to Create service account
   2. Select your project.

   3. In the Service account name field, enter a name. The Google Cloud console fills in the Service account ID field based on this name.

      In the Service account description field, enter a description. For example, Service account for quickstart.

   4. Click Create and continue.

   5. Grant the following roles to the service account: Logging > Logs Viewer > Logs Writer, Monitoring > Monitoring Editor > Metrics Writer, Trace > Trace Admin > Trace Agent.

      To grant a role, find the Select a role list, then select the role.

      To grant additional roles, click Add another role and add each additional role.

      Note: The Role field affects which resources the service account can access in your project. You can revoke these roles or grant additional roles later.
   6. Click Continue.

   7. In the Service account users role field, enter the identifier for the principal that will attach the service account to other resources, such as Compute Engine instances.

      This is typically the email address for a Google Account.

   8. Click Done to finish creating the service account.

10. Read the Microservices observability overview.
11. Read about the two supported environment variables, decide which to use, and determine the values required by the environment variable.

cli

1. Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.
2. Install the Google Cloud CLI.

3. To initialize the gcloud CLI, run the following command:

   gcloud init

4. Create or select a Google Cloud project.

   Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

   • Create a Google Cloud project:

     gcloud projects create PROJECT_ID

     Replace PROJECT_ID with a name for the Google Cloud project you are creating.

   • Select the Google Cloud project that you created:

     gcloud config set project PROJECT_ID

     Replace PROJECT_ID with your Google Cloud project name.

5. Make sure that billing is enabled for your Google Cloud project.

6. Enable the Compute Engine, and Microservices API APIs:

   gcloud services enable compute.googleapis.com microservices.googleapis.com

7. Set up authentication:

   1. Create the service account:

      gcloud iam service-accounts create SERVICE_ACCOUNT_NAME

      Replace SERVICE_ACCOUNT_NAME with a name for the service account.

   2. Grant roles to the service account. Run the following command once for each of the following IAM roles: roles/logging.logWriter, roles/monitoring.metricWriter, roles/cloudtrace.agent:

      gcloud projects add-iam-policy-binding PROJECT_ID --member="serviceAccount:SERVICE_ACCOUNT_NAME@PROJECT_ID.iam.gserviceaccount.com" --role=ROLE

      Replace the following:

      • SERVICE_ACCOUNT_NAME: the name of the service account
      • PROJECT_ID: the project ID where you created the service account
      • ROLE: the role to grant
      Note: The --role flag affects which resources the service account can access in your project. You can revoke these roles or grant additional roles later.

   3. Grant the required role to the principal that will attach the service account to other resources.

      gcloud iam service-accounts add-iam-policy-binding SERVICE_ACCOUNT_NAME@PROJECT_ID.iam.gserviceaccount.com --member="user:USER_EMAIL" --role=roles/iam.serviceAccountUser

      Replace the following:

      • SERVICE_ACCOUNT_NAME: the name of the service account
      • PROJECT_ID: the project ID where you created the service account
      • USER_EMAIL: the email address for a Google Account
8. Install the Google Cloud CLI.

9. To initialize the gcloud CLI, run the following command:

   gcloud init

10. Create or select a Google Cloud project.

    Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

    • Create a Google Cloud project:

      gcloud projects create PROJECT_ID

      Replace PROJECT_ID with a name for the Google Cloud project you are creating.

    • Select the Google Cloud project that you created:

      gcloud config set project PROJECT_ID

      Replace PROJECT_ID with your Google Cloud project name.

11. Make sure that billing is enabled for your Google Cloud project.

12. Enable the Compute Engine, and Microservices API APIs:

    gcloud services enable compute.googleapis.com microservices.googleapis.com

13. Set up authentication:

    1. Create the service account:

       gcloud iam service-accounts create SERVICE_ACCOUNT_NAME

       Replace SERVICE_ACCOUNT_NAME with a name for the service account.

    2. Grant roles to the service account. Run the following command once for each of the following IAM roles: roles/logging.logWriter, roles/monitoring.metricWriter, roles/cloudtrace.agent:

       gcloud projects add-iam-policy-binding PROJECT_ID --member="serviceAccount:SERVICE_ACCOUNT_NAME@PROJECT_ID.iam.gserviceaccount.com" --role=ROLE

       Replace the following:

       • SERVICE_ACCOUNT_NAME: the name of the service account
       • PROJECT_ID: the project ID where you created the service account
       • ROLE: the role to grant
       Note: The --role flag affects which resources the service account can access in your project. You can revoke these roles or grant additional roles later.

    3. Grant the required role to the principal that will attach the service account to other resources.

       gcloud iam service-accounts add-iam-policy-binding SERVICE_ACCOUNT_NAME@PROJECT_ID.iam.gserviceaccount.com --member="user:USER_EMAIL" --role=roles/iam.serviceAccountUser

       Replace the following:

       • SERVICE_ACCOUNT_NAME: the name of the service account
       • PROJECT_ID: the project ID where you created the service account
       • USER_EMAIL: the email address for a Google Account
14. Read the Microservices observability overview.
15. Read about the two supported environment variables, decide which to use, and determine the values required by the environment variable.

Create and connect to a Compute Engine VM

Use these instructions to create and connect to a Compute Engine VM instance. On the VM you deploy your application and then instrument the application with Microservices observability.

1. Create a VM instance:

   gcloud compute instances create grpc-observability-vm \
     --image-family=debian-11 \
     --image-project=debian-cloud \
     --service-account=SERVICE_ACCOUNT_NAME@$PROJECT_ID.iam.gserviceaccount.com
   

2. Connect to the VM instance:

   gcloud compute ssh --project=$PROJECT_ID grpc-observability-vm
   

Deploy your application to the Compute Engine VM

You can either deploy an application of your choice to the Compute Engine VM that you created in the previous step, then skip this step, or you can use an example to continue with instructions in your preferred language.

C++

1. After you connect to the VM instance, run the following command.

   sudo apt-get update -y
   sudo apt-get install -y git build-essential clang
   git clone -b v1.54.0 https://github.com/grpc/grpc.git --depth=1
   

Go

1. Make sure that you have Go installed.

   sudo apt-get install -y git
   sudo apt install wget
   wget https://go.dev/dl/go1.20.2.linux-amd64.tar.gz
   sudo rm -rf /usr/local/go && sudo tar -C /usr/local -xzf \
   go1.20.2.linux-amd64.tar.gz
   export PATH=$PATH:/usr/local/go/bin
   

2. Clone the gRPC-Go examples.

   git clone https://github.com/grpc/grpc-go.git
   cd grpc-go/
   git checkout -b run-observability-example
   875c97a94dca8093bf01ff2fef490fbdd576373d
   

Java

1. After you connect to the VM instance, make sure that you have Java 8 or later installed.

   sudo apt update
   sudo apt upgrade
   sudo apt install git
   sudo apt-get install -y openjdk-11-jdk-headless
   

2. Clone the grpc-java repository.

   export EXAMPLES_VERSION=v1.54.1
   git clone -b $EXAMPLES_VERSION --single-branch --depth=1 \
   https://github.com/grpc/grpc-java.git
   

Create gRPC Google Cloud Observability config file

You need separate gRPC Google Cloud Observability config file to enable Microservices observability for both server and client. The location of this file is exported as GRPC_GCP_OBSERVABILITY_CONFIG_FILE in the future steps. Use the following instruction on how to set up the different parameters in the config file.

Example GRPC_GCP_OBSERVABILITY_CONFIG_FILE

{
  "project_id": "your-project-here",
  "cloud_logging": {
    "client_rpc_events": [
    {
      "methods": ["google.pubsub.v1.Subscriber/Acknowledge", "google.pubsub.v1.Publisher/CreateTopic"],
      "exclude": true,
    },
    {
      "methods": ["google.pubsub.v1.Subscriber/*", "google.pubsub.v1.Publisher/*"],
      "max_metadata_bytes": 4096,
      "max_message_bytes": 4096,
    }],
    "server_rpc_events": [{
      "methods": ["*"],
      "max_metadata_bytes": 4096,
      "max_message_bytes": 4096
    }],
  },
  "cloud_monitoring": {},
  "cloud_trace": {
    "sampling_rate": 0.5,
  }
  "labels": {
    "SOURCE_VERSION": "J2e1Cf",
    "SERVICE_NAME": "payment-service-1Cf",
    "DATA_CENTER": "us-west1-a"
  }
}

The following sections contain instructions for enabling data collection in your configuration for the individual components. If you used the gRPC example in this tutorial, you can use this config as is (after updating your-project-here) or use this as a template for your application.and an example showing the configuration information in an environment variable.

Enable metrics

To enable metrics, add the cloud_monitoring object to the configuration and set its value to {}.

For more information about metrics, see Metrics definitions.

Enable tracing

To enable tracing, do the following:

1. Add the cloud_trace object to the configuration.
2. Set the cloud_trace.sampling_rate to 0.5 which to randomly traces 50% of RPCs.

If you plan to enable tracing across services, ensure that the services support the propagation of trace context received from upstream (or started by itself) to downstream.

For more information about tracing, see Trace definitions.

Enable logging

To enable logging, do the following:

1. Add the cloud_logging object to the configuration.
2. Add a pattern to either or both of client_rpc_events and server_rpc_events specifying the set of services or methods for which you want to generate transport-level event logging and the number of bytes to log for headers and messages.

For more information about logging, see Log record definitions.

Instrument your applications for the observability plugin

To instrument your applications so that they can use the Microservices observability plugin, use the following instructions for your preferred language.

C++

You can use C++ with Microservices observability as of gRPC C++ v1.54. The example repository is in GitHub.

1. Observability support is only available through the Bazel build system. Add the target grpcpp_gcp_observability as a dependency.

2. Opting in Microservices observability requires an additional dependency (an observability modul) and the following code changes to existing gRPC clients, servers, or both:

   #include <grpcpp/ext/gcp_observability.h>
   
   int main(int argc, char** argv) {
     auto observability = grpc::GcpObservability::Init();
     assert(observability.ok());
     …
     // Observability data flushed when object goes out of scope
   }
   

   Before any gRPC operations, including creating a channel, server, or credentials, invoke the following:

   grpc::GcpObservability::Init();
   

   This returns absl::StatusOr<GcpObservability> which should be saved. The status helps determine whether observability was successfully initialized. The accompanying GcpObservability object controls the lifetime of observability, and automatically closes and flushes observability data when it goes out of scope.

Go

1. Microservices observability plugins are supported for gRPC Go versions v1.54.0 and later. The example repository is in GitHub.

With the Go module, opting in Microservices observability requires an observability module and the following code:

import "google.golang.org/grpc/gcp/observability"

func main() {
  ctx, cancel := context.WithTimeout(context.Background(), time.Second)
  defer cancel()
  if err := observability.Start(ctx); err != nil {
    log.Warning("Unable to start gRPC observability:", err)
  }
  defer observability.End()
  …
}

The observability.Start call parses the configuration from environment variables, creates exporters accordingly, and injects collection logic to client connections and servers created after the call. The deferredobservability.End call cleans up resources and ensures that buffered data is flushed before the application closes.

After the application code is updated, run the following command to update the go.mod file.

go mod tidy

Java

To use Microservices observability with Java applications, modify your build to include the grpc-gcp-observability artifact. Use gRPC version 1.54.1 or later.

In the build snippets in the Gradle and Maven build tool sections, grpcVersion is set to the value 1.54.1.

The example repository is in GitHub.

1. To successfully instrument your Java applications for Microservices observability, add the following code to main().

...
import io.grpc.gcp.observability.GcpObservability;
...

// Main application class
...

public static void main(String[] args) {
...
  // call GcpObservability.grpcInit() to initialize & get observability
  GcpObservability observability = GcpObservability.grpcInit();

...
  // call close() on the observability instance to shutdown observability
  observability.close();
...
}

Note that you must call GcpObservability.grpcInit() before any gRPC channels or servers are created. The GcpObservability.grpcInit() function reads the Microservices observability configuration and uses that to set up the global interceptors and tracers that are required for the logging, metrics, and trace features in each channel and server created. GcpObservability.grpcInit() is thread safe and must be called exactly once. It returns an instance of GcpObservability that you must save in order to call close() later.

GcpObservability.close() de-allocates resources. Any channel or servers created afterwards don't perform any logging.

GcpObservability implements java.lang.AutoCloseable, which is closed automatically if you use try-with-resources as follows:

...
import io.grpc.gcp.observability.GcpObservability;
...

// Main application class
...

public static void main(String[] args) {
...
  // call GcpObservability.grpcInit() to initialize & get observability
  try (GcpObservability observability = GcpObservability.grpcInit()) {

...
  } // observability.close() called implicitly
...
}

Use the Gradle build tool

If you are using the Gradle build tool, include the following:

def grpcVersion = '1.54.1'

...

dependencies {
...
implementation "io.grpc:grpc-gcp-observability:${grpcVersion}"
...
}

Use the Maven build tool (pom.xml)

If you are using the Maven build tool, include the following:

<properties>
...
<grpc.version>1.54.1</grpc.version>
...
</properties>

...

<dependencies>
...
<dependency>
<groupId>io.grpc</groupId>
<artifactId>grpc-gcp-observability</artifactId>
<version>${grpc.version}</version>
</dependency>
...
</dependencies>

Run your application

Follow the instructions in this section only if you used the gRPC example for the tutorial. You can modify the run command to target your application binary.

Run Server

C++

1. Create a SSH session into the VM.

2. Export environmental variables. Use steps defined above to create server_config.json.

     export GOOGLE_CLOUD_PROJECT=$PROJECT_ID
     export GRPC_GCP_OBSERVABILITY_CONFIG_FILE="$(pwd)/examples/cpp/gcp_observability/helloworld/server_config.json"
   

3. Run the server application shell cd grpc tools/bazel run examples/cpp/gcp_observability/helloworld:greeter_server

Go

1. Create a SSH session into the VM.

2. Export environmental variables. Use steps defined above to create server_config.json.

   export GRPC_GCP_OBSERVABILITY_CONFIG_FILE=./server/serverConfig.json
   

3. Run the server application shell go run ./server/main.go

Java

1. In the examples directory, open the README file and follow the instructions in the file.
2. When the instructions tell you to open another terminal window, issue this command: shell gcloud compute ssh --project=$PROJECT_ID grpc-observability-vm

Run Client

C++

1. Create another SSH session into the VM.

2. Export environmental variables. Use steps defined above to create client_config.json file.

     export GOOGLE_CLOUD_PROJECT=$PROJECT_ID
     export GRPC_GCP_OBSERVABILITY_CONFIG_FILE="$(pwd)/examples/cpp/gcp_observability/helloworld/client_config.json"
   

3. Run the client application

   cd grpc
   tools/bazel run examples/cpp/gcp_observability/helloworld:greeter_client
   

Go

1. Create another SSH session into the VM.
2. Export environmental variables. Use steps defined above to create client_config.json file. shell export GRPC_GCP_OBSERVABILITY_CONFIG_FILE=./client/clientConfig.json

3. Run the client application

   cd grpc-go/examples/features/observability
   go run ./client/main.go
   

Java

1. In the examples directory, open the README file and follow the instructions in the file.
2. When the instructions tell you to open another terminal window, issue this command: shell gcloud compute ssh --project=$PROJECT_ID grpc-observability-vm

Clean up

To avoid incurring charges to your Google Cloud account for the resources used in this tutorial, either delete the project that contains the resources, or keep the project and delete the individual resources.

Delete the project

Caution: Deleting a project has the following effects:

• Everything in the project is deleted. If you used an existing project for the tasks in this document, when you delete it, you also delete any other work you've done in the project.
• Custom project IDs are lost. When you created this project, you might have created a custom project ID that you want to use in the future. To preserve the URLs that use the project ID, such as an appspot.com URL, delete selected resources inside the project instead of deleting the whole project.

If you plan to explore multiple architectures, tutorials, or quickstarts, reusing projects can help you avoid exceeding project quota limits.

Delete a Google Cloud project:

gcloud projects delete PROJECT_ID

Delete individual resources

1. Delete the instance:

   gcloud compute instances delete INSTANCE_NAME

What's next

• For detailed information about configuration data, trace definitions, metrics definitions, and log definitions see Microservices observability reference.
• For information about viewing logs, traces, and metrics, see View traces, metrics, and log entries