Monitoring time-series data with OpenTSDB on Cloud Bigtable and GKE

This tutorial describes how to collect, record, and monitor time-series data on Google Cloud by using OpenTSDB running on Google Kubernetes Engine (GKE) and Bigtable.

Time-series data is a highly valuable asset that you can use for several applications, including trending, monitoring, and machine learning. You can generate time-series data from server infrastructure, application code, and other sources. OpenTSDB can collect and retain large amounts of time-series data with a high degree of granularity.

This tutorial shows software engineers and architects how to create a scalable collection layer for time-series data by using GKE. It also shows how to work with the collected data by using Bigtable. This tutorial assumes that you are familiar with Kubernetes and Bigtable.

The following diagram shows the high-level architecture of this tutorial:

The preceding diagram shows multiple sources of time-series data, such as IoT events and system metrics, that are stored in Bigtable by using OpenTSDB deployed on GKE.

Objectives

• Create a Bigtable instance.
• Create a GKE cluster.
• Deploy OpenTSDB to your GKE cluster.
• Send time-series metrics to OpenTSDB.
• Visualize metrics using OpenTSDB and Grafana.

Costs

This tutorial uses the following billable components of Google Cloud:

• Compute Engine
• GKE
• Bigtable
• Cloud Storage

To generate a cost estimate based on your projected usage, use the pricing calculator. New Google Cloud users might be eligible for a free trial.

When you finish this tutorial, you can avoid continued billing by deleting the resources you created. For more information, see Cleaning up.

Before you begin

1. In the Google Cloud Console, on the project selector page, select or create a Google Cloud project.

   Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

   Go to project selector

2. Make sure that billing is enabled for your Cloud project. Learn how to confirm that billing is enabled for your project.

3. Enable the Bigtable, Bigtable Admin, GKE, Compute Engine APIs.

   Enable the APIs

4. In the Cloud Console, go to the Dashboard page.

   Go to Dashboard

   Make a note of the project ID because it's used in a later step.

5. In the Cloud Console, activate Cloud Shell.

   Activate Cloud Shell

Preparing your environment

1. In Cloud Shell, set the default Compute Engine zone:

   gcloud config set compute/zone COMPUTE_ZONE
   

   Replace COMPUTE_ZONE with the zone where you are creating your Bigtable cluster—for example, us-central1-b.

2. Clone the git repository that contains the sample code:

   git clone https://github.com/GoogleCloudPlatform/opentsdb-bigtable.git
   

3. Go to the sample code directory:

   cd opentsdb-bigtable
   

Creating a Bigtable instance

This tutorial uses Bigtable to store the time-series data that you collect, so you must create a Bigtable instance.

Bigtable is a key/wide-column store that works well for time-series data. Bigtable supports the HBase API, so you can use software designed to work with Apache HBase, such as OpenTSDB. For more information about the HBase schema used by OpenTSDB, see HBase Schema.

A key component of OpenTSDB is the AsyncHBase client, which enables you to bulk-write to HBase in a fully asynchronous, non-blocking, thread-safe manner. When you use OpenTSDB with Bigtable, AsyncHBase is implemented as the AsyncBigtable client.

This tutorial uses a Bigtable instance with a single-node cluster. When moving to a production environment, consider using Bigtable instances with larger clusters. For more information about picking a cluster size, see Understanding Bigtable performance.

• Create a Bigtable instance:

  gcloud bigtable instances create INSTANCE_ID \
      --cluster=CLUSTER_ID \
      --cluster-zone=COMPUTE_ZONE \
      --display-name=OpenTSDB \
      --cluster-num-nodes=1
  

  Replace the following:

  • INSTANCE_ID: The permanent identifier for the instance.
  • CLUSTER_ID: The permanent identifier for the cluster.
  • COMPUTE_ZONE: The zone where the cluster runs. We recommend using the same default Compute Engine zone that you previously set.

Creating a GKE cluster

GKE provides a managed Kubernetes environment. After you create a GKE cluster, you can deploy Kubernetes Pods to it. This tutorial uses GKE and Kubernetes Pods to run OpenTSDB.

OpenTSDB separates its storage from its application layer, which enables it to be simultaneously deployed across multiple instances. By running in parallel, OpenTSDB can handle a large amount of time-series data.

1. In Cloud Shell, create a GKE cluster:

   gcloud container clusters create CLUSTER_NAME --scopes \
       "https://www.googleapis.com/auth/devstorage.read_only",\
       "https://www.googleapis.com/auth/logging.write",\
       "https://www.googleapis.com/auth/monitoring",\
       "https://www.googleapis.com/auth/bigtable.data",\
       "https://www.googleapis.com/auth/bigtable.admin",\
       "https://www.googleapis.com/auth/servicecontrol",\
       "https://www.googleapis.com/auth/service.management.readonly",\
       "https://www.googleapis.com/auth/trace.append"
   

   Replace the following:

   • CLUSTER_NAME: the name of your new cluster

   This operation can take a few minutes to complete. Adding the scopes to your GKE cluster allows your OpenTSDB container to interact with Bigtable and Container Registry.

   The rest of this tutorial uses a prebuilt container, gcr.io/cloud-solutions-images/opentsdb-bigtable:v2, located in Container Registry. The Dockerfile and entrypoint script used to build the container are located in the build folder of the tutorial repository.

2. Connect to your GKE cluster:

   gcloud container clusters get-credentials CLUSTER_NAME
   

Creating a ConfigMap with configuration details

Kubernetes uses the ConfigMap to decouple configuration details from the container image in order to make applications more portable. The configuration for OpenTSDB is specified in the opentsdb.conf file. A ConfigMap containing the opentsdb.conf file is included with the sample code.

Edit the OpenTSDB configuration to use the project name, instance identifier, and zone that you used when creating your instance.

1. In Cloud Shell, click Open Editor.
2. To open the ConfigMap file in the editor, go to opentsdb-bigtable > configmaps > opentsdb-config.yaml

3. In the opentsdb-config.yaml file, replace the placeholder text for the following variables with the values that you previously set:

   • Replace google.bigtable.project.id with your project ID.
   • Replace google.bigtable.zone.id with the value that you used for COMPUTE_ZONE.
   • Replace google.bigtable.instance.id with the value that you used for INSTANCE_ID.

4. To change from the editor to the Cloud Shell terminal, click Open Terminal.

5. Create a ConfigMap from the updated opentsdb-config.yaml file:

   kubectl create -f configmaps/opentsdb-config.yaml
   

   Note: OpenTSDB offers various configuration options. To apply your configuration, modify the opentsdb.conf ConfigMap and push the changes to the cluster. Some changes might require you to restart processes.

Creating OpenTSDB tables in Bigtable

Before you can read or write data using OpenTSDB, you need to create tables in Bigtable to store that data.

1. To create the tables, create a Kubernetes job.

2. In Cloud Shell, launch the job:

   kubectl create -f jobs/opentsdb-init.yaml
   

   The job can take up to a minute or more to complete. Verify that the job has completed successfully:

   kubectl describe jobs
   

   The output shows that one job has succeeded when Pods Statuses shows Success

3. Examine the table creation job logs:

   pods=$(kubectl get pods --selector=job-name=opentsdb-init \
       --output=jsonpath={.items..metadata.name})
   kubectl logs $pods
   

   The output is similar to the following:

   create 'tsdb-uid',
     {NAME => 'id', COMPRESSION => 'NONE', BLOOMFILTER => 'ROW', DATA_BLOCK_ENCODING => 'DIFF'},
     {NAME => 'name', COMPRESSION => 'NONE', BLOOMFILTER => 'ROW', DATA_BLOCK_ENCODING => 'DIFF'}
   0 row(s) in 3.2730 seconds
   
   create 'tsdb',
     {NAME => 't', VERSIONS => 1, COMPRESSION => 'NONE', BLOOMFILTER => 'ROW', DATA_BLOCK_ENCODING => 'DIFF'}
   0 row(s) in 1.8440 seconds
   
   create 'tsdb-tree',
     {NAME => 't', VERSIONS => 1, COMPRESSION => 'NONE', BLOOMFILTER => 'ROW', DATA_BLOCK_ENCODING => 'DIFF'}
   0 row(s) in 1.5420 seconds
   
   create 'tsdb-meta',
     {NAME => 'name', COMPRESSION => 'NONE', BLOOMFILTER => 'ROW', DATA_BLOCK_ENCODING => 'DIFF'}
   0 row(s) in 1.9910 seconds
   

   The output lists each table that was created. This job runs several table creation commands, each using the format of create TABLE_NAME. The tables are successfully created when you have output in the form of 0 row(s) in TIME seconds.

   • TABLE_NAME: the name of the table that the job creates
   • TIME: the amount of time it took to create the table
   Note: Bigtable automatically performs data compression, so it disables user-configurable compression at an HBase level.

Data model

The tables that you created store data points from OpenTSDB. In a later step, you write time-series data into these tables. Time-series data points are organized and stored as follows:

Field	Required	Description	Example
metric	Required	Item that is being measured—the default key	sys.cpu.user
timestamp	Required	Unix epoch time of the measurement	1497561091
tags	At least one tag is required	Qualifies the measurement for querying purposes	hostname=www cpu=0 env=prod
value	Required	Measurement value	89.3

The metric, timestamp, and tags (tag key and tag value) form the row key. The timestamp is normalized to one hour, to ensure that a row does not contain too many data points. For more information, see HBase Schema.

Deploying OpenTSDB

The following diagram shows the architecture to deploy OpenTSDB by using Bigtable:

This tutorial uses two OpenTSDB Kubernetes deployments: one deployment sends metrics to Bigtable and the other deployment reads from it. Using two deployments prevents long-running reads and writes from blocking each other. The Pods in each deployment use the same container. OpenTSDB provides a daemon called tsd that runs in each container. A single tsd process can handle a high throughput of events per second. To distribute load, each deployment in this tutorial creates three replicas of the read and write Pods.

1. In Cloud Shell, create a deployment for writing metrics:

   kubectl create -f deployments/opentsdb-write.yaml
   

   The configuration information for the write deployment is in the opentsdb-write.yaml file in the deployments folder of the tutorial repository.

2. Create a deployment for reading metrics:

   kubectl create -f deployments/opentsdb-read.yaml
   

   The configuration information for the reader deployment is in the opentsdb-read.yaml file in the deployments folder of the tutorial repository.

In a production deployment, you can increase the number of tsd Pods that are running, either manually or by using autoscaling in Kubernetes. Similarly, you can increase the number of instances in your GKE cluster manually or by using cluster autoscaler.

Creating OpenTSDB services

In order to provide consistent network connectivity to the deployments, you create two Kubernetes services: one service writes metrics into OpenTSDB and the other reads.

1. In Cloud Shell, create the service for writing metrics:

   kubectl create -f services/opentsdb-write.yaml
   

   The configuration information for the metrics writing service is contained in the opentsdb-write.yaml file in the services folder of the tutorial repository. This service is created inside your Kubernetes cluster and is reachable by other services running in your cluster.

   Note: In a production environment, you can expose the service to the rest of your network by using an internal load balancer or you can expose it to the internet by adding a LoadBalancer in the service definition.

2. Create the service for reading metrics:

   kubectl create -f services/opentsdb-read.yaml
   

   The configuration information for the metrics reading service is contained in the opentsdb-read.yaml file in the services folder of the tutorial repository.

Writing time-series data to OpenTSDB

There are several mechanisms to write data into OpenTSDB. After you define service endpoints, you can direct processes to begin writing data to them. This tutorial uses Heapster to demonstrate writing data. Your Heapster deployment collects data about Kubernetes and publishes metrics from the GKE cluster from which you are running OpenTSDB.

• In Cloud Shell, deploy Heapster to your cluster:

  kubectl create -f deployments/heapster.yaml
  

Examining time-series data with OpenTSDB

You can query time-series metrics by using the opentsdb-read service endpoint that you deployed earlier in the tutorial. You can use the data in various ways. One common option is to visualize it. OpenTSDB includes a basic interface to visualize metrics that it collects. This tutorial uses Grafana, a popular alternative for visualizing metrics that provides additional functionality.

Running Grafana in your cluster requires a similar process that you used to set up OpenTSDB. In addition to creating a ConfigMap and a deployment, you need to configure port forwarding so that you can access Grafana while it is running in your Kubernetes cluster.

1. In Cloud Shell, create the Grafana ConfigMap using the configuration information in the grafana.yaml file in the configmaps folder of the tutorial repository:

   kubectl create -f configmaps/grafana.yaml
   

2. Create the Grafana deployment using the configuration information in the grafana.yaml file in the deployments folder of the tutorial repository:

   kubectl create -f deployments/grafana.yaml
   

3. Get the name of the Grafana Pod in the cluster and use it to set up port forwarding:

   grafana=$(kubectl get pods --selector=app=grafana \
     --output=jsonpath={.items..metadata.name})
   Note:Note:
   kubectl port-forward $grafana 8080:3000
   

   Verify that forwarding was successful. The output is similar to the following:

   Forwarding from 127.0.0.1:8080 -> 3000
   

4. To connect to the Grafana web interface, in Cloud Shell, click Web Preview and then select Preview on port 8080.

   For more information, see Using web preview.

   A new browser tab opens and connects to the Grafana web interface. After a few moments, the browser displays graphs like the following:

   

   This deployment of Grafana has been customized for this tutorial. The files configmaps/grafana.yaml and deployments/grafana.yaml configure Grafana to connect to the opentsdb-read service, allow anonymous authentication, and display some basic cluster metrics. For a deployment of Grafana in a production environment, we recommend that you implement the proper authentication mechanisms and use richer time-series graphs.

Cleaning up

To avoid incurring charges to your Google Cloud account for the resources used in this tutorial, either delete the project that contains the resources, or keep the project and delete the individual resources.

Delete the individual resources

1. Delete the Kubernetes cluster to delete all the artifacts that you created:

   gcloud container clusters delete CLUSTER_NAME
   

   To delete the Kubernetes cluster, confirm by typing Y.

2. To delete the Bigtable instance, do the following:

   1. In the Cloud Console, go to Bigtable.

      Go to Bigtable

   2. Select the instance that you previously created, and then click Delete instance.

Delete the project

Caution: Deleting a project has the following effects:
• Everything in the project is deleted. If you used an existing project for this tutorial, when you delete it, you also delete any other work you've done in the project.
• Custom project IDs are lost. When you created this project, you might have created a custom project ID that you want to use in the future. To preserve the URLs that use the project ID, such as an appspot.com URL, delete selected resources inside the project instead of deleting the whole project.
1. In the Cloud Console, go to the Manage resources page.

   Go to Manage resources

2. In the project list, select the project that you want to delete, and then click Delete.
3. In the dialog, type the project ID, and then click Shut down to delete the project.

What's next

• To learn how to improve the performance of your uses of OpenTSDB, see Bigtable Schema Design for Time Series Data.
• To learn how to migrate from HBase to Bigtable, see Migrating data from HBase to Bigtable.
• The video Bigtable in Action, from Google Cloud Next 17, describes field promotion—an important performance improvement.
• To learn more about default scopes for GKE clusters, see cluster scopes.
• Try out other Google Cloud features for yourself. Have a look at our tutorials.