Edit on GitHub
Report issue
Page history

Sensor data collection and analytics with Cloud IoT Core

Author(s): @kingman ,   Published: 2019-09-25

Google Cloud Community tutorials submitted from the community do not represent official Google Cloud product documentation.


This tutorial shows you how to monitor an environment and conduct an ad-hoc analysis on the collected data.

In this tutorial, you do the following:

  • Set up a Coral Environmental Sensor Board with a Raspberry Pi Zero W with a 40-pin header.
  • Set up Cloud Functions for sensor data processing.
  • Set up BigQuery for data storage.
  • Set up Google Sheets to integrate with BigQuery and IoT Core.
  • Collect sensor data from the Coral Environmental Sensor Board and process the data on Google Cloud Platform.

The architecture is shown in the following diagram:

high-level overview

In this tutorial, the sensor data is generated by the Coral Environmental Sensor Board and streamed to Cloud IoT Core over the MQTT protocol. The sensor data is automatically published on Cloud Pub/Sub by Cloud IoT Core. Data published on Cloud Pub/Sub automatically triggers a Cloud Function that processes the data and stores it in BigQuery. The data is loaded into Google Sheets for analysis using the Sheets data connector for BigQuery. To control the Coral Environmental Sensor Board, command messages are sent from the spreadsheet using the Cloud IoT Core API.

Prerequisites

Costs

This tutorial uses billable components of Google Cloud Platform (GCP), including the following: * Cloud IoT Core * Cloud Pub/Sub * Cloud Functions * BigQuery

This tutorial should not generate any usage that would not be covered by the free tier. You can use the Pricing Calculator to generate a cost estimate based on your projected production usage.

Before you begin

  1. Select or create a GCP project.
  2. Make sure that billing is enabled for your GCP project.
  3. Enable the Cloud IoT, Cloud Functions, Cloud Pub/Sub, and Cloud BigQuery APIs.

When you finish this tutorial, you can avoid continued billing by deleting the resources you created. For more information, see Cleaning up.

Set up the devices

In this section, you set up the Coral Environmental Sensor Board and Raspberry Pi.

Attach the devices

Attach the Coral Environmental Sensor Board to the 40-pin header of the Raspberry Pi and power on the Raspberry Pi by plugging the power cable into it.

board setup

Install the library and driver

Install the Coral Environmental Sensor Board library and driver on the Raspberry Pi:

  1. Follow the instructions for installing the Python library on the Coral Environmental Sensor Board official page.
  2. Reboot the Raspberry Pi board.

Download the tutorial source code to the board

In the Raspberry Pi shell, run the following command to download the necessary source code to the Raspberry Pi:

mkdir -p "$HOME"/enviro-board
cd "$HOME"/enviro-board
wget https://raw.githubusercontent.com/GoogleCloudPlatform/community/master/tutorials/cloud-iot-enviro-board-workshop/enviro-device/cloud_config_template.ini
wget https://raw.githubusercontent.com/GoogleCloudPlatform/community/master/tutorials/cloud-iot-enviro-board-workshop/enviro-device/core.py
wget https://raw.githubusercontent.com/GoogleCloudPlatform/community/master/tutorials/cloud-iot-enviro-board-workshop/enviro-device/enviro_demo.py

Get the public key of the secure element of the sensor board

  1. Run the following command in the Raspberry Pi shell to get the cryptoprocessor public key of the Coral Environmental Sensor Board:

    cd /usr/lib/python3/dist-packages/coral/cloudiot
    
    python3 ecc608_pubkey.py
    
  2. Copy the public key and save it so that you can use it later, when creating the device identity in Cloud IoT Core.

Get the tutorial source code in Cloud Shell

  1. In the GCP Console, open Cloud Shell.
  2. Clone the source code repository:

    cd $HOME
    git clone https://github.com/GoogleCloudPlatform/community.git
    

Create the public key file for the sensor board

The cryptoprocessor exposes an elliptic curve public key, which you got in a previous section. In the current section, you save the public key in Cloud Shell in PEM format and wrap the key with -----BEGIN PUBLIC KEY----- and -----END PUBLIC KEY-----.

  1. Create the key file in Cloud Shell and name it device_pub_key.pem:

    touch $HOME/community/tutorials/cloud-iot-enviro-board-workshop/device_pub_key.pem
    
  2. Add the start wrapper to the key file:

    echo '-----BEGIN PUBLIC KEY-----' >> $HOME/community/tutorials/cloud-iot-enviro-board-workshop/device_pub_key.pem
    
  3. Add the public key to the key file:

    echo 'replace with public key string' >> $HOME/community/tutorials/cloud-iot-enviro-board-workshop/device_pub_key.pem
    
  4. Add the end wrapper to the key file:

    echo '-----END PUBLIC KEY-----' >> $HOME/community/tutorials/cloud-iot-enviro-board-workshop/device_pub_key.pem
    

Set up the device identity on GCP

For a device to communicate with Cloud IoT Core, the device identity needs to be created in Cloud IoT Core.

  1. To make it easier to run commands when you create cloud resources, set environment variables in Cloud Shell to hold the names and properties of the resources:

    export EVENT_TOPIC=enviro-event
    export REGISTRY_ID=enviro-registry
    export REGION=europe-west1
    export DEVICE_ID=enviro-board
    export DATASET=enviro_dataset
    export TABLE=sensor_data
    
  2. Create the Cloud Pub/Sub topic:

    gcloud pubsub topics create $EVENT_TOPIC
    
  3. Create the Cloud IoT Core registry:

    gcloud iot registries create $REGISTRY_ID \
      --region $REGION \
      --event-notification-config=topic=$EVENT_TOPIC
    
  4. Create the the sensor board identity in the newly created Cloud IoT Core registry with the public key:

    cd $HOME/community/tutorials/cloud-iot-enviro-board-workshop/
    gcloud iot devices create $DEVICE_ID \
      --region=$REGION \
      --registry=$REGISTRY_ID \
      --public-key=path=device_pub_key.pem,type=es256
    

Verify the data ingestion setup

You now have the building blocks set up to ingest data from the Coral Environmental Sensor Board to GCP.

In this section, you verify the end-to-end integration between the sensor board and Cloud Pub/Sub.

Create the event topic subscription

Messages sent from the device to Cloud IoT Core are automatically published on Cloud Pub/Sub.

Create a subscription to the Cloud Pub/Sub topic:

gcloud pubsub subscriptions create verify-event \
--topic=$EVENT_TOPIC

Later, you use the subscription to get the messages.

Configure the Raspberry Pi to send sensor data to Cloud IoT Core

  1. In the Raspberry Pi shell, set your GCP project ID as an environment variable:

    export PROJECT_ID=your-gcp-project-id
    
  2. In the Raspberry Pi shell, set your GCP project ID in the device application configuration file with this command:

    envsubst < $HOME/enviro-board/cloud_config_template.ini > $HOME/enviro-board/cloud_config.ini
    

Download the CA certificate

Run the following in the Raspeberry Pi shell to download the Google root CA certificate:

cd $HOME/enviro-board/
wget https://pki.goog/roots.pem

The CA certificate is used to establish the chain of trust to communicate with Cloud IoT Core using the Transport Layer Security (TLS) protocol.

Run the streaming script

The streaming script reads the sensor measurement values of the sensor board and publishes the data to Cloud Pub/Sub through Cloud IoT Core.

  1. Start the script with the following commands in the Raspberry Pi shell:

    cd $HOME/enviro-board/
    python3 enviro_demo.py --upload_delay 10
    
  2. Let the script run for 20 seconds.

  3. Press Ctrl+C to stop the script.

Verify sensor data in Cloud Pub/Sub

The sensor values sent from the Raspberry Pi are published to Cloud Pub/Pub. In this step, you pull the messages from the Cloud Pub/Sub subscription created in a previous step and verify that the values are delivered to Cloud Pub/Sub.

  1. Run the following command in Cloud Shell to pull messages from the Cloud Pub/Sub subscription:

    gcloud pubsub subscriptions pull verify-event --auto-ack
    
  2. Verify that you get the messages from the Coral Environmental Sensor Board.

Set up the Cloud Function to process sensor data

In this section, you deploy the Cloud Function that gets triggered by the sensor data messages published to Cloud Pub/Sub. The function parses the message and adds the values to BigQuery.

In Cloud Shell, run the following:

cd $HOME/community/tutorials/cloud-iot-enviro-board-workshop/functions
gcloud functions deploy enviro \
 --set-env-vars=DATASET=${DATASET},TABLE=${TABLE} \
 --region ${REGION} \
 --trigger-topic ${EVENT_TOPIC} \
 --runtime nodejs8 \
 --memory 128mb

Set up data storage

In this section, you create the data set and table in BigQuery to store the sensor data.

In Cloud Shell, run the following:

bq mk $DATASET

bq mk ${DATASET}.${TABLE} $HOME/community/tutorials/cloud-iot-enviro-board-workshop/bq/schema.json

Start the sensor data stream

In this section, you restart the demonstration script on the Raspberry Pi to generate sensor data that triggers the Cloud Function to store the data in BigQuery. You can control the interval at which the sensor data is sent to GCP by setting the upload_delay parameter.

In the Raspberry Pi shell, run the following:

cd $HOME/enviro-board/
python3 enviro_demo.py --upload_delay 15

View sensor data in BigQuery

  1. Open the BigQuery console.
  2. Paste the following query into the Query editor:

    SELECT * FROM `[PROJECT_ID].enviro_dataset.sensor_data`
    ORDER BY time DESC
    LIMIT 20
    

    Replace the placeholder [PROJECT_ID] with your project ID.

  3. Click Run.

  4. Verify that a table with sensor data is returned.

  5. Press Ctrl+C to stop the sensor data streaming from the Raspberry Pi shell.

Perform data analytics and device control

In this section, you start from the sample spreadsheet and configure it to retrieve data from your cloud project and send commands to your device.

Set up Google Sheets

  1. Open the sample spreadsheet.
  2. Click File > Make a copy.
  3. Enter a name for the copy, and click OK.

Configure OAuth2

Create an OAuth client and configure your spreadsheet to use the client:

  1. In Google Sheets, select OAuth Configuration in the IoT menu.

    This opens the Oauth credential configuration dialog box.

  2. Copy the authorized redirect URI shown in the Oauth credential configuration dialog box.

  3. Go to the Credentials page.

  4. Click Create credentials, and choose OAuth client ID.

  5. In the Application type section, choose Web application.

  6. Give the credential an identifiable name.

  7. Paste the authorized redirect URI that you retrieved in the second step into the Authorized redirect URIs field.

  8. Click Create.

  9. Make a copy of the client ID and the client secret that are shown after the client is created.

  10. Go back to your spreadsheet and paste the client ID and the client secret values into their corresponding fields in the Oauth credential configuration dialog box.

  11. Click Save, and wait for the config saved! response.

  12. Click Close.

Configure Cloud IoT Core

In this section, you configure your spreadsheet with your GCP project-specific values. The spreadsheet needs these values to retrieve device data from Cloud IoT Core.

  1. In Google Sheets, select IoT Core Configuration in the IoT menu.
  2. Fill in values of your GCP project, Cloud IoT Core region and registry.
  3. Click Save, and wait for the config saved! response.
  4. Click Close.

Load devices

In this section, you go through the authorization steps that enable the spreadsheet to retrieve information from GCP, and you use the spreadsheet to fetch your device ID from Cloud IoT Core.

  1. In Google Sheets, select Load Devices in the IoT menu.

    The first time you use this command, a sidebar opens.

  2. Click Authorize in the sidebar and continue with the authorization to allow the script to access the Cloud IoT Core API on your behalf.

  3. When the OAuth2 flow is successful, a new tab opens with this message: Success! You can close this tab.

  4. Close the tab.

  5. Select Load Devices in the IoT menu again, and verify that your device name appears in the Devices column.

  6. Click the checkbox next to the device name cell to select it.

Set up the BigQuery connector

In this section, you configure the connector between Google Sheets and BigQuery to load data from your sensor_data table in BigQuery.

  1. In Google Sheets, choose Data > Data connectors > Connect to BigQuery.
  2. Choose your GCP project from the list and click Write query.
  3. Paste the following query into the BigQuery query editor, replacing the placeholder [PROJECT_ID] with your project ID:

    SELECT * FROM `[PROJECT_ID].enviro_dataset.sensor_data`
    WHERE time > TIMESTAMP(CURRENT_DATE())
    AND device_id = @DEVICE_ID
    ORDER BY time
    
  4. Add a new parameter:

    1. Click Parameters > Add.
    2. Fill in DEVICE_ID for Name and Sheet1!C2 for Cell reference.
    3. Click Add.
  5. Click Insert result.

Send commands to your board

In this section, you use your spreadsheet to send arbitrary string messages to your Raspberry Pi, and the messages appear on the OLED display of the Coral Environmental Sensor Board.

  1. In the cell under Command, write a simple text message.
  2. Choose IoT > Send command to device.
  3. Verify that you get a Device is not connected error message when the demo script is turned off on your Raspberry Pi.
  4. Restart the demo script from your Raspberry Pi shell by running the following: python3 enviro_demo.py
  5. Click IoT > Send command to device to resend the message.
  6. Verify that the message appears on the OLED display.

Analytics in Google Sheets

Here are some things that you can do to further experiment with the data and device command functions from within your spreadsheet:

  • Create time series graphs over the sensor data.
  • Derive moving time average values from raw sensor data.
  • Create recurring jobs that auto-load the data from BigQuery.
  • Create function that monitors the sensor values and automatically sends commands when configured threshold values are reached or exceeded.

Cleaning up

To avoid incurring charges to your Google Cloud Platform account for the resources used in this tutorial, you can delete the project.

The easiest way to eliminate billing is to delete the project you created for the tutorial:

  1. In the GCP Console, go to the Projects page.
  2. In the project list, select the project you want to delete, and click Delete.
  3. In the dialog, type the project ID, and then click Shut down to delete the project.

Next steps

Submit a tutorial

Share step-by-step guides

Submit a tutorial

Request a tutorial

Ask for community help

Submit a request

View tutorials

Search Google Cloud tutorials

View tutorials

Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 4.0 License, and code samples are licensed under the Apache 2.0 License. For details, see our Site Policies. Java is a registered trademark of Oracle and/or its affiliates.