Over-the-air software updates for embedded Linux with Mender on Google Cloud
Contributed by Google employees.
This tutorial demonstrates how to successfully deploy an over-the-air (OTA) software update solution for embedded Linux devices using Mender on Google Cloud.
- Deploy a Mender management server on GCE
- Integrate device identity and lifecycle between IoT Core and Mender
- Configure a live sample device with a base image
- Push an over-the-air update to the device with an IoT Core application
Before you begin
This tutorial assumes that you already have a Google Cloud account set up and have completed the getting started guide, including enabling the IoT Core API. You need to associate Firebase with your Google Cloud project. To add Firebase to your Google Cloud project, use the Firebase Console and choose Add project. Select your Google Cloud project and click Add Firebase.
- For most commands, we recommend that you use Cloud Shell. If you want to use your local command line, you need the Cloud SDK and Firebase tools.
- You need to ensure that the required environment variables are set in each shell environment. Use the variables shown below for every new session.
- To work with the part of the tutorial that images a real device, you need to have a Raspberry Pi 3 device, SD Card, and the ability to plug the Raspberry Pi in to Ethernet on your LAN (Wi-FiCloud configurations are not supported in this tutorial).
This tutorial uses billable components of Google Cloud, including the following:
- IoT Core
- Compute Engine
- Cloud Storage
- Cloud Functions for Firebase
- Cloud Logging
Use the Pricing Calculator to generate a cost estimate based on your projected production usage.
Mender is an open source remote update manager for embedded Linux devices. The aim of the project is to help secure connected devices by providing a robust and easy software update process.
Some of the key features of Mender:
- OTA update server and client
- Full system image update
- Symmetric A/B image update client
- Bootloader support: U-Boot and GRUB
- Volume formats: MBR and UEFI partitions
- Update commit and rollback
- Build system: Yocto Project (meta-mender)
- Remote features: deployment server, build artifact management, device management console
More information on Mender can be found here.
Mender management server: The central point for deploying updates to a population of devices. Among other things, it monitors the current software version that is installed on each device and schedules the rollout of new releases.
Mender build system: The software build system generates a new version of software for a device. The software build system is a standard component, such as the Yocto Project. It creates build artifacts in the format required by the target device. There will be different build artifacts for each type of device being managed.
Mender client: Each device runs a copy of the Mender update client, which polls the Management Server from time to time to report its status and to discover if there is a software update waiting. If there is, the update client downloads and installs it.
Mender on GCP: high-level architecture diagram
The following architecture diagram provides a high-level overview of the various components on Google Cloud to enable OTA updates with Mender and Google Cloud IOT Core:
Setting up Mender Server on Compute Engine
Mender Management Server deployment options
There are several options for successfully setting up Mender services with Google Cloud. This tutorial will use a minimally configured Mender Management Production Server to test the end-to-end workflow:
Mender Management Demo Server: For quickly testing the Mender server, Mender provides a pre-built demo version that does not take into account production-grade issues like security and scalability.
Mender Management Production Server: Mender Server for production environments includes security and reliability aspects of Mender production installations.
Hosted Mender Service: Hosted Mender is a secure management service, so you don't have to spend time maintaining security, certificates, uptime, upgrades, and compatibility of the Mender server. Simply point your Mender clients to the Hosted Mender service.
Preparing the project and shell environment
Mender Management server requirements from Mender are outlined here. We will be using the base instructions as documented for setting up a production environment and deploying on Google Cloud. However, this is minimally configured and not suited for actual production use.
Set up the Google Cloud Shell environment. (You will use several different shell environments.)
If you are not using Cloud Shell you must first run these commands in your local environment:
gcloud auth login gcloud config set project [MY-PROJECT] # replace with the name of your project
Enable some of the APIs we will be using:
gcloud services enable compute.googleapis.com cloudiot.googleapis.com pubsub.googleapis.com
The Compute API takes a minute or two to enable.
Open firewall ports so that we can reach the Mender server after it is installed:
gcloud compute firewall-rules create mender-ota-443 --allow tcp:443 gcloud compute firewall-rules create mender-ota-9000 --allow tcp:9000
Set environment variables that we will use in later commands:
export FULL_PROJECT=$(gcloud config list project --format "value(core.project)") export PROJECT="$(echo $FULL_PROJECT | cut -f2 -d ':')" export CLOUD_REGION='us-central1'
Create two Cloud Storage buckets that you will use for images and updates:
gsutil mb -l $CLOUD_REGION gs://$PROJECT-mender-server gsutil mb -l $CLOUD_REGION gs://$PROJECT-mender-builds
Installing Mender Management Server
gcloud beta compute --project $PROJECT instances create "mender-ota-demo" --zone "us-central1-c" --machine-type "n1-standard-2" --subnet "default" --maintenance-policy "MIGRATE" --scopes "https://www.googleapis.com/auth/cloud-platform" --metadata=startup-script-url=https://raw.githubusercontent.com/GoogleCloudPlatform/community/master/tutorials/cloud-iot-mender-ota/server/mender_server_install.sh --min-cpu-platform "Automatic" --tags "https-server" --image "ubuntu-1604-xenial-v20180814" --image-project "ubuntu-os-cloud" --boot-disk-size "10" --boot-disk-type "pd-standard" --boot-disk-device-name "mender-ota-demo"
The startup script will take roughly 3-5 minutes to completely install all the prerequisites including Docker CE, Docker compose and Mender Server.
Navigate to the Mender UI by clicking on the external IP address of mender-ota-demo, which can be found on the Cloud Console Compute Engine page. In most browsers you will get a certificate warning and you will need to click advanced and proceed or similar. In an actual production environment, you would provision this server with a trusted certificate.
When you are on the Mender UI login page, using credentials created by the startup script will take you to the Mender Dashboard.
- Username - email@example.com
- Password - mender_gcp_ota
Congratulations! You just finished creating the Mender Server on Google Cloud.
Hosted Mender Service
The above steps are for self-managed Open Source Mender Management Server on Google Cloud. Mender also provides fully managed Hosted Mender service.
The next section describes how to use a Yocto Project image for a raspberry Pi3 device.
Imaging device with pre-built Mender Yocto Images
This section outlines the steps involved in configuring and working directly with provided prebuilt images.
If you would like to learn how to build your own images on GCE with Yocto, see these extra instructions.
Using the existing Cloud Shell environment, clone the source repository for this tutorial.
git clone https://github.com/GoogleCloudPlatform/community.git
Update the prebuilt image with some configuration specific for your project.
This will take a couple minutes to pull the default image, update with configuration for your project, and then upload it back to one of the buckets you created earlier.
To flash this image, we need to switch back to your local machine.
You can use
gsutil on your machine as suggested by the update script's output, or download this updated image from the console from the storage browser; it will be in the [project]-mender-builds folder.
Provisioning a new device (writing the image to the Raspberry Pi3 device)
Note: You may also want to use the Etcher GUI tool instead of the
ddcommand line tool outlined below.
Insert the SD card into the SD card slot of your local PC where you have the
Use this command to determine where the drive is mounted:
Unmout the drive:
on OS X:
diskutil unmountDisk /dev/disk3 (assuming /dev/disk 3 is SD card)
Enter this command to write the image to the SD card, and adjust the local path to your
sudo dd if=/Users/LOCAL_PATH/updated-demo-image-raspberrypi3.img of=/dev/disk2 bs=1m && sudo sync
Depending on the image size, it may take roughly 20 minutes, so please be patient until the image is completely written to the SD card.
Next, you will configure the device to connect to Mender Management Server and IoT Core with the same private/public key pair.
Mender Client integration: IoT Core and Mender Management Server
This section outlines the steps to connect the Mender Client (running on a Raspberry Pi3 device) to IoT Core as well as Mender Server using the same public/private key. Additionally the steps will deploy an OTA update to the device remotely as depicted in the diagram below.
Key components you will use in this section:
- IoT Core
- Cloud Functions/Firebase Functions
- Cloud Logging
- Mender Server on Google Cloud
- Raspberry Pi3 (device/client)
Using the Cloud Shell environment, create an IoT Core registry and Pub/Sub topic for this tutorial that will be used for the device to authenticate and send telemetry data.
export REGISTRY_ID=mender-demo gcloud pubsub topics create iot-telemetry gcloud pubsub topics create registration-events gcloud iot registries create $REGISTRY_ID --region=$CLOUD_REGION --event-notification-config=subfolder="",topic=iot-telemetry
Integrate IoT Core device lifecycle events and Mender Preauthorization
Using the Cloud Shell environment, you will configure IoT Core audit logs to route to a Pub/Sub topic.
Create a log export for IoT Core device creation events to Pub/Sub:
gcloud beta logging sinks create device-lifecyle \ pubsub.googleapis.com/projects/$PROJECT/topics/registration-events \ --log-filter='resource.type="cloudiot_device" protoPayload.methodName="google.cloud.iot.v1.DeviceManager.CreateDevice"'
Give the log exporter system-account permission to publish to your topic:
gcloud beta pubsub topics add-iam-policy-binding registration-events \ --member $(gcloud beta logging sinks describe device-lifecyle --format='value(writerIdentity)') \ --role roles/pubsub.publisher
Deploy Firebase Functions to call Mender Preauthorization API
This will link device creation in IoT Core to Device preauthorization in Mender.
Be sure you associated Firebase with your Cloud project as noted in Before you begin.
Deploy Firebase Functions to subscribe to the
registration-events Pub/Sub topic that you created in the last step to preauthorize IoT Core Devices with the Mender Server every time a new device is created in IoT Core.
Using the repo you cloned into Cloud Shell earlier, switch to the
cd ~/community/tutorials/cloud-iot-mender-ota/auth-function/functions firebase login --no-localhost firebase use --add $PROJECT
Let's set the environment variables for the functions. Replace the IP address for
mender.url with the external IP address of your Mender server.
You may be prompted to verify your compute instance zone, the default may be incorrect, so answer
export GCP_IOT_MENDER_DEMO_HOST_IP_ADDRESS=$(gcloud compute instances describe mender-ota-demo --project $PROJECT --format="value(networkInterfaces.accessConfigs.natIP)") firebase functions:config:set mender.url=https://$GCP_IOT_MENDER_DEMO_HOST_IP_ADDRESS firebase functions:config:set firstname.lastname@example.org firebase functions:config:set mender.pw=mender_gcp_ota npm install firebase deploy --only functions
Create the device in IoT Core
Let's bring up the Raspberry Pi device and extract the public key, so you can create a device in the IoT Core registry. The same private/public key pair will be used to authorize the device in Mender Server as well.
On your local PC, open terminal or console to perform the following commands. This needs to be a shell that has access to the Raspberry Pi on your local network, so it cannot be Cloud Shell. Find and add the IP address of your Raspberry Pi device below. To locate the IP address of your Raspberry Pi device, you can invoke the
nmap command for host discovery, as shown below, by replacing the subnet range with one that matches your own local network.
sudo nmap -sn 192.168.1.0/24 export DEVICE_IP=YOUR_RASPBERRY_PI_IP_ADDRESS
We use a random ID generated by the OS on first boot as our device ID in IoT Core. This can be adapted to any potential HW based identifier such as a board serial number, MAC address, or crypto key id:
export DEVICE_ID=$(ssh root@$DEVICE_IP /usr/share/mender/identity/mender-device-identity| head -n 1 | cut -d '=' -f 2)
You will be prompted several times for the root password, which is
Extract the public key from the device:
scp root@$DEVICE_IP:/var/lib/mender/rsa_public.pem ./rsa_public.pem
Now create an IoT Core Device with the public key (
rsa_public.pem), which you extracted in the last step (please make sure you are in the same directory where you have extracted the
rsa_public.pem file). Run the following command from the same local console or terminal where you have ssh access to the device. You may need to set your project in
gcloud config set project YOUR_PROJECT_ID export REGISTRY_ID=mender-demo export CLOUD_REGION=us-central1 # or change to an alternate region; export PROJECT=$(gcloud config list project --format "value(core.project)") gcloud iot devices create $DEVICE_ID --region=$CLOUD_REGION --project $PROJECT --registry=$REGISTRY_ID --public-key path=rsa_public.pem,type=RSA_PEM
If you do not have
gcloud installed locally, you can create this device in the Cloud Console.
After the device is created in IoT Core, the Firebase function deployed earlier will make REST API call to the Mender Server to preauthorize the device with the same public key credentials used to create the device in IoT Core. After the preauthorization and device activation steps are complete, the function will push a config update to IoT Core, which will configure the Mender client on the device with the specific IP address of the Mender server.
This may take a couple minutes, because the client on the device has backed off from constantly trying to reconnect to IoT Core. You can monitor the IoT Core console Configuration & state history pane to see when the config has been pushed and acknowledged by the device.
Verify the device heartbeat in Mender Server and IoT Core
You can confirm the same from the Cloud Console as shown below under device details.
Now open the Mender Management Server, and make sure you are able to see that the device is authorized and able to communicate.
Sign in to the Mender Server that you created part of the earlier steps: Mender Management Server on Google Cloud. Click Devices to make sure you can see the Raspberry Pi 3 device as shown below.
This confirms that the device has successfully connected to IoT Core and Mender Server with the same private/public key.
Push an OTA software update
As part of the last step, let's perform an over-the-air (OTA) update by deploying a Mender artifact from Mender Server to the client. The Mender artifact includes a software update of MQTT for IoT Core example, which will be deployed and executed on device boot.
Create a subscription that we can use to verify this sample gets deployed:
gcloud beta pubsub subscriptions create test-reader --topic iot-telemetry
First, let's download the Mender artifact
gcp-mender-demo-image-raspberrypi3.mender, part of the Build step from the GCS bucket (or download the sample from here, and let's upload to the Mender Server under artifacts as shown below.
Next, you need to create a deployment and select the device to which you want to deploy the artifact. From Mender Server, click Create Deployment, and select the target artifact as release-2 and group all devices. Click Create deployment. You only have one device currently, which is
Raspberry Pi3. For various classes and types of devices that Mender supports, you can create groups and apply the target artifacts accordingly (Raspberry Pi3, Beaglebone, etc.)
While the deployment completes, you can verify that the current image is not sending data:
gcloud beta pubsub subscriptions pull test-reader --limit 100 --auto-ack
Deployment completion may take some time, as the update agent checks in periodically and writes the updated partition. Progress can be monitored from the Mender Server Dashboard by clicking In progress deployments.
After the deployment is finished, you should be able to see that the deployment is successful from the Mender Dashboard, and the new release of the software update should be deployed on the device, which can be confirmed by logging into the device and running
mender -show-artifact. This should output
release-2. Additionally you can confirm the telemetry events are being sent by the software update by repeating the pull command:
gcloud beta pubsub subscriptions pull test-reader --limit 100 --auto-ack
This completes the tutorial where you have successfully deployed a Mender OTA solution on Google Cloud, including building a Mender Yocto custom embedded OS image for a Raspberry Pi device, and integrated with an IoT Core solution.
Since this tutorial uses multiple Google Cloud components, please be sure to delete the Google Cloud resources after you are done with the tutorial. Ideally, you have done this tutorial in an ephemeral project created just for this tutorial, which allows you to clean up resources simply by deleting the project.
If you would like to see how to further update the images by modifying the device software and doing an additional deployment, see the extra instructions on building your own images.