Guest post: Building IoT applications with MQTT and Google Cloud Pub/Sub

[Editor’s note: Today we hear from Agosto, a Google Cloud Premier Partner that has been building products and delivering services on Google Cloud Platform (GCP) since 2012, including Internet of Things applications. Read on to learn about Agosto’s work to build an MQTT service broker for Google Cloud Pub/Sub, and how you can incorporate it into your own IoT applications.]

One of our key practice areas is Internet of Things (IoT). Using the many components of GCP, we’ve helped customers rapidly move their ideas from product concept to launch.

Along the way, we evaluated several IoT platforms and repeatedly came to the conclusion that we’d be better off staying on the GCP stack than a single IoT platform with costly licensing hooks and closed-source practices. Our clients also like being able to build scalable, functional prototypes using pre-existing and standard reference architectures and tools.

One of the many challenges we faced along the way was picking an efficient transport for two-way messaging between “things” and GCP. After evaluating a number of emerging and mature protocols, we settled on Message Queuing Telemetry Transport (MQTT). The MQTT protocol has been around since the early 2000’s and is now an ISO Standard. Originated in 1999 by Andy Stanford-Clark and Arlen Nipper, it's lightweight, has solid documentation and has tens of thousands of production deployments. Furthermore, many existing pre-IoT or “Machine to Machine” projects already use MQTT as their transport from embedded device to the back-office. With MQTT, we’ve been able to increase velocity and reduce complexity for our IoT products and services.

MQTT is a great transport protocol, but it can be challenging to manage at scale, particularly when it comes to scaling message storage and delivery systems. As one of the earliest Google partners to develop a set of reusable tools, reference architectures and methods for accelerating IoT products to market, we’ve been impressed with Google Cloud Pub/Sub, a durable, low-latency and scalable service for handling many-to-many asynchronous messaging. But Cloud Pub/Sub uses HTTPS to transfer data. Over numerous small requests, all those HTTP headers add up to a lot of extra data — a no-go when you’re dealing with a constrained device that communicates over a mobile network, and where you pay for each byte in mobile data charges, battery usage — or both.

We needed to bridge the gap between IoT-connected devices and Cloud Pub/Sub, and began investigating ways to connect MQTT to Cloud Pub/Sub using and extending RabbitMQ.

After initial load tests showed this approach was viable, Google asked Agosto to develop an open-source, highly performant MQTT connection broker that integrates with Cloud Pub/Sub. With low network overhead (Agosto has seen up to 10x less compared to HTTPS in scenarios we've tested) and high throughput, MQTT is a natural fit for many scenarios.

The resulting message broker integrates messaging between connected devices using a MQTT client and Cloud Pub/Sub; RabbitMQ performs the protocol conversion for two-way messaging between the device and Cloud Pub/Sub. This means administrators of the RabbitMQ compute infrastructure don't have to concern themselves with managing the durability of the data, or scaling storage.

Our message broker can support both small and very large GCP projects. For example, with smaller projects and IoT prototypes, you can rapidly deploy a single node of Agosto’s MQTT to Pub/Sub Connection Broker supporting up to 120,000 messages per minute for as little as $25/month for the compute costs. Larger production deployments with load-balanced brokers can support millions of concurrent connections and much higher throughput.

Download the broker, follow the instructions and learn more about leveraging MQTT and GCP for your IoT project.
GitHub: https://github.com/Agosto/gcp-iot-adapter

And if you're looking for a more customized implementation of our MQTT to Pub/Sub Connection broker, visit our website to learn more about our offerings.