Huq: Building a powerful data platform to deliver near real-time location data at scale
About Huq
By using first-party location data to measure footfall in retail, industrial, and transport environments, Huq enables its customers to make more informed strategic decisions with near real-time economic data.
Tell us your challenge. We're here to help.
Contact us
About DoiT International
DoiT finds saving potential and optimization in every cloud or technology stack, helping businesses build and run powerful cloud solutions at scale.
DoiT International supported Huq's data pipeline migration to Google Cloud, providing the right tools to optimize its stack so it can respond to customer demands in real time, while controlling costs.
Google Cloud results
- Supports Huq in building a highly flexible Google Cloud stack to meet customer demands in near real time
- Offers immediate access to proprietary tools including superQuery, to increase BigQuery efficiency
- Provides support on a transparent basis that will adapt flexibly to evolving projects
50% cost reduction enables deeper investment in product
To adapt decisively to changing customer and market trends, it’s mission critical for businesses to understand their market environment. In a world where business happens equally in stores and online, it’s important to keep on top of both. As a data company, Huq measures the real-world factors that affect its customers to help them gain a holistic view of their marketplace and to make more successful business decisions.
Huq produces a wide range of economic and societal benchmarks using mobility data that serves as a proxy for footfall in retail, transport, town centers, and workplaces. This anonymized resource—updated in near real time—incorporates high volumes of rich, descriptive data that enables complex analysis of consumer behavior and how that evolves. It’s an offering that’s becoming even more important for Huq’s customers as they navigate the wide-scale disruption to movement and footfall brought on by COVID-19.
"Our customers require increasingly fast turnaround times and more regular access to high-frequency data," says Isambard Poulson, co-founder and CTO at Huq. "While previously a hedge-fund customer might use our data to make mid-term or long-term decisions, customers of all sizes are now keen to understand changes in near real time: how new policies impact behavioral patterns, for example, providing an up-to-date view of how a particular country's economy is faring as a result of COVID-19."
"We're a compact technical team, which means we're able to move very fast and rapidly respond to changes. But when you're adopting new technologies, you need to leverage cutting-edge expertise. The key reasons we chose DoiT International were the quality of communication and the team’s deep technical know-how."
—Isambard Poulson, co-founder and CTO, HuqHuq prides itself on being able to adapt to its customers’ needs. To make this happen, the company has to be extremely flexible. "Our mission is to make our data easily accessible by offering the right level of granularity for a customer's needs," says Isambard. "That means people with limited data engineering skills can use our analytics products via dashboard solutions, or simply receive a visualized PDF report, while more sophisticated customers might access up to billions of rows of anonymized geospatial data per day." To accommodate these needs, Huq decided to migrate to Google Cloud in 2019 so that it could more easily adapt its data pipeline and react quickly to evolving customer demands.
After kicking off the project, Huq realized that it needed support to solve its day-to-day technical questions. It also wanted to make sure it was using Google Cloud in the most efficient way possible, especially when it came to controlling costs. Without effective cost controls, its ability to respond quickly to customer demands and develop innovative products would be limited by the costs associated with high usage. Looking for a partner with proven experience in cloud optimization, it decided to work with DoiT International.
"We're a compact technical team, which means we're able to move very fast and rapidly respond to changes," says Isambard. "But when you're adopting new technologies, you need to leverage cutting-edge expertise. The key reasons we chose DoiT International were the quality of communication and the team’s deep technical know-how."
Finding the right solutions based on a tailored approach
Isambard first turned to DoiT in May 2020 on the recommendation of another DoiT client, the CTO of a company in the investment sector with similar needs. "Having gone through previous evaluations with potential partners, I was initially skeptical of what they could offer us, but the client had a similar profile to us, so we knew it could be a good fit," says Isambard. "It was beneficial to have a personal recommendation." Huq has an agnostic attitude to providers, preferring open-source to proprietary technologies, so when looking for implementation support, it wanted a partner that kept its options open. "We didn't want to be tied into a relationship where we might lose control of our internal processes," says Isambard.
The DoiT International team impressed Huq with its tailored approach. "Many partners come in with an agenda, making suggestions based on their existing product knowledge rather than listening to your needs," says Isambard. "DoiT took the time to gain a deep understanding of our requirements and offer solutions that aligned with those needs. We didn't feel tied into anything we couldn't walk away from."
"Our business model is built around flexibility," says Rebeca Fenoy-Anthony, Cloud Business Lead at DoiT International. "As a reseller for Google Cloud, our on-demand engagement model enables our customers to enter into a relationship with us, without any fixed constraints.”
These initial meetings reassured Huq that DoiT was the right fit. "The onboarding process was very smooth, and the tools that were immediately going to be available to us were highly compelling," says Isambard.
Offering support that inspires clients with confidence
When DoiT came on board, Huq had already begun its migration to Google Cloud. "The first goal for the project was to migrate our core data processing pipeline from AWS to Google Cloud using Apache Beam and Dataflow," says Isambard. Huq's enriched data asset comprises 24 billion new rows of anonymized geospatial data every month, along with 200 million rows of continually refreshed spatial reference data. Its data record incorporates 73 properties with diverse data types over five years of history, so its key requirements are being able to quickly run queries over this extensive data asset and build and maintain multiple custom data pipelines.
The Huq team had completed the early stages of the implementation itself, but still had technical hurdles to overcome. DoiT helped to provide solutions, such as optimizing the machine types Huq uses on Dataflow. With DoiT's support, Huq built a data pipeline that rapidly ingests, processes, analyzes, and exports almost one billion geo-location records every day. To do that, Huq uses Cloud Composer for its workflow orchestration service to schedule tasks such as loading the core data and managing the daily processing pipelines. On arrival, the source data is first loaded into Cloud Storage and enriched with Dataflow. Once the data has been processed, multiple pipelines transport it back into Cloud Storage or BigQuery for clients to access or export it as Looker Studio reports.
"As well as providing rapid answers to our questions, DoiT International reassured us that we were moving in the right direction," says Isambard. "That gave us the confidence to continue our migration, and we now have an architecture that offers scale and flexibility on an economically viable basis."
"We use BigQuery extensively, so having unlimited access to the superQuery IDE tool is very helpful for us. It enables us to optimize the way we query our data, making it more efficient so we can respond more flexibly to our customers."
—Isambard Poulson, co-founder and CTO, HuqStaying on top of costs with reOptimize CMP and superQuery IDE
One of the key goals for DoiT was to optimize Huq's costs on Google Cloud. "As a rapidly growing company, we want to grow our volume without increasing costs," says Isambard. That's critical because, without optimization, the cost of running queries could become too high for Huq to continue offering the high-volume, quick-response approach its clients rely on. "It's not just a question of our bottom line or profitability," says Isambard. "Without optimizations, we're not able to iterate rapidly and deliver for our clients. It has a tangible impact on our outputs."
To do that, Huq uses a DoiT International technology platform called reOptimize, which provides enhanced visibility into cloud usage by breaking it down according to services, project, or cost allocation, as well as predictive analytics and other cost-optimization functions. In addition to offering tips on how to use BigQuery efficiently, such as using Flex Slots to respond to rapid demand, working with DoiT gives Huq access to superQuery, an AI-powered integrated development environment (IDE) for BigQuery.
"We use BigQuery extensively, so having unlimited access to the superQuery IDE tool is very helpful for us," says Isambard. "It enables us to optimize the way we query our data, making it more efficient so we can respond more flexibly to our customers."
"On the cloud, it can be hard to keep control of costs," says Rebeca. "However, with a combination of the right engineering power, the right tools, and the right alerts, you can stay on top of those costs and make sure the cloud is working for you."
"We're consuming about four times more data on a daily basis, from around 200 million data points a day to 800, but at the same time, our costs have fallen by 50%. In the context of a challenging economic environment, that's something to be proud of."
—Isambard Poulson, co-founder and CTO, HuqHelping Huq to develop new services and stay competitive
With the help of DoiT International, Huq has been able to cut its cloud bill in half even while its data needs quadrupled. "Our volumes of data have grown rapidly since we started working with DoiT," says Isambard. "We're consuming about four times more data on a daily basis, from around 200 million data points a day to 800, but at the same time, our costs have fallen by 50%. In the context of a challenging economic environment, that's something to be proud of."
Now that it's finished its migration and is able to rapidly provide access for customers at a wide range of granularities, Huq wants to focus on scaling up its business. "With our new stack, we could increase the volume of data we process 100-fold, without any changes to our stack" says Isambard. That means Huq can focus on creating new products for its customers, opening up the possibility of tracking particular sectors in real time.
"I'm really looking forward to seeing what we can achieve with Google Cloud and DoiT International," says Isambard. "We're in the process of continuous evolution, and DoiT will support us in the process of expanding our products, helping us to explore the technologies we need to generate those new services."
Tell us your challenge. We're here to help.
Contact us
About Huq
By using first-party location data to measure footfall in retail, industrial, and transport environments, Huq enables its customers to make more informed strategic decisions with near real-time economic data.
About DoiT International
DoiT finds saving potential and optimization in every cloud or technology stack, helping businesses build and run powerful cloud solutions at scale.