TELUS: Modernizing a legacy on-prem data stack with Google Cloud
Decommissioned 30% of obsolete and unused data through modernization for substantial OpEx savings
Modernized and optimized hundreds of enterprise-grade data pipelines, 3,000 ETL mappings, and 1,700 tables to Google Cloud
Enabled TELUS employees to leverage 14+ PB of structured, enterprise-grade data for improved analytics and gen AI use cases at scale
Created an integrated self-serve, cloud-powered, data analytics platform in BigQuery
Established a robust, self-serve architecture to enable data exploration and data productionalization at scale across the enterprise
TELUS collaborated with Google Cloud and Onix to transform how it captures and manages data, which significantly improved the data-driven decisions toward its gen AI and analytics capabilities.
TELUS collaborated with Google Cloud and Onix to transform how it captures and manages data, which significantly improved the data-driven decisions toward its gen AI and analytics capabilities.
Modernizing 200+ data pipelines with 100+ systems of record and 14 PB of data
The speed of innovation at Google Cloud has been excellent. There are continued updates to BigQuery, Cloud Composer, Dataflow, Vertex AI, enabling us to accelerate data-driven decision-making across our organization, deliver faster, more personalized customer solutions, and harness the transformative potential of gen AI.
Stephen Lee
Head of Enterprise Data Engineering, TELUS
Headquartered in Canada, TELUS is a dynamic global communications technology company with more than 100,000 employees serving multiple industries through its TELUS Digital, TELUS Health, TELUS Agriculture and Consumer Goods, and TELUS Communications businesses.
At the end of 2021, TELUS recognized the need to overhaul and modernize its legacy on-prem data stack (such as Hadoop and Exadata) running on legacy on-prem infrastructure.
At that time, there was no standard method governing how teams built data pipelines and used datasets. With existing on-prem infrastructure approaching end-of-life and the availability of new AI/ML tools, TELUS knew it was time to reimagine its data strategy.
It sought to create a repeatable pattern of data ingestion from multiple source systems (Common Ingestion Framework) that expedite modernization on Google Cloud while eliminating the dependency on legacy platforms and vendors.
TELUS hoped the enhanced pipeline reliability and simplified data engineering support model, would enable faster time to market and drive real business outcomes.
This work also centered around TELUS' ideal Enterprise Datahub that would ultimately support some of the following common persona-specific use cases:
- Marketing teams: Experimenting with curated customer datasets for targeted marketing campaigns and personalized customer experiences.
- Finance teams: Leveraging revenue, billing, and usage data insights to create dashboards for official financial reporting (RGU and MRC).
- Data scientists and ML engineers: Leveraging high-quality datasets to create, train, and deploy ML models in Vertex AI for various use cases such as Next Best Action and propensity models.
TELUS' Enterprise Data Engineering team's goal was to modernize the legacy data stack on BigQuery to create a consolidated, cloud-based, self-serve analytics platform in the Enterprise Datahub that would empower its employees to innovate, supercharge data-driven decision-making, and unlock the enormous opportunities presented by gen AI.
TELUS chose Google Cloud as the foundation for its revamped data architecture based on the environment's reliability, scalability, and proven success supporting a data warehouse in the cloud. Solution elements include BigQuery and supporting services such as Dataflow, Cloud Composer, Cloud SQL, and Vertex AI. Moreover, the move to Google Cloud would eliminate hardware and infrastructure maintenance demands on TELUS.
TELUS collaborated with Google Cloud and partner Onix to complete the complex, multi-year project, which included restructuring all 200+ enterprise-grade pipelines and migrating 14 petabytes of data. Onix is a trusted consultancy in leading large-scale modernization on Google Cloud. It offers valuable IP through an automated code conversion tool, Raven, and a data validation tool, Pelican.
This partnership between Google Cloud, Onix, and TELUS' Enterprise Data Engineering team created a cloud-fueled, best-in-class, self-serve analytics platform that would empower the TELUS team to expedite their digital transformation and optimize the way they do business.
Delivering modern data experiences to all end-users
Today, TELUS uses BigQuery as its Cloud Data Warehouse and Enterprise Datahub, Cloud Composer for pipeline orchestration, and Dataflow as the core pipeline engine that moves and transforms the data from Google Cloud Storage into BigQuery – a critical element of the restructuring process when the company moved from on-prem to cloud.
These critical Google Cloud services make up the new Enterprise Datahub, which is now the single, unified source of truth for all enterprise grade data. The Enterprise Datahub has a robust self-serve architecture used by technical and non-technical team members, including data scientists, analysts, engineers, citizen data scientists, BI teams, and marketing teams, for data experimentation and to productionalize data at scale.
The user can get set-up with an individual Google Cloud console workbench that is graphics-based, ephemeral, and acts as a development environment for data experimentation. Once the user is ready to productionalize a process, they will create a BI layer project to test, schedule, and deploy the jobs.
In partnership with Google Cloud and Onix, we’ve created a self-serve, cloud powered, data analytics platform that serves as our single source of truth for all enterprise-grade data and supports our analytics and operational use cases. The timing worked out beautifully for us, and we were ready and prepared to capitalize on the enormous opportunities of gen AI.
Stephen Lee
Head of Enterprise Data Engineering, TELUS
A common use case includes creating a daily job to obtain the latest revenue and usage data to create a financial dashboard using sanctioned data visualization tools.
As part of the modernization to BigQuery, Onix’s Raven tool offered TELUS the flexibility to approach the data transformation in a way that aligned with its unique vision. Rather than a complete lift-and-shift modernization, TELUS was able to restructure and future-proof its data pipelines with code conversion automation support from Raven. Pelican by Onix offered the necessary data validation to make sure new pipelines ran on Google Cloud.
Onix's tools and continuous support, including the round-the-clock coverage for analysis, development, testing, and post-production, was critical to the modernization's success. Onix also provided the training and knowledge transfer to ensure TELUS could manage its own pipelines through strategic meetings, documentation, and transferring ownership of completed work.
TELUS has worked with AI and ML for over a decade. Its recent data modernization efforts future-proofed those capabilities that provided team members with easier access to unified, reliable data and real-time business insights. Most notably, this improves how TELUS leverages data for tangible business outcomes, fostering innovation and positioning TELUS at the forefront of enriched data decision-making.
TELUS completed the massive overhaul of hundreds of enterprise-grade data pipelines and 14 PBs of data in only 2.5 years, meeting its ambitious goals and laying the foundation to support more advanced gen AI solutions that require elevated, consistent, and centralized data management.
Looking forward, TELUS will continue to work on low-code and no-code solutions to further open its Enterprise Data Hub and gen AI tools to its global team.
TELUS is a dynamic global communications technology company that serves multiple industries through its TELUS Digital, TELUS Health, TELUS Agriculture and Consumer Goods, and TELUS Communications businesses.
Industry: Technology
Location: Canada
Products: BigQuery, Cloud Composer, Dataflow, BigQuery Analytics Hub, BigQuery Omni, Cloud SQL, Cloud Storage, Looker, Vertex AI, Assured Workloads
About Google Cloud partner- Onix
Onix is a trusted cloud consulting company and leading Google Cloud partner that helps companies get the most out of their technology with innovative, cloud-powered solutions and services.
