10 months to innovation: Definity's leap to data agility with BigQuery and Vertex AI

At Definity, a leading Canadian P&C insurer with a history spanning over 150 years, we have a long tradition of innovating to help our customers and communities adapt and thrive. To stay ahead in our rapidly evolving industry, we knew a unified data foundation was key to realizing the business and customer experience opportunities offered by modern analytics and AI. 

While our legacy on-premises Cloudera platform had served us well, it could no longer support our growing needs for scale, innovation, and harnessing the power of data and AI. So, we embarked on a critical mission: modernizing our data infrastructure.

Legacy limitations stifling innovation

We faced a combination of interconnected challenges, which impact many organizations today:

• Limited scalability and AI/ML workload support: Our existing infrastructure, constantly running at 80% utilization, was stretched thin. Processing billions of daily events for real-time analytics and scaling AI and ML workflows was a constant battle, limiting our ability to gain timely insights and develop innovative, data-driven products and experiences.

• Data silos, fragmented insights: Our data resided in various systems, creating a fragmented view of our business. This made it difficult to get a holistic understanding of our customers and hindered initiatives like building a comprehensive customer 360º view and delivering personalized recommendations at a moment of relevance.

• Escalating costs: Maintaining and scaling our Cloudera platform, which hosted massive data volumes (200TB compressed, 1PB uncompressed), was increasingly expensive and diverting valuable fiscal and people resources away from strategic priorities.

Faced with these pressing issues, the timing of our next renewal presented a strategic window of opportunity. We had a critical decision to make — migrate both technology and business platforms within 10 months or invest in upgrading our legacy Cloudera environment.

Building a unified data and AI platform with Google Cloud

We chose Google Cloud and its powerful duo, BigQuery and Vertex AI, to build the Strategic Data Platform (SDP) — our new modern data analytics platform. BigQuery's serverless architecture, unmatched scalability, built-in ML capabilities, and seamless integration with Vertex AI made it the ideal solution to power our data-driven transformation.Our migration was a remarkably fast-paced effort, carried out in close collaboration with with Google Cloud and Quantiphi, a Google Cloud partner. 

Like many enterprises, we adopted a hybrid approach. We retained Databricks on Google Cloud for specific ETL workloads, utilizing Quantiphi's expertise in converting legacy systems. At the same time, we migrated the bulk of our data processing to BigQuery for optimal cost-efficiency and performance. We also used Cloud Composer to orchestrate our complex data pipelines and ensure secure, private connectivity within our Google Cloud environment, a crucial requirement for handling sensitive customer data.

As a result, our dedicated team of over 100 Definity employees completed the migration in just ten months — 50% faster than the industry average. This rapid transition was aided by innovative tools, such as the "nifi-migration" solution built by Google Cloud Consulting. This open-source tool provided a visual and highly configurable way to automate real-time data flow between different systems, minimizing disruption and helping us surpass our initial migration timeline expectations.

Our CTO, Tatjana Lalkovic, who championed this effort to consolidate our structured and multimodal data to accelerate our AI/ML use cases, shared her perspective on the impact of our decision, saying:

“As we reimagined where data and AI could take our business, industry, and customer experience, Google Cloud BigQuery and Vertex AI stood out as modern, enterprise-ready serverless solutions prepared to meet the AI moment — not just today but for the foreseen future. The speed and success of this migration has created a lot of trust in our partnership and has been a significant boost to our digital transformation to streamline operations, improve products, and better serve customers.”

Strategic Data Platform - High level design on Google Cloud

Transforming insurance with data and AI

The results of our migration to BigQuery and Vertex AI has been transformative for Definity. We’ve seen exceptional user satisfaction, with the SDP achieving a remarkable Net Promoter Score (NPS) of 9.9 out of 10. The move has also saved us millions on our annual spend on non-strategic technologies and delivered a roughly 75% reduction in planned downtime for our digital platforms. 

Performance has also dramatically improved, with processes to gain insights that once took days now completing in an average of 4.5 hours. Moreover, migrating to Google Cloud has helped us increase agility and innovation. We’re now able to double our business releases per year — achieving a 30% increase in testing automation, a 63% improvement in deployment time, and a 10x faster infrastructure setup. 

By combining structured and unstructured data in BigQuery, we’ve unlocked new analytical possibilities and improved price-performance. This unified data foundation has empowered our business intelligence tools with richer, more comprehensive data, leading to more informed business decisions. The seamless integration with Vertex AI has enabled us to develop, deploy, and scale AI models, driving innovation in areas like fraud detection, automated intake, and personalized call center experiences. At the same time, we benefit from Google Cloud’s strong commitment to data security and privacy, helping us to strengthen our security posture and keep our customers safe.

As our VP of Data, Ron Mills, said:

"BigQuery's serverless architecture has been a game-changer. The 'nothing to manage' approach is a huge differentiator. For enterprises like us that are migrating from on-prem clusters constantly running at 80% capacity, it's like night and day."

Lessons learned from our migration journey

Migrating a core data platform is a significant undertaking, and we’ve learned a lot along the way. For other organizations considering the same journey, here are some key takeaways from our experience:

• One team, one goal: Foster a collaborative environment where technology and business teams, vendors, contractors, and consultants work together seamlessly towards a shared objective.

• Leadership trust and commitment: Executive leadership trust in the delivery team's decision-making is crucial for maintaining momentum and navigating challenges.

• Be bold: Don't be afraid to think outside the box, make timely decisions, and be prepared to adapt quickly to unforeseen setbacks.

• Plan for the unknown: Anticipate potential roadblocks and have a core team dedicated to developing alternative solutions and addressing unforeseen issues.

• Strong business partnership: A trusted relationship with business teams is essential for smooth user acceptance testing, change management, and avoiding unnecessary disruptions during the migration.

• Balanced governance: Independent governance should provide guidance and support calculated risk-taking, acting as a partner in problem-solving rather than a blocker.

• Motivated team: Cultivate a team-oriented environment where ownership of the project extends beyond leadership to every team member.

• Transparent communication: Maintain open and consistent communication among all stakeholders (in our case, over 250 people) to ensure everyone is aligned and informed.

• Fast fail and incremental delivery: Avoid a "big bang" approach. Embrace incremental releases (we aimed for 2-5 daily releases) to learn quickly, adapt, and iterate.

• Parallel run: Plan for a parallel run of your systems on both the legacy and target cloud platforms to ensure a smooth transition and validate the new environment.

A data-driven future with limitless potential

Our migration to BigQuery and Vertex AI is just the first step in Definity’s data transformation journey. With a modern, scalable, and AI-ready data foundation now in place, we are empowered to unlock even greater value from our data and continue to lead innovation in the insurance industry. We are excited about the possibilities that lie ahead and are already actively developing our next AI use cases, including several focused on legal summarization and IT functions. We are confident that our partnership with Google Cloud will be instrumental in helping us achieve our goals.

Learn how to start your own migration journey to BigQuery today.