DevOps Awards Winner VMO2 on how to achieve continuous improvement with DORA

In this blog post, we’re highlighting VMO2 for the DevOps achievements that earned them the ‘Continuous improvement with DORA’ award in the 2022 DevOps Awards. Launched in June 2021, Virgin Media O2 (VMO2) combined the UK’s largest, most reliable mobile network with a broadband network that offered the fastest widely-available broadband speeds. Since then, they’ve accrued 47.7 million UK connections across broadband, mobile, TV, and home phone, all in the name of their mission of upgrading the UK. To learn more about the 2022 DevOps Awards winners and how they used DORA metrics and practices to grow their businesses, start here.

As a previous DORA DevOps Award winner, we deeply understand how DevOps creates a consistent, productive work environment. Since receiving the award we’ve spread what we’ve learned to teams across the company, and have even started giving presentations at conferences and written blog posts showcasing our DevOps evolution to a wider audience.

Even with all of this transformation, we realized there was always more to learn and grow, especially with the complications of a changing telco landscape. Not only did we still have highly coupled data dependencies on legacy infrastructure and multiple, overlapping processes that required heavy outsourcing, we still had issues born from merging two huge telco operators. After reviewing our systems, we realized that we were running servers and clusters from most of the major providers.

Objectives

To overcome all of this technical debt, the company realized that we needed to migrate the most critical of these source systems to Google Cloud on the VM side while also improving the underlying tech and culture. Specifically, our goals were to:

• Remove complexity from the overall organization

• Provide personalization and support to customers

• Create future-ready infrastructure in the most cost-effective way

Solution

To accomplish this, we partnered closely with Google Cloud to create a strategy that centers around three key pillars: DevOps, MLOps, and Analytics Engineering.

For our DevOps decentralization effort, we shifted from a centralized model where an individual team was responsible for all building efforts, to a decentralized one, with 28 teams building together daily, including over 200+ engineers and analysts. Each of these teams have their own data products with mostly independent code bases or data models, but all of them share common libraries, best practices, and principles. And as these teams discover new best practices, they are shared across teams, further driving a culture of growth and collaboration.

Since the beginning of our MLOps journey in early 2022, we now have over 50 data scientists and engineers, with company-wide adoption of Vertex AI pipeline-based in-house platform, leading to reduced time-to-production and increased scalability.

• Time-to-market: By using CDC-tracked raw data with BigQuery and our adopted dbt (a data modeling tool) cloud, an 8-person team helps 160 analysts use self-service tools to make more informed moves without waiting for data.

• Data ingestion: To streamline ingestion, we instituted a Vortex platform to ingest multiple forms of third-party data for batch and streaming, while we collaborate with other teams to ingest internal data sources using Ab Initio and Cloud Composer.

• Feature rollout: With a repository of ML templates and MLOps best practices and a shared MLOps Python package, our teams have scalable solutions to deploy quickly and securely following CI/CD practices.

The Analytics Engineering team uses dbt to continue their mission of ensuring that data is trusted and accessible so that teams can develop at pace in a controlled environment to get more value from data. Thanks to this and our other efforts, the teams production area processes 250TB daily with 60k queries run against created data models and contain 91k lines of dbt code.

Results

This commitment to continuous improvement has yielded tangible results in terms of value creation through cost reduction, revenue generation, and customer satisfaction. In the short term, this includes:

• Driving digital store conversions (CVR) of 458% as well as tripling the average revenue uplift against the legacy through commercial and recommendation engines, and an additional 6% CVR increase from recommending relevant products

• Saving network engineers 3000 days of working time, which comes out to £800k cost savings

• Creating 50% of the third-party incumbent value for 10% of the annual spend through our internal call routing

In addition to these immediate returns, we have also seen medium-to-long-term returns that will help unlock the company's future growth, including:

• Increasing first-time resolution in the channel that the customer chooses through omnichannel, data product-driven solutions, and putting their digital channel share on the road to reaching 80% by 2026

• Helping the company make better decisions when it comes to investing in the network — part of a £4B program to upgrade the UK

• Bringing mobile and fixed services together in converged experiences for Virgin Media O2 customers, saving millions of pounds through decommissioning, work efficiencies, and shared practices and techniques

Stay tuned for the rest of the series highlighting the 2022 DevOps Award Winners and read the 2022 State of DevOps report to dive deeper into the DORA research.