Telecom providers are leading the way on automating data. Here's how others can follow

In the coming age of automation, communications giants across the world are showing us the way.

After spending years migrating from on-premises environments to the cloud, and then wrangling the biggest of data out of silos and into data lakes and data oceans, the communication service providers and telecom companies (colloquially known as CSPs) are turning their tremendous potential for innovation towards automation and artificial intelligence. These organizations are using cutting edge tools to enhance their network performance, understand and solve customers problems in real time, and manage data at scale.

As every organization in every industry has been wrestling with more and more data, one of the best places to look for guidance and inspiration are these CSPs. They are the vehicles for virtually every conversation, message, transaction, and digital process zipping around the world these days. In effect, the data they handle is everyone else’s as well as their own — the uber-data, if you will. With such exponential loads bearing down on their systems, and their bottom lines, they have to find better and faster ways to direct it all at every turn.

Indeed, through the process of building robust cloud environments and taking advantage of the speed of innovation that the cloud enables, CSPs have built cultures focused on the organization and utilization of data. And they’ve realized that, once data is well organized and accessible, it allows for a variety of extraordinary use cases that were once distant objectives on a project wishlist.

“To be able to do smart automation, you have to have an understanding of what it is you’re re automating,” Ben Clinch, principal enterprise architect for data governance at BT, said at Google Cloud Next ‘22.  "It's really important that an organization develops a data management and data governance culture and is organizing the way that it ascribes its data and its intent around data in a way that can be automated.”

In the case of BT, Clinch explained that the company has organized its data into a formal ontology — like a knowledge graph — that allows it to be machine readable across the organization. The data can then be semantically discovered and recorded in a central repository, so that teams can then understand the nature of the data and how it should be treated in order to fulfill the organization's aims. 

Essentially BT has created a knowledge graph of its data that helps it automate insights and fulfill business objectives.

Like BT, Vodafone has built a massive, globe-spanning data operation across two dozen countries, taking on all the challenges of scale and localization that entails. In the last couple of years, the carrier has worked hard to bring AI and automation to its data science operations, resulting in a process it calls AI Booster that helps spin up new machine learning instances quickly and at minimal cost.

“You’re going from a lot of manual processes and multiple teams, to a completely automated environment — the cost aspect is pretty obvious,” Sebastian Mathalikunnel, the AI strategy lead at Vodafone, said at Next '22. “If you think about it, that’s only going to be financially viable if the cost does not  scale linearly with the number of instances, or with the amount of infrastructure that is spun up.”

“The cost of AI Booster,” he added, “is almost 100% traceable to the exact compute that was run against that infrastructure.” 

Data automation at scale

Vodafone has one of the largest data footprints in the world. It’s so big that they don’t call its unified repository a data lake, but a data ocean. Before it migrated to Google Cloud, Vodafone had 17 petabytes of data on 600 physical servers with 1,300 data pipelines. After migrating to the cloud and organizing its data into the ocean, it will have more than 11,000 data pipelines, a growing team of data scientists, and a solid plan on how to handle and utilize all that data.

According to Dr. Cengiz Ucbenli, head of Vodafone’s big data and AI group, the company has three main pillars when it comes to data and artificial intelligence innovation: real-time AI, AI at scale, and AI everywhere. For a company floating in an ocean of data, that kind of automation is ambitious.

“We need to understand our customers’ pain points,” Cengiz said in a recent Q&A with Google Cloud. “We need to make decisions in a timely manner when we are extending our network, introducing new services and offering new price plans. Instead of making predictions on a monthly basis, we are now able to do it on a real-time basis.” 

One of the dangers that a company like Vodafone faces is the issue of data sprawl. Too much data, too spread out, with too much to parse and understand. Vodafone solved this problem by building a taxonomy within its data ocean, separating out regional and localized data, financial data, operational data, and so forth. This greatly improves tracking, management, and analysis of the vast streams Vodafone ingests on a minute-to-minute and even second-to-second basis. 

“Vodafone is pretty mature in its data science journey,” Mathalikunnel said. “Looking back it was this exact problem of size and scale of Vodafone data science operations that led us to believe that we could have a problem on our hands.”

The next step after organization and taxonomy was to automate the handling of the data in an efficient manner, and this is where AI Booster comes in. This scalable, unified ML platform built entirely on Google Cloud was created with automation, scalability, and security at its core. AI Booster allows Vodafone to build machine learning models in a matter of weeks, instead of months with its previous manual processes.

“If you have the use case you want to scale up in one of these environments, and want it to scale across markets, it used to take a lot of time and effort to replicate the same thing across markets because each environment was different,” Mathalikunnel said. “We call this a horizontal scaling problem. And complimentary to that was the vertical scaling problem, which is simply how do we take a data science process from proof-of-concept to production in the quickest possible manner.”

Vodafone uses AI Booster for a variety of uses, including understanding customer sentiment, recommendations, and loyalty programs. 

“Net Promoter Score was the first use case that data scientists trusted us to take on board into AI Booster,” Mathalikunnel said. “What we’re trying to do with NPS is get to know or measure the happiness of our customers with our products. Where are we doing a good job, where can we improve? As you can imagine, it’s a pretty important use case for us to have.”

Automation for governance and compliance

For its part, BT found that anything above a petabyte in data needs to take advantage of smart automation, otherwise it becomes too cumbersome and expensive to handle it manually, especially when managing for risk, governance, and compliance.

“The data management itself is not optional,” Clinch said. “We're thinking in terms of semantic meaning of data organized around an information model that is often a formal ontology, like a knowledge graph. That allows it to be machine readable across an organization. It also allows active metadata to be able to semantically discover the data and record that centrally, so that you can then understand the nature of the data and how it should be treated to be able to fulfill the organization's aims.” 

BT’s data architecture aligns closely with Google Cloud’s Dataplex, a suite of tools that enables organizations to centrally discover, manage, monitor, and govern their data across data lakes, data warehouses, and data marts with consistent controls. 

By organizing within Dataplex, organizations can make their data more discoverable and thus can better understand their levels of risk and compliance — concerns that are an increasing focus for every industry, whether banking or healthcare, retail or manufacturing.

“We have a saying at BT about winning with tech,” Clinch said. “When you're going along that journey of maturity of data capabilities, you need to be thinking in terms of: How are we going to be able to implement, embed, and automate it — and then design it from the ground up in a way that allows you to be able to scale.”

As more organizations look to scale their data operations, their automation efforts will have to grow in kind. As these CSPs have shown, it's about the only way to keep up with our growing sea of data, and arguably the best way to capitalize on it. 

More from Google Cloud's telecom leaders: