FIS analyzes market events and disruptors with Google Cloud Platform

As a global leader in financial services technology, FIS runs U.S. market analysis using Google Cloud Platform. FIS’ Market Reconstruction Platform can collect, link and store data on every equities and options trade lifecycle event and then produce feedback reports within a few hours. Its high-performance system can adjust to fluctuating market activity and support complex analytics.

Understanding market disruptors

FIS has built a market reconstruction tool that could help determine the potential cause for events that disrupt the securities market, such as the “flash crash” of 2010. FIS’ system not only can store billions of transactions, but also allow compliance and risk executives to conduct surveillance and on-demand querying, including market reconstruction.

The system can process and link up to 15 terabytes of data daily in four hours and can store it for six years, as mandated by law. “That amounts to about 30 petabytes of data,” says Neil Palmer, chief technology officer of Advanced Technology business at FIS. “There’s not much out there on that scale, and certainly not in financial services. It’s a huge undertaking.”

“Battle-tested way beyond what I need”

Palmer’s team needed a platform with high computing power, but also wanted to avoid the cost and maintenance associated with building and operating an on-premise system. After evaluating multiple cloud options, they were drawn to Google’s pioneering work with large-scale data processing and analysis.

“As we researched further it became clear that Google’s entire business is based on ingesting data and extracting value on this sort of scale,” Palmer says. “It’s a significant comfort that what I’m using is battle-tested way beyond what I need.”

Traditional technology stacks and data centers would have issues with data volumes this large since it requires a lot of horsepower. Since the system must link events in a short amount of time, including many hops and nodes, FIS needed a technology that can handle flexible schemas, time series data and huge amounts of scale. The solution was Google Cloud Bigtable.

The FIS team can filter on Cloud Bigtable columns and construct them on the fly, giving them a taxonomy by which they can link events. Cloud Bigtable also gives FIS the ability to scale the platform up or down based on trading volume, thus driving significant cost efficiencies.

“Scalability is a huge benefit of using Google Cloud Platform,” Palmer says. “A traditional fixed-hardware IT solution in this scenario could result in millions of dollars of equipment sitting idle on many trading days.”

FIS also relies on Google Cloud Dataflow to quickly process, format and validate incoming data before sending it to Google Cloud BigQuery for analysis. Cloud Dataflow provides FIS with a managed services environment that supports batch and stream data processing, which allows the FIS team to focus on data processing tasks, instead of cluster management.

Six weeks to prototype

Built in six weeks by a team of six engineers, FIS’s system performed impressively, processing about 25 billion simulated market events in 50 minutes. The system sustained 22 million event reads per second and 16 million event writes per second.

“If you had told me a year ago that we would accomplish this in six weeks, I would have thought you were joking,” he says. “What we’ve been doing thus far would not be possible without Google Cloud Platform.”

FIS’ architecture and Google Cloud Platform technology may bring significant possibilities for reducing operational costs and time to market while increasing scalability at the same time.