Financial services firm processes 25 billion stock market events per hour with Google Cloud Bigtable

FIS, a global financial technology and services firm and frequent leader of the FinTech Top 100 list, recently ran a load test of their system on Google Cloud Platform to process, validate and link U.S. stock exchange market events. FIS used Google Cloud Dataflow and Google Cloud Bigtable to process 25 billion simulated market events in 50 minutes, generating some impressive statistics in the process.

Cloud Bigtable achieved read rates in excess of 34 million events per second and 22 million event writes per second using 3500 Cloud Bigtable server nodes and 300 n1-standard-32 VMs with Cloud Dataflow. Additionally, Cloud Bigtable provided sustained rates of over 22 million event reads per second and 16 million event writes per second for extended periods of time.

Moreover, Cloud Bigtable was also able to achieve significant I/O bandwidth rates during the load test: read bandwidth peaked at 34 GB/s while write bandwidth peaked at 18 GB/s. Cloud Bigtable sustained significant bandwidth for input and output for 30 minutes as well: 22 GB/s for reads and 13 GB/s for writes.

For FIS, these performance capabilities make it possible to process an entire day’s worth of U.S. equities and options data and make it available for analysis within four hours.

For the complete set of benchmark results, see these slides. You can see a more detailed description of the overall system architecture presented by Neil Palmer and Todd Ricker from FIS and Carter Page, engineering manager for Google Cloud Bigtable:

We look forward to working with other innovative companies like FIS to help them address data processing challenges with the performance, scalability and NoOps approach that Cloud Bigtable provides.