How Citadel Securities is reimagining quantitative research on the cloud

Citadel Securities is one of the largest market makers in the world, executing more than $400 billion in trades a day. The company builds and deploys powerful, industry-leading quantitative models that price hundreds of thousands of securities across markets and geographies. The work required to build these models is known as quantitative research.

Citadel Securities partnered with Google Cloud to build the next-generation of its quantitative research technology platform and increase research productivity while improving price-performance ratio.

Citadel Securities’ approach to building the platform

Citadel Securities employs hundreds of quantitative researchers who scour vast volumes of market data to build models that accurately identify the price of thousands of financial securities. Researchers must conduct tests to validate how these models perform before they can go live. For example, a model could be tested against past market events or a variety of simulated market conditions. These simulations are complex and require thousands of processors across thousands of servers working in concert.

There are two key goals when building a successful quantitative research platform capable of supporting this cutting-edge work:

• Maximize productivity by reducing the time it takes for a quantitative researcher to run a test or experiment
• Optimize price-performance ratio by increasing the number of simulations run per dollar spent on compute and storage

Citadel Securities initially built an on-premises data center and used high-performance hardware custom-built for running complex and data heavy simulations. As its research workloads and team grew, and as it pursued increasingly complex techniques, the firm required a platform that would scale accordingly. A key challenge was the fact that the timing and volumes of research (and therefore the required number of cores) could fluctuate greatly.

Citadel Securities’ options were to build its own data center to accommodate for the top of the demand peak, therefore minimizing the research turnaround time, or build for below the peak to achieve better price-performance ratios, risking bottlenecks and slower turnaround times.

At this point — in 2017 — Citadel Securities started working with Google Cloud. The operating question was not “How do we replicate what we do on-prem in the cloud?” but rather, “What can we build that would provide us with the most productive, scalable and cost-efficient research platform?”

Citadel Securities was no longer thinking about IT in terms of how many servers to buy, but instead asking questions like: How many tests do we want to run per day? What types of cores work best for different tests? How quickly should a researcher get a result on a test?

By making the tech platform more elastic, the researcher could set the requirements of the platform. The technology would no longer constrain the research.

Architecting the solution on Google Cloud

Citadel Securities needed to provide a platform to its researchers that would deliver practically unlimited scale, both vertically and horizontally, while also maximizing resource and cost efficiency. By working with Google Cloud’s engineering team, Citadel Securities unlocked scalable infrastructure and built specialized tools, such as GUI interfacess for job status, and the ability to profile performance and monitor efficiency.

Google Cloud's infrastructure eliminates the overhead of managing in-house data centers by providing demand-based scaling, hardware management and patching, with security built-in. Google Cloud’s diverse storage portfolio cost-effectively supports the spectrum of Citadel Securities’ data needs by analyzing access patterns and automatically transitioning the data through different storage classes, while also solving for high-throughput and low-latency requirements.

Citadel Securities leverages a strategic mix of on-demand and preemptible hardware for massive parallel compute tasks and premium hardware in co-located racks for high-performance clusters that require low-latency connectivity. This approach optimizes costs and performance across diverse workloads.

Large quantities of data flow into the cloud, thanks to dedicated link interconnectivity with Citadel Securities’ data centers. Centralized data lakes with granular controls empower collaboration across teams, driving broader, strategy-spanning insights

Researchers optimize their productivity by selecting the ideal hardware (whether GPU-accelerated, compute-optimized, or memory-optimized) for their specific workloads. With near-instant scalability, researchers can get to work quickly, without their project waiting in a queue. Advanced networking (200 Gbps) and co-located virtual machines ensure maximum performance, while providing all the insight metrics that the researchers require.

Robust security, including encryption and posture management, helps the firm meet stringent compliance requirements. "Security by default" streamlines the process, ensuring research platforms are protected from the outset. As a result of moving these workloads to Google Cloud, the Citadel Securities Research Platform Technology team can now focus on delivering differentiated capabilities that drive bigger, better, and faster research — not maintaining physical server farms.

Realizing the benefits

The end result was an unlock for Citadel Securities’ business. Now, the firm can spin up more than 1 million cores concurrently, enabling hundreds of quantitative researchers to run jobs in parallel. This means researchers’ jobs aren’t sitting in a queue, and everything runs as it is submitted, creating faster time-to-insights and increasing productivity. Single workloads that once took many hours now take seconds to complete.

In addition, Citadel Securities’ costs — measured in costs-per-research-hour — have dropped precipitously. The ability to combine different consumption models and to scale up and down on both compute and storage has massively increased the firm’s resource utilization (including core and memory utilization), slashing operational overhead and increased performance per dollar.

The firm’s platform — and therefore its researchers — can better stay on the cutting edge of technology, and the platform’s ability to adapt to new hardware requirements has also shifted drastically.

Google Cloud’s infrastructure platform has not only solved Citadel Securities’ cost-versus-performance quandary, it’s also brought forth new possibilities for research strategies that the firm never would have achieved before.

As Citadel Securities continues to push the frontiers of technology and markets, it now has the flexible research platform necessary to support its long-term growth.