Databases

Symphony handles millions of messages with Bigtable, saving 40% on database costs & increasing resiliency

February 26, 2024

Olivier Richaud

VP Platforms and SRE, Symphony

Try Gemini 1.5 models

Google's most advanced multimodal models in Vertex AI

Editor's note: Symphony is on a mission to revolutionize the way financial markets and trading teams connect and collaborate. To achieve that goal, they needed a managed database that could handle large amounts of data, while also providing the scalability, reliability, and flexibility required to support its growing user base. Bigtable fit the bill, and after a smooth migration, Symphony’s core technology teams can stay focused on development and innovation.

As a pioneer in secure communication and collaboration, Symphony sits at the heart of information exchange for financial institutions worldwide. Millions of messages traverse our platform daily, playing a mission-critical role in facilitating seamless communication and collaboration across financial institutions — all without compromising on the lightning-fast performance that keeps markets moving.

The resilience, scalability, and durability of our platform is critical to ensuring that financial institutions can rely on us as the trusted foundation for their secure communication needs. Our platform’s robust infrastructure empowers these institutions to operate with confidence and agility in today's dynamic markets.

Embracing managed services to harmonize database management

Prior to migrating to Bigtable, Google Cloud’s NoSQL database service, we used HBase to power our messaging and collaboration platform. Managing this open-source, non-relational database was time-consuming and expensive, requiring dedicated resources to handle operations and maintenance tasks. The site reliability engineering (SRE) team found themselves operating numerous HBase clusters, some of which required specific maintenance windows not only for the HBase software itself, but also for the underlying infrastructure.

Our main goal was to transition from HBase to a managed service solution that could meet our performance needs while also being cost-effective, resilient, scalable, and seamlessly integrated with our existing infrastructure. Ultimately, we aimed to maintain a global, worldwide 24/7 team to run and operate key data stores, keeping the team knowledgeable and up to date while delivering in-depth security across the board. This is what led us to Bigtable.

Bigtable takes center stage in Symphony's platform

Bigtable emerged as our choice for a managed database for many reasons:

HBase compatibility: The Bigtable client is compatible with the HBase client, facilitating successful adaptations in our Java backends.
Data durability: Bigtable scales linearly while maintaining performance, relying on core Google infrastructure and storage for reliability.
High availability with fully managed replication: Fully managed automatic replication between Bigtable clusters enhances resilience.
Developer-friendly with observability: Bigtable integrates with our monitoring stack, allowing fallback to other standby clusters if needed.
Seamless integration with other Google Cloud services: Bigtable seamlessly integrates with Google Cloud services, including Dataflow, which simplifies the development process for efficient data processing.
A fully managed service that just works: Bigtable strikes the perfect balance of performance, scalability, durability, and ease of use. It also simplifies DevOps deployment, promoting agility and clear separation of duties between developers and cloud infrastructure.

Orchestrating a successful migration

Prior to our planned downtime, we captured snapshots of our HBase data stored in Cloud Storage. Then, leveraging Dataproc, we exported the data to a larger Hadoop cluster that connected to our HBase cluster.

We used Dataflow jobs to import the data from Cloud Storage to Bigtable, performing essential optimizations along the way. To maintain business continuity, we injected the data into the replicated Bigtable cluster, helping to ensure zero impact on our production systems. Within minutes, our data seamlessly and fully replicated into the production cluster.

We achieved a successful migration by harnessing the combined power of Dataproc and Dataflow. With careful planning and coordination, we seamlessly moved our HBase data to Bigtable, ensuring minimal downtime for customers.

Storing millions of messages at 40% reduced operational costs

Bigtable has significantly improved Symphony’s operational efficiency, offering enhanced SLAs, improved performance, and better cost control. This allows our teams to focus on automation, scalability, and other critical tasks rather than maintaining and upgrading core infrastructure services. Automated patching and seamless integration support a higher level of security and access control.

Here are the major achievements we've seen to date with our migration to Bigtable:

We reduced database operational costs by a remarkable 40%, accompanied by a significant decrease in administration and maintenance activities. These tasks were known to disrupt workflow as they often required unplanned emergency actions.
Bigtable exhibits exceptional stability in terms of response times and latency even when scaled out, which is crucial for delivering a reliable service. Accessing Symphony messages of varying sizes is accomplished in 15ms (p95) or less.

Symphony has made great progress in migrating core services to Google Cloud. Over 400 customers — including Symphony flagship customers — have successfully transitioned to the Google Cloud platforms in the US, EMEA, and APAC regions. Moving forward, this Bigtable migration will continue to unlock new possibilities. Storing massive amounts of data at scale becomes possible without incurring performance penalties. Global replication offers new opportunities for scalability and resilience. With Dataflow and Dataproc, analytics consolidation becomes easier while maintaining security and confidentiality.

In addition, we are actively exploring opportunities to implement AI, along with the capabilities of Confidential Space to boost confidentiality and security. Bigtable will continue to serve as the central store for our core messaging data, providing an excellent one-stop shop to access and process the vast amounts of data we handle.