Kaiko: Choosing Cloud Bigtable for a managed data storage solution that’s on the money
About Kaiko
Kaiko is a digital assets data provider that covers all major cryptocurrency exchanges. Founded in 2014 and based in Paris, the Kaiko team combines years of expertise in data science, finance, and distributed systems to provide its clients with access to high-quality and transparent cryptocurrency data.
Tell us your challenge. We're here to help.
Contact usKaiko built a scalable infrastructure using Google Cloud managed services including Cloud Bigtable, harnessing autoscaling on Google Kubernetes Engine to easily handle its data growth.
Google Cloud results
- Supports the storage and delivery of 50 million daily transactions with Cloud Bigtable and Cloud Memorystore
- Provides a global network of data centers that helps keep latency low
- Simplifies systems operations thanks to autoscaling on Google Kubernetes Engine
Enables market data delivery to clients in under 150 ms
From Bitcoin to ZCash, the cryptocurrency market is becoming increasingly attractive to institutional investors. As a result, there’s a growing demand for transparent and high-quality cryptocurrency data to provide the insights necessary for quantitative funds, financial institutions, and regulators to make informed trading decisions. Paris-based digital assets data provider Kaiko collects, normalizes, and distributes market data from more than 10,000 currency pairs across 70 exchanges, providing access to both historical and real-time data according to its clients’ needs.
“We work with financial services, so it’s really important that our data is reliable, and that our platform is always available. It’s the quality of our service that attracts our clients to switch to Kaiko from other providers, so our system needs to be robust, redundant, and scalable.”
—Michael Inthilith, Cloud Engineer, Kaiko“Crypto markets are uniquely fast-moving and complex,” explains Michael Inthilith, Cloud Engineer at Kaiko. “Historically, it’s been a real challenge to integrate dozens of independent and disparate data exchanges and build highly available and low-latency systems.” Kaiko currently processes 50 million transactions a day.
To offer a reliable service to its customers and support with its expanding business, Kaiko needed a managed database service that could handle petabytes of data and scale easily. Choosing Cloud Bigtable on Google Cloud has been the perfect fit.
“We work with financial services, so it’s really important that our data is reliable and that our platform is always available,” says Michael. “It’s the quality of our service that attracts our clients to switch to Kaiko from other providers, so our system needs to be robust, redundant, and scalable.”
Scaling up with Google Cloud managed database services
Whether it’s because of an expanding client base, more frequent traffic peaks, or multiplying data sources, scaling up a system’s data processing capabilities can be costly and time consuming. As the number of data assets it handled grew, Kaiko realized that its previous provider couldn’t offer the managed services it needed to expand its infrastructure quickly and easily. “We wanted to be able to scale without affecting our software and the quality of our service,” explains Michael.
To access the managed database services it was looking for, Kaiko implemented Cloud Bigtable as its main data storage solution, using Cloud Memorystore for its Redis instances. “We accumulate data in Redis, then batch our data every eight hours into Google Cloud Bigtable,” says Michael. “It means we can easily handle the terabytes of data we ingest every day.”
“Google Kubernetes Engine automatically scales from 1 to 100 pods of containers, depending on whether there is a spike or a fall in buying and selling activity. Since setting it up, we haven’t had to do anything to manage the traffic. It’s like magic.”
—Michael Inthilith, Cloud Engineer, KaikoAlthough Kaiko was initially attracted by Cloud Bigtable, it quickly decided to expand its use of the Google Cloud ecosystem. “No other cloud platforms offered the level of services or support that we experience with Google Cloud, ” says Michael. In order to be able to scale automatically, it migrated its normalization and deduplication software to Google Kubernetes Engine (GKE), using Cloud Pub/Sub to manage data ingestion and Cloud Build to support continuous deployment. Kaiko also uses Stackdriver for its monitoring and logging needs, including for the workloads that are still running on its legacy infrastructure.
“Google Kubernetes Engine automatically scales from 1 to 100 pods, depending on whether there is a spike or a fall in buying and selling,” says Michael. “Since setting it up, we haven’t had to do anything to manage the traffic. It’s like magic.”
Deploying multi-regional resources to keep latency low
Not only does Kaiko have clients around the globe, the exchanges it is connected to to collect trade data are also located worldwide. Keeping latency low is particularly important for its real-time feature, which enables clients to see every executed action on an exchange in a live feed accessed through a websocket communications protocol. It uses Cloud Load Balancing to distribute its global traffic and keep services available.
Thanks to the global Google Cloud network and Terraform, Kaiko is able to easily replicate its infrastructure for new regions and use multi-regional deployment to keep its real-time delivery under 150 ms. “Being able to access redundantly deployed resources across multiple data centers is a great selling point for us,” says Michael. “As we use Terraform, we can replicate our infrastructure in a new region in seconds. We are also able to backfill data to any region in case of outages, easily merging data using Cloud Bigtable.”
“As we can easily replicate our infrastructure from region to region and autoscale with Google Kubernetes Engine, we save a lot of time. That’s important because our tech team is relatively small, and we want our engineers to concentrate on adding business value.”
—Michael Inthilith, Cloud Engineer, KaikoDelivering at speed anywhere in the world
Since implementing managed services on Google Cloud, Kaiko is able to spend more time migrating its infrastructure and developing new product features, such as its livestream. “As we can easily replicate our infrastructure from region to region and autoscale with Google Kubernetes Engine, we save a lot of time,” says Michael. “As a growing team, this is important so that we can use that time to concentrate on adding business value.”
As well as saving time while maintaining the quality of its service, Kaiko also expects to reduce costs, thanks to Google Cloud. “Reducing costs was not our primary concern when we decided to switch to Google Cloud, but we project that after fully migrating our monitoring and alerting solution, monitoring costs will be halved,” says Michael. “Features such as preemptible nodes on GKE also enable us to make savings where possible, and Cloud Billing really helps us keep an eye on our spending.”
Kaiko is currently running its legacy solution alongside Google Cloud, but it plans to continue migrating as many of its services as possible to prepare for future spikes in demand.
“There may still be some resistance among financial institutions regarding cryptocurrencies, but very soon that mindset will change,” says Michael. “When it does, our service is ready and waiting, and thanks to Google Cloud, we can deliver it at speed anywhere in the world.”
Tell us your challenge. We're here to help.
Contact us
About Kaiko
Kaiko is a digital assets data provider that covers all major cryptocurrency exchanges. Founded in 2014 and based in Paris, the Kaiko team combines years of expertise in data science, finance, and distributed systems to provide its clients with access to high-quality and transparent cryptocurrency data.