Reshaping Flipkart’s technological landscape with a mammoth cloud migration

More than a third of consumers in India turn to Flipkart for their shopping needs, giving the platform a user base of more than 450 million individuals. With 1.1 million sellers, the platform offers over 150 million different products across 80 different categories and millions of shipments daily.

Flipkart was looking to reshape their technological landscape and future-proof their operations, and did so on Google Cloud. The fruits of this relationship are evident, as they successfully launched two of their pivotal platforms, Flipkart Data Analytics Platform (FDP), and Content Catalog Object Store, on the Google Cloud infrastructure among other things that were migrated to Google Cloud.

The biggest day of the year for Flipkart is the Big Billion Day (BBD) sale, akin to the global shopping extravaganza Black Friday Cyber Monday. Big Billion Day spans September and October, coinciding with India's vibrant festival season. Over the course of multiple days, Flipkart experiences a transaction volume surge of six to eight times the ordinary business-as-usual (BAU) volume.

Migrating large-scale, complex data clusters within a short timeframe

Flipkart’s existing data platform was built on top of open-source big data technologies, with extensive customizations tailored to cater to Flipkart's unique business requirements. This intricate environment was spread across self-managed data centers located in Chennai and Hyderabad. Facing tight schedules for migration within approximately eight months, Flipkart was faced with a range of challenges:

• Migrating large-scale Hadoop clusters: Flipkart needed to move substantial Hadoop clusters with significant computational power and memory resources to Google Cloud.
• Complex data migration: The migration spanned beyond infrastructure, encompassing the transfer of over 10k facts, journals, and snapshots, along with 15,000 ETL (Extract, Transform, Load) jobs. This intricate operation involved processing a colossal 10PB of data in batch and 2PB of data in near-real-time, every day
• Data ingestion layers: To facilitate data ingestion at scale, the team needed to transition from on-premises Kafka to Google Cloud’s Pub/Sub. This intricate operation involved ingesting 130+ billion messages per day, 60 TB of compressed data on a daily basis during regular operations, and 500 TB per day during peak event periods. Furthermore, the process encompassed 3,000 topics, spanning both general data and sensitive Personally Identifiable Information (PII).
• Data processing and analysis: Flipkart needed to create a robust processing platform capable of handling a staggering 1.25 million messages per second for real-time analysis. This was done using Dataproc. Additionally, the platform also needed to efficiently process around two petabytes of messages in real-time and ten petabytes of messages daily, in batch mode.
• Migrating self-managed Hive: The migration extended to critical user-facing components, including the migration of the existing self-managed Hive to Hive on Dataproc. This transition aimed to service the requirements of approximately 20 teams and a user base of nearly 350 technical users. Remarkably, the team only had four months with which to accomplish this.
• Security assurance: Ensuring a robust security posture for both GoogleCloud- and self-managed services was of the utmost importance, and demanded meticulous planning and rigorous implementation. Google security experts collaborated closely with the Flipkart team, conducting a thorough evaluation of all migrated components to verify their security status and evaluate potential risks and solutions. This assessment process utilized frameworks such as security posture reviews.

Navigating these challenges was not undertaken in isolation, as Flipkart collaborated with the Google Cloud team, which shared its expertise throughout the migration. What was initially planned as a pre-BBD lift-and-shift migration, but a subsequent modernization phase post-BBD quickly evolved into a much more intricate and nuanced process. Recognizing the complexity of Flipkart's use cases, a multidisciplinary team comprising more than 85 members from Google Cloud and partner organizations was convened to steer this ambitious migration journey.

Exceeding SLAs with a solid cloud infrastructure

With Google Cloud, the team built 15 tools, each designed to cater to unique requirements and challenges presented by Flipkart's intricate ecosystem. As the migration encompassed various elements, including data ingestion, batch processing, messaging, and more, these utilities served as vital cogs in the migration machinery.

Fine-tuning and performance optimization efforts were relentless, ensuring that the new infrastructure met and exceeded service level agreements (SLAs). Rigorous testing, including several rounds of scale testing, reaching up to five times the BAU volume, solidified the foundation for a smooth execution of the BBD sale event on the Google Cloud infrastructure.

The impact of this migration was transformative. Flipkart completed their data platform migration from on-premises to Google Cloud within just eight months. The migration included the smooth transfer of substantial components, such as Hive clusters with a significant number of vCPU cores and a colossal volume of Google Cloud Storage, processing a massive amount of messages in batch and real time. Not to be overlooked, the migration of the content management store, encompassing 5.4 billion objects, stands as a testament to the scale and complexity of this endeavor.

Future-proofing the company through a journey of transformation

The real-world impact of the Google Cloud infrastructure coming to life became apparent very quickly. Not only did Flipkart successfully execute the BBD sale event with a 7x increase in transactional data over BAU, but the event's smooth execution fostered a sense of trust and confidence among internal and external stakeholders alike. This migration journey was not only a technological feat but also a testament to collaboration, innovation, and the potential for transformational change within the realm of ecommerce.

Looking back, both Flipkart and Google Cloud learned invaluable lessons. The migration served as a real-world stress test, pushing the boundaries of large-scale infrastructure within the JAPAC region. And as industries continue to evolve, Flipkart's migration to Google Cloud stands as a model of successful partnership and technological advancement. It not only highlights the power of collaboration, but also the boundless potential of combining innovative technology and human ingenuity.

If you’re ready to migrate, read more about the Rapid Migration Program (RaMP) by Google Cloud, or sign up for a free IT discovery and assessment, to figure out what your IT landscape looks like and ensure a successful migration.