Mercari: Running microservices and enabling smartphone payment with Google Cloud
About Mercari
Established in February 2013, Mercari Inc. is Japan’s largest peer-to-peer marketplace. The business expanded from Japan to the United States in September 2014 and the United Kingdom in March 2017. As of December 16, 2017, the mobile app had been downloaded 100 million times and by early 2018, the organization had grown to 400 employees. The organization has also created sister services that specialize in the sale of books, CDs, and DVDs.
Tell us your challenge. We're here to help.
Contact usWith Google Cloud, Mercari has created a microservices architecture running in Google Kubernetes Engine that enables the rapid development of new features for the United States version of its marketplace, while minimizing infrastructure management resource requirements. The business has added Cloud Spanner to provide a fully managed relational database solution to support the Merpay integrated smartphone payment service.
Google Cloud results
- Avoids need to employ a full-time infrastructure engineer
- Processes thousands of transactions per second with up to 99.999% availability
- Gains ability to reserve compute capacity in advance to manage demand peaks
- Removes infrastructure as a concern during development process
Created a search function in less than one month with a microservices architecture
Established in February 2013, Mercari Inc. has grown from a consumer-to-consumer “flea market” to Japan’s largest peer-to-peer marketplace. Mercari enables users to take three simple steps to sell goods and services on its app. First, they take a photo; second, they write a description; and third, they press a submit button to send the post live. This process takes only about three minutes to complete.
Another key feature of Mercari is the use of an escrow settlement system to keep transactions safe and more secure. Furthermore, users can buy and sell without revealing any more information about themselves than they want to.
The service launched in the United States in September 2014 and in the United Kingdom in March 2017. As of December 16, 2017, the mobile application had been downloaded 100 million times and by early 2018, the organization had grown to 400 employees. The organization has also created sister services that specialize in the sale of books, CDs, and DVDs.
“There are several reasons we chose Google Cloud. First, its load balancing service is very good. Second, we can reserve compute capacity in advance to accommodate a lot of users gaining access to our system or during a load test. I also like Google Cloud’s ease of use and its compatibility with Kubernetes, which supports a range of container runtimes.”
—Suguru Namura, Chief Technology Officer, MercariA multicloud architecture
Mercari operates a multicloud architecture with different cloud services supporting different regions. These services typically ran a standardized technology stack developed in Japan. However, while this model enabled the rapid launch of new features and services, making the Mercari marketplace a compelling proposition for audiences in different markets proved more difficult.
“In order to make Mercari even more popular in the United States, I thought it would be necessary to adopt a user interface that matched the local culture,” says Suguru Namura, Chief Technology Officer at Mercari. “Therefore, at the end of 2016, we decided to completely renew the United States version of our app.” The business opted to move from a monolithic to a microservices architecture.
“A microservices architecture enhances scalability and productivity and enables us to recruit skilled people in the markets we are expanding into,” adds Suguru. “Many engineers in the United States are oriented towards modern programming languages such as Go rather than PHP, which is commonly used in conventional, monolithic systems.”
Furthermore, Mercari’s monolithic architecture had become too large and unwieldy for engineers joining the business to work with effectively. A microservices architecture would enable teams in the United States to work independently to develop new functions.
“Creating high quality functions—such as a renewed search function—that make our marketplace easier to use is quick and easy with microservices running in Google Cloud.”
—Suguru Namura, Chief Technology Officer, MercariAn API gateway on Google Cloud
To complete the process quickly, the Mercari United States team elected to wrap its conventional system with an API gateway developed on Google Cloud. The API gateway enables groups of microservices to access APIs and provides features such as authentication and security policy enforcement. This approach enabled the business to implement microservices efficiently step by step.
“There are several reasons we chose Google Cloud,” says Suguru. “First, its load balancing service is very good. Second, we can reserve compute capacity in advance to accommodate a lot of users gaining access to our system or during a load test. I also like Google Cloud’s ease of use and its compatibility with Kubernetes, which supports a range of container runtimes.”
The organization elected to run the microservices architecture in a containerized environment through Google Kubernetes Engine, supporting the rapid development and iteration of its applications.
Taichi Nakashima, Site Reliability Engineering team member, says the decision meant Mercari United States was able to “start development in the microservices architecture without thinking about infrastructure.”
“The fact Google Kubernetes Engine is a managed service means the United States team does not need to employ a full-time infrastructure engineer.”
—Suguru Namura, Chief Technology Officer, MercariNew search function developed in one month
The new architecture running in a containerized environment enabled the business to seamlessly create a version of its marketplace optimized for the needs and behaviors of customers in the United States. The new application, released in April 2017, featured an innovative personalized home screen, giving Mercari an edge over its competitors.
“Creating high quality functions—such as a renewed search function—that make our marketplace easier to use is quick and easy with microservices running in Google Cloud,” says Suguru. Developing the new search function took less than one month rather than the two months required to complete a similar project in a monolithic architecture.
Running the new environment in Google Cloud minimized Mercari’s infrastructure administration requirements. “The fact Google Kubernetes Engine is a managed service means the United States team does not need to employ a full-time infrastructure engineer,” says Suguru.
Mercari is also using Google Cloud services to analyze user behavior and display of user browsing and “like” histories in near real time.
The success of the microservices architecture running in Google Cloud has prompted Mercari Japan to adopt a similar architecture. Meanwhile, the United States team will continue to promote microservices in a containerized environment, making the role of Google Kubernetes Engine in drawing out the full potential of Kubernetes more important to the business than ever. In the future, the United States arm of the business is looking at more managed services and serverless technologies, such as Cloud Functions, to further streamline the development process.
“We are also exploring opportunities with exciting products such as App Engine,” says Suguru.
“For services like Merpay, Cloud Spanner’s lack of downtime. scalability and consistency were very attractive.”
—Keisuke Sogawa, Chief Technology Officer, MerpayDeploying Cloud Spanner
When Mercari launched a new mobile payment platform, it chose Cloud Spanner as part of its data portfolio. This provided the business with easy scalability to handle millions of new users, a fully managed service that minimized overhead costs, and deep integration with other Google Cloud services.
Mercari launched Merpay, a mobile payment service that can be used through Mercari in Japan, in 2017. With more than 85 million users, Merpay is now accepted at 1.8 million merchants and ecommerce sites in Japan, supporting payments via the DOCOMO ID contactless system and QR code.
Prioritizing availability and scalability
When Mercari started building Merpay, the business was looking for a new database. In the past, Mercari had used MySQL with bare metal hardware—however, because of the amount of data, it required additional expertise to manage and maintain the hardware, software, and MySQL implementation. Having built much of the microservices architecture for its Mercari app using Google Kubernetes Engine, it was natural to look at Google Cloud’s managed services when deciding upon its new database infrastructure for Merpay.
For the database, Mercari focused on requirements around availability, scalability, and performance. A single payment transaction required multiple steps, each with writes, requiring high-write throughput and low latency from the database. Needing a solution that would support reliable payment processing 24/7/365, Mercari chose Spanner, which offers up to 99.999% availability with zero downtime for planned maintenance and schema changes.
The business worked closely with Google Cloud’s Premium Support, Technical Account Management (TAM), and Strategic Cloud Engineer and Cloud Consultant teams to implement Spanner, including separating the payment processing—originally one of the functions in Mercari—as a microservice.
A nod to easier nodes and better scale
After launch, the number of Merpay users reached 2 million in only a few months. Mercari had 45 Spanner nodes at the time of launch of Merpay and about 50 nodes four months later. Because it had optimized the application side during those four months, it didn’t have to add many nodes to keep up with the growing traffic.
Whenever Mercari needed to scale up the serving and storage resources in its instances, Spanner made it easy to increase nodes as needed in the Google Cloud console.
Building powerful pipelines
Mercai’s data pipelines are used for KPI analytics, fraud detection, credit scoring, and customer support use cases. Because Cloud Spanner integrates so easily with other Google Cloud data services, the data in Spanner is easily accessible for analytics. From the Spanner database for each microservice, Mercari creates both batch and streaming data pipelines using Pub/Sub and Dataflow, as well as Apache Flink, and loads the data into BigQuery and Cloud Storage.
The business is able to quickly aggregate the data required for data platforms such as BigQuery and Cloud Storage. The original data is stored in Spanner and managed by microservices, but analysis through BigQuery is centrally managed by the Data Platform team. This team determines the confidentiality level of data and centrally manages BigQuery permissions using Terraform. By centrally managing data access permissions on the data platform rather than on individual microservices, Mercari can set appropriate security for individual users.
Mercari’s full data portfolio also includes Looker, which it uses for product analysis, accounting analysis, test performance visualization, operation monitoring, development efficiency analysis, and HR analysis. The business also uses Dataproc, Cloud Composer, Data Catalog, and Looker Studio. The Dataflow template created for the pipeline is also published as OSS.
With Spanner and other Google Cloud services, the Merpay platform is flexible, secure, scalable, and highly available. As Spanner eliminates overhead, the business can devote engineering resources to developing new tools and solutions for its customers. It is looking ahead at providing cryptocurrency service, for example, and is now working on a project to migrate Mercari’s monolithic system that is still on-premises entirely over to Google Cloud.
Tell us your challenge. We're here to help.
Contact us
About Mercari
Established in February 2013, Mercari Inc. is Japan’s largest peer-to-peer marketplace. The business expanded from Japan to the United States in September 2014 and the United Kingdom in March 2017. As of December 16, 2017, the mobile app had been downloaded 100 million times and by early 2018, the organization had grown to 400 employees. The organization has also created sister services that specialize in the sale of books, CDs, and DVDs.