By solving data governance challenges, Mercari improves engineering efficiency and deepens data-driven insight
About Mercari
Mercari, Inc. is a Japanese ecommerce company founded in 1991 with offices in the United States. Mercari is operated by Mercari, Inc., established February 2013, and its group company Merpay, Inc., established November 2017, offers Merpay, a mobile payment service.
Tell us your challenge. We're here to help.
Contact usMercari and Looker team to organize and streamline data for analysis.
- Looker has made viewing data as easy as ever within Mercari
- Looker has been streamlined as the primary communication channel for Mercari
- Looker provides flexibility when analyzing and streamlining vast amounts data
Mercari is a selling app that makes it super easy to sell (or buy) almost anything. It is used by 16.57 million users monthly with a total of 1.1 billion items online (as of March 2020) and an annual transaction amount of 530.7 billion yen (as of June 2019), making it the largest flea market app in Japan.
Data governance challenges in a microservices architecture
Mercari’s mission is “to create trust for a seamless society.” With this mission statement in mind, the data analyst team works on data analysis in order to create a good product, build relationships with stakeholders, and ensure robust operations. Mr. Hasegawa works on the data management team, responsible for mediating between developers and analysts.
“The role of the data management team is to establish governance for both the upper process where data is generated from the customer-used app, as well as the lower process where this data is collected and analyzed for business use,” explains Mr. Hasegawa.
At Mercari, systems are developed based on the concept of microservices. Each service, such as pay, search, checkout, and sell, is developed separately, and its data storage is also independent, exchanging data with other services through APIs. While there were benefits of smaller teams being able to expedite developments, there were also challenges in not knowing which microservice had what data. Organizing such situations in a cross-sectional way, the data management team’s role is to provide support for the overall business.
“An analyst who analyzes data must collect and study all data. Data from each microservice is stored in a separate environment. Sometimes the same data is maintained in different databases, so we need to consider how to link services. Data governance that standardizes this rule and intermediate data maintenance would be our job,” details Mr. Oshitani. The company culture at Mercari Group gives much attention to data. Viewing data daily and deciding what should be done with it has become a habit for each employee.
In the past, an employee who belonged to the customer support or data analysis team wrote their own SQL query and analyzed it. Later, BI tools were implemented company wide, and it became easy for anyone to access data from a dashboard. However, this also created inconveniences. The style and quality of queries largely depended on the individual, therefore company metrics lacked consistency. Also, data source relationships became unknown, and graphical depictions began taking longer. Furthermore, there were many instances where a dashboard to see necessary data already existed, yet another member created a dashboard with similar features. This led to a cost increase in internal data infrastructure management and maintenance that could not be overlooked.
“We hope to further improve internal data literacy. Using Looker with a better understanding of its features will allow examination of data from a deeper viewpoint.”
—Mr. Noda, Analytics Department, MercariLooker established as the data platform standard
At Mercari U.S., Looker was used earlier than it was at Mercari Japan, as some employees had prior experience using Looker. Importing this experience from the U.S. to Japan, Mercari Japan decided to use Looker as well.
In its trial stages before the official implementation, various Looker features received positive feedback from engineers within the company. Engineers who create dashboards especially liked that they could use GitHub for management. Not only would definitions of KPIs and aggregate queries that form the basis of the dashboard become manageable using GitHub as code in LookML, Looker's centralized, accessible modeling layer, it would also be possible for the dashboard itself to become coded and be version controlled. In addition, when KPI and aggregate queries are created, participants can review them on GitHub, so data quality is maintained. Other positive feedback included the availability of dashboard browsing status check and access control on a graph extension, quick processing with operations done on Google BigQuery, and easy integration with Slack.
In early 2018, the analyst team in Mercari Japan initiated a task force of data-competent personnel from accounting, CRM, and marketing. Mr. Hasegawa and the data analyst team proposed setting an end date to completely close the BI tool used up until that point, then make a total transition to Looker. Mercari Japan management approved this proposal, with almost no changes.
“The analyst team makes it a routine to work with management and each product team to promote data governance together with each department," Mr. Hasegawa shares. "A major reason why this transition plan was readily approved was that management and the staff trusted us.”
“The biggest achievement is how the acknowledgement that 'Looker is a place to view data' has now set roots within the company. In the past, there were those who used BI tools and others who used spreadsheets, but Looker has now become the common communication channel," says Mr. Hasegawa. "Another positive outcome was that after transitioning to Looker, data sources were nicely organized and arranged in order.”
Mercari’s analytics team is in charge of reporting insight obtained from data analyses to management and testing hypotheses created by project managers. How were these tasks affected by Looker?
“We would like to spend more time in extracting insight and digging deeper into data, rather than monitoring data. By using Looker, we can now focus on finding the insight that we were looking for in the first place,” explains Mr. Noda.
“By writing LookML with an understanding of database architecture, the true power of Looker is brought out.”
—Mr. Ositani, Data Management Department, MercariIncreased work efficiency with Slack integration
The engineers’ increased work efficiency is also notable with the integration of Slack, a chat tool. Many companies already utilize the function to share Looker-created graphs by posting them on Slack, but some also need to share numerical data in its text format in order to be imported and pasted to use in other apps. So at Mercari, a code to generate text data is written in Looker, and its result is posted on Slack.
In a conventional database operation, the format of data taken from an external data source had to be adjusted, using ETL tools in a prior conversion process. Looker is equipped with a simple ETL function that converts data on the spot to make it available for reuse, flexibly accommodating member needs
Also, Mercari uses BigQuery as its data warehouse. In case there is an error with BigQuery overload, there needs to be some way to alert the administrator. It takes some time and effort to do these settings in BigQuery, however, with Looker, notification scripts can be easily entered in Slack. Engineers love this function.
Even in user management, Slack is used to increase work efficiency. For a user to enter LookML codes, he would need permission to access developer tools. This permission is granted by the administrator each time a user requests it, but at Mercari, there is a system where the user can request access to a bot on Slack for automatic approval. Of course, there is an issue of governance and cost in liberally providing access permission to developer tools. So in the future, a procedure is considered for permission of users with no access for a set period of time to be automatically cancelled.
By heavily using Looker throughout Mercari, more of its benefits started to surface. “A standard BI tool is good enough if we just need to write queries and generate graphs. But with Looker, we can get desired insights by combining current data with past data, without writing complex queries," explains Mr. Hasegawa.
At Mercari, approximately 1,800 employees use Looker, and more employees now have permission to use Explorers, allowing them to build dashboards. Employees can not only browse data, but also can obtain a deeper insight by processing and combining data on their own. A culture with emphasis on data is evolving.
“When building a report output screen in LookML, we can combine detailed choices such as which query to use, and which data to operate," details Mr. Oshitani. "Appropriate settings can also be applied to large-scale data in LookML to extract a deep insight. Superior flexibility between layers and steps is what other BI tools do not have.”
In the future, Mercari will enhance data/Looker literacy employee training.
Tell us your challenge. We're here to help.
Contact us
About Mercari
Mercari, Inc. is a Japanese ecommerce company founded in 1991 with offices in the United States. Mercari is operated by Mercari, Inc., established February 2013, and its group company Merpay, Inc., established November 2017, offers Merpay, a mobile payment service.