Shiseido: building a data analysis platform using BigQuery for 80% cost savings

The cosmetics industry is one of the industries that has been strongly affected by the COVID-19 pandemic, as people’s habits changed — both in how the shop and how they want to show up, whether that's at work, to a party, or just out and about. Even as our lives have found a new normal of get-togethers and hybrid work, the market has continually evolved since 2020. 

Shiseido, one of Asia and the world's largest beauty brands has responded by shifting our business model to focus on each consumer, in order to accumulate profits over the medium- to long-term, rather than increasing sales in the short-term. 

To do that, we need to understand each consumer like never before. And to do that, we had to understand our data at a level we never had before.

I joined Shiseido in early 2022, as Data Analytics Group Manager in the Customer Strategy & Planning (CS&P) department, and this responsibility fell under my purview. I oversee a team that builds strategies, creates business plans, and deploys them to distribution and sales sites. 

As a starting point, we decided it was imperative to improve our data utilization process.

Shifting from an inefficient data analysis environment to BigQuery

Previously, Shiseido did not have expertise to manage data in-house. Therefore, we had to outsource data processing and analysis. We were also outsourcing the data processing services on a project basis, which led to segmenting workloads; that segmentation caused is to frequently duplicate our data processing work.

This also burdened our development team, with new requests constantly being made. We were in a situation where we had 20 to 30 inefficient database servers running in parallel. The costs and time required for analysis-processing started to grow beyond what was realistic.

In order to improve this situation, we decided to build a data analysis platform that could be used organization-wide. We set three key goals for our new data analysis platform, which were faster processing, reduced costs, and a structure to streamline operations and customization. After conducting proof of concept (POC) on all major cloud platforms, we ultimately adopted Google Cloud. 

In particular, we found that BigQuery offered attractive processing performance that came with cost reductions. From the perspective of data utilization, the many features related to AI were also attractive.

System configuration with ease of operation and future expandability in mind

This diagram shows the system configuration of our new data analysis platform built on Google Cloud.

Yoko Nishio, Platform Development Manager (Data Engineering) at BrainPad Inc., who was in charge of development, explained the points unique to Shiseido:

“One key feature is that we deliberately separated the application layer and the infrastructure layer, and adopted a configuration in which both layers can be accessed through a shared virtual private cloud.”

In the future, our line-of-business units will use the application layer, while the infrastructure layer will be used for the information systems of the entire Shiseido Group. The latter will be independently operated and maintained by Shiseido Interactive Beauty, Shiseido’s group company that leads its digital transformation. To do this, Shiseido Group must meet high-security regulations when building the system. 

Splitting the system infrastructure into two parts allows for more versatility and heightened security. If an employee uses Google Cloud in a project, they can simply connect to the infrastructure layer via the virtual private cloud in order to share the project internally, and in a secure manner. As for the application layer, we proactively selected managed services that are easier to handle, as they will be managed by line-of-business teams.

During the development, the first priority was to launch the system on time. To achieve this, the existing data processing method was retained and and migrated. Nishio says that this was the most difficult part of the development.

Seamless data migration enabled by SQL and DataFlow

The biggest challenge to executing on time was migrating the existing processes for the data processing area. 

The stored procedures that were running on the existing SQL Server were large-scale, dynamic SQL modules with about 3,000 rows. In other words, they were quite complex. Rather than rebuild them, we prioritized the schedule and decided to migrate approximately 80 of the existing 130 processes in the processing area to BigQuery.We first used Google Cloud’s migration tool in order to convert the existing processes in the stored procedures, but due to the complexity of the existing structure, we were not able to automatically convert the processes and had to handle it manually. Also, the existing environment had a loose collation, where it did not have the ability to distinguish between the half-sized and full-sized letters used on Japanese keyboards. Because of that, it was necessary to take measures on the source data side to obtain equivalent results in BigQuery, where data is strictly differentiated.Data migration in the existing environment was just as difficult. Because the data was siloed, it was harder to organize and select tables when consolidating to BigQuery. On the positive side, we migrated over 100TB of data easily thanks to SQL and Dataflow. In particular, SQL Server and Cloud SQL were highly compatible, which allowed us to carry out the migration with zero import errors or value discrepancies.

Enhancing functionality with multi-cloud support and AI/ML

Following the successfully delivered POC in July 2022, requirements were defined in November, and development began in December. By the end of June 2023, we had begun full-scale operations of our data analysis platform. 

The results exceeded our expectations. By consolidating our previously disparate servers into Google Cloud, we reduced the weight of storage and improved processing efficiency, resulting in an 80% cost reduction and a 90% reduction in processing time. We’re steadily expanding our in-house operations, which has allowed the team to move away from our dependence on outsourced companies. This has had the benefit of adding more clarifications to the workload. 

Having optimized the data processing aspect of our existing environment, we’re now working toward a second phase. This is designed to further refine processing performance, accelerate the provision cycle by adding external database linkage and automating validation checks, and democratize data analysis by integrating with Looker, Google Cloud's business analytics platform. 

We have also begun working to improve analysis through various AI/ML measures using Vertex AI. 

As our third phase, we plan to go multi-cloud by integrating our data from Google Cloud and other storage clouds owned by the Shiseido IT department. We have high hopes for BigQuery Omni, which allows us to cross-utilize data held in other clouds. We are also paying attention to topic-generation AI. For example, we have begun creating a system for coupon distribution content that automatically generates text and images based on customer classification.

A year-long development process, in review

Our interactions with the support team over the past year of development have left a lasting impression on us. In developing our platform, the Google Cloud team provided generous support for analysis that went beyond infrastructure. We believe that by leveraging Google Cloud, we will be able to comprehensively upgrade our data analytics.I would also like to take on the challenge of changing the image of Shiseido. In a not-so-distant future, we’ll have a greater presence in our customers’ lives, and not just be seen as a company that sells cosmetics products in the storefront. For instance, wouldn’t it be interesting if we can provide a service in which a customer wakes up in the morning to wash their face, and through the image taken by the mirror of that customer, a communication would start between that person and Shiseido, where Shiseido would provide the optimal lotion through the dedicated device.It may sound like a distant goal, but I think it’s crucial for us to start thinking in a broader perspective in order for Shiseido to continue to grow in the long term. We understand the need to deliver more value to our customers as soon as possible, and we are committed to working towards that.