Why retailers should switch to Google Cloud Retail Search

Many of us already know about Google Search, which provides relevant results with low latency, all while accommodating the user’s personal preferences and history. The results are quite relevant to each individual, such that they quite rarely browse to the next page of results. Retail Search is a service provided by Google Cloud for retailers to use similar Google Search type capabilities, but with the retailers' own products.

Today, retailers have many choices to build a search system to power their ecommerce website. So why should they use Retail Search? To understand the reason, we should first try to understand:

• How critical is the search system for ecommerce retailers? This can help retailers put more focus on those systems that can drive up their business performance. 

• What process do they follow to build one? This can help retailers identify if the process can be improved so that it reduces the total development and operating cost. 

• What challenges do they face? This can help retailers easily compare if other search providers can help solve most of the challenges. 

Criticality of search systems for ecommerce retailers

Let’s try to answer a simple question. How critical is a search system for ecommerce retailers? To answer this question, we should know which pages on the retailer’s website are powered by the search system and how important they are for conversion rate. 

Sales funnel

Typically many ecommerce retailers classify the sales funnel into three categories:

• Top of the funnel (TOF): Home page, landing pages, category pages, search pages, product detail pages 

• Bottom of the funnel (BOF): Cart page, checkout pages, order confirmation page

• Post purchase: Order management, order cancellation, account pages

Customer traffic is generally higher on the top-of-funnel (TOF) pages and it decreases as it reaches the bottom-of-funnel (BOF) pages. Noticeably, there are more TOF pages than BOF pages. We can state that TOF pages are what drives the customers to eventually convert. A customer ends up spending a lot more time on TOF pages then they do on BOF pages, so clearly retailers need to place a lot of emphasis on those pages. As important as the cohesive customer experience is for the entire sales journey, TOF pages play a much significant role in that journey. Customers typically land up on TOF pages first before they can begin the purchase process. 

As TOF pages are the main contributor for initiating the customer journey, a poor customer experience on these pages can lead to a high abandonment rate. 

So far we can state that the TOF pages: 

• Could be more prudent in conversions 

• Have more types and number of pages 

• Have more customer traffic 

• Initiate the customer journey 

Page distribution powered by search

Clearly TOF pages are very critical for retailers, however which TOF pages are actually powered by the search system or other backend system? Let's go over the main TOF pages.

Category pages
Let's start with the category pages with an example like Apparel -> Mens -> Shirts category page.

This page simply shows products that belong to the Shirt sub category under Men's sub category under the Apparel category. Which system does the ecommerce website query to fetch the products in that  category? Although there could be many source systems powered by some database, more often than not, it is the search system that is leveraged to fetch products for those category pages. In this example, a special type of query is sent to the search system that does not search for any keyword but rather simply asks the system to filter the products in the category called Apparel -> Mens -> Shirts. So although the customer did not explicitly invoke search with a keyword, the ecommerce website still leverages the search system to power these pages. Note that sometimes category pages are completely hand-curated by merchandisers, in which case, the search system is not invoked. However, typically the category pages are way more dynamic and are powered by the search system itself. 

Landing pages
What about the landing pages? These are the pages where a customer directly lands on the website because they have been advertised from some external sources, email or other marketing outlets. These pages can be completely static and fully curated manually by merchandisers.

However, many advanced retailers also make these pages dynamic by breaking them down into several sections, many of which show products from different categories. As shown in the figure above, this is one approach retailers take while rendering landing pages. Note that some retailers may completely end up showing a different layout with different data on these pages. 

So we again ask the question of which system is used to show those top five to 10 products per each category section.  Again, more often than not, it’s the search system. Just like category pages are powered by issuing filter queries to the search system, similar but multiple queries are issued to the search system to fill in the data for multiple category sections in these landing pages. 

Home page
Let’s try the home page now. Many ecommerce websites have home pages broken down into various sections like recommendations, best sellers in a few categories, deals, promotions, hero images, etc.

The top products for the best sellers category and products from deals category again follow a similar browsing pattern like the one in the landing page. Multiple queries are invoked and sent to the search system to filter products in that respective category. Note the deals could merely be yet another category that the retailers have hand curated. 

Product detail pages
What about product detail pages (PDP)? A recommended way to use the search system is to retrieve product identifiers based on criteria and then decorate the results with additional product attributes from another source system like a product catalog service or product information management (PIM) system.

Typically, product detail pages are not powered by search engines, however some retailers try to get even the product details from the search system and treat it as the product data’s source of truth. Although not recommended, some retailers leverage search systems to power product detail pages as well. 

Product detail pages typically also have a section to show other products of similar type or have a recommendation section that provides a bit of a personalization experience to the customer. Retailers can build yet another recommendation engine to power this functionality. Activity data, product catalog data, etc., is generally fed into a recommendation engine just like it would be fed into a search system. Note that Google offers Discovery AI for retail that combines the capabilities of retail search and recommendations all while using the same data and data pipelines, so that retailers don’t have to manage two systems and two data formats. 

Search pages
Lastly, let's consider search pages. This is the product result page which shows products that match the search criteria using the keyword supplied.

This page is indeed powered by a backend search system. Retailers have merchandisers who have to invest a lot of effort to tune the search system and show the right set of products for different search terms.

The centrality of search

From this analysis, we can see that: 

• Modern customers desire an easier and faster way to find products they are interested in.

• Retailers may build and enhance functionality on TOF pages to assist those modern customers in their journeys.

• Functionality on TOF pages may end up being powered by a core backend search system and recommendation system.

Given that TOF pages are very important for customers and the majority can be powered by search systems, it can be concluded that search systems could be one of the most critical systems for ecommerce retailers. As the influence of search systems on the overall ecommerce journey is so large, retailers should carefully choose a good search system that can provide the best customer experience with least effort. 

Implementing search

So let’s see how ecommerce retailers approach implementing a search system and what challenges they face. Here is the typical process of implementing search that retailers follow immaterial of the above choices 

Product data integration 

The retailer builds some type of pipeline to get the source product data ingested into the search engine of choice. They start testing the search engine by first bulk importing all the product data. In this process, they learn that the source data will have to be adapted to adhere to the schema prescribed by the search engine of the choice. 

After initial testing, the retailer creates yet another streaming pipeline to ingest product catalog updates incrementally. Note that these updates may also include availability data, pricing data, inventory data, etc., that are finally clubbed together and stored in the search system for later retrieval. A messaging service is generally leveraged to achieve this streaming capability.

Challenges faced 

• Product data may be in many different sources and the retailer will have to develop some form of aggregation from all those sources. This requires an investment of time, money and resources. 

• At this stage, the search engine works mainly on relevancy based on term frequency and inverse document frequency.  

• The retailer realizes that the source product data requires a lot of cleanup as the keyword search does not always return the products in the order they would like. 

• A flat and wide product document is curated in the search engine to host all the product attributes including SKU availability, inventory, pricing information. 

• Furthermore, even for clean source product data, relevancy is based on frequency and not optimized for business KPIs like product margins and sales. For example, searching for a term may end up returning the order of products that the search engine deemed relevant because of the frequency of the term instead of product performance.

Manually fine tuning search results 

Merchandisers start manually configuring the search engine to match the relevancy they want. In this process, they add rules related to synonyms, spell checks, redirects, boost and bury, etc. Note that Google recommends to use merchandising rules judiciously and sparingly, as authoring too many is considered an anti-pattern.

Challenges faced 

• The retailer realizes that it takes a lot of effort to keep fine-tuning search rules regularly. 

• Furthermore, if the product catalog is too large, they require a lot more resources to assign fine-tuning duties separated by category or vertical. 

• Finally, even after authoring hundreds of rules, there are still many tail-end queries that do not yield relevant results. Tail-end queries typically are multi-word phrases. For example, the merchandiser may have added a boost rule for a query like dress but a tail-end query like black dress for women may still not work for them. 

• Developers may have to build yet another merchandising rules interface for business users if the search system does not provide it out of the box.

• Some rules end up being authored based on intuition rather than real data, leading to lower optimized revenue than expected.

• Some rules start showing biases for a particular brand or product. 

Product performance ingestion

To automate and optimize relevancy, retailers will build other pipelines to add attributes related to product performance into the search engine. These new performance attributes vary from conversion rate to impressions on the websites. Retailers have to adjust the schema of the product documents in search engines to make room for these attributes. Once they are indexed as part of the product catalog, more search rules are added to further overlay the effect of boosting/burying based on product performance. This provides retailers the ability to optimize relevancy automatically instead of manually. However, they have to integrate performance data into the search index first to make use of this capability.

Challenges faced 

• Retailers have to keep altering the schema of the product documents in search engines to add any type of performance attributes. The search engine does not separate product data and product performance data and they have to work with a single wide column product catalog. 

• They realize that even after automating relevancy, there are many cases where they still have to manually override the order of the results. These are both single-term and multi-term phrases that the customer may search for but that merchandisers have not incorporated in their triggers for their rules. 

Machine learning

At this stage, simple keywords like watches, dress, etc., work quite well for the retailer. However, the intent of the user with multi-phrase keywords like halloween dress, blue shirt for men still falls short. Simply matching the keywords — even with all the authored search rules — does not yield optimal results. For example, searching for four terms like blue shirt for men yields different results than searching for shirts in the apparel category, filtered by attributes like color and gender of blue and men, respectively. To achieve this, the retailer must build a machine learning model to parse the queries and break down the user’s intent, and build search queries accordingly.

Challenges faced 

• The retailer has to hire ML resources who can build models to leverage a simple natural language processing (NLP) capability. 

• The ML model requires constant tuning with new keywords. 

• There is ever growing backend logic to generate the correct query for any multi phrase terms. 

Personalization 

The retailer realizes that relevancy is uniform across the board and does not take into account customers preferences, past purchases, etc. Any time a new feature (like inventory, fulfillment) is required from the search engine, more data has to be added into the product catalog, leading to a very wide product document. The same however, can not be easily done with personalization data. Because personalization data is unique to each customer, adding it directly in the product document to make systematic personalized queries is not quite feasible. At this stage, sometimes a separate system of personalization is built outside the search purview, both systems are queried and results are merged, providing a subpar personalization experience. This also often leads to counts mismatch on faceting and pagination.

Challenges faced 

• The inability to add personalization capabilities to a search engine leads the retailer to develop a proprietary one from scratch.

• Building a separate one leads to yet another integration between the two systems and keeping data in sync. 

• Personalization data is sometimes broken into higher-level segmentation (like east region, west region, etc.) instead of to individuals to accommodate it in the search system’s product document. This leads to a subpar personalized experience, as it is generalized with groups.

Conclusion 

To summarize, here is what we learned today: 

• Top of funnel pages are very important for ecommerce retailers, as that is where customers spend most of their time browsing. 

• Most TOF pages for many ecommerce websites can be powered by a search engine. 

• The search system is one of the most critical components in an ecommerce website. 

• The effort needed to implement a good search system is quite large. 

• Even after a multi-phased approach of building the search system, the final product can still fall short in terms of personalization, tail-end queries, relevancy, or customer intent. 

• Although some retailers are able to build a good search system for their ecommerce website, not all of them may get the same success. 

• After years of development and ongoing fine-tuning of a complex search system, there are some areas that retailers still cannot get right. 

Traditionally, retailers have chosen between a variety of search engine technologies, including: 

1. Self-managed open-source search engines (Solr, Elastic)

2. Self-managed proprietary search engines (Endeca)

3. Fully managed hosted search engines (Algolia) 

Google Cloud Retail Search is Google's answer to implement search for retailers using the power of AI and solve for most of the shortcomings stated above.

• It begins with relevancy based on intent, leverages personalization from the start, continuously learns and uses AI to build context from the most complex of queries, all while being fully managed. 

• Unlike other search solutions, retail search strives to improve revenue optimization through higher revenue per visit (RPV) by focusing on not just relevance but also for buyability and personalization. 

• Additionally, a recommendation service is offered alongside retail search, using the same data so retailers don’t have to manage two separate systems.

Visit our Discovery AI for Retail page to learn how Retail Search can help you optimize your customers’ search experience, and add recommendations capabilities to your ecommerce site.