Understanding search responses & multi-detection

Vision API Product Search can detect and find matching products for a single product in a search image, or identify and list matching products for multiple products if they are present in your image.

Single-product images & response

This search image contains a single product.

dress image in cloud storage bucket
gs://cloud-ai-vision-data/product-search-tutorial/images/468f782e70ba11e8941fd20059124800.jpg

Searching for similar products with Vision API Product Search returns a response that indicates two bounding boxes in the image, one box around the product and another box around the entire image.

Expand the section below to view the entire response.

Bounding boxes & matching products

The following two products (bounding boxes) in the image are specified by the two sets of vertices returned from the request.

Matching products are listed for each box identified by Vision API Product Search.

image with 3 dresses in cloud storage bucket
gs://cloud-samples-data/product_search/multiple_dresses.jpg

In the API response bounding boxes and matching products results are listed in the productGroupedResults field. They are grouped by products detected in an image. Each boundingPoly entry lists the region specified with normalized vertices (on a 0 to 1 scale, not corresponding to the actual pixel values of the image). Matching products for that specific region are then listed.

The following section from the response corresponds to the bounding box around the dress.

Matching products are listed after the results field, and are ranked by matching score in descending order. Scores range from 0 (no confidence) to 1 (full confidence).

image of 5 matching products
Several matching products for the dress

The other boundingPoly returned in the response is around the entire image. Its normalized vertices are listed as:

  • [{},{"x": 1},{"x": 1,"y": 1},{"y": 1}]

When coordinate values are 0 they are omitted. Thus, the bounding poly specified above corresponds to:

  • [{"x": 0,"y": 0},{"x": 1, "y": 0},{"x": 1,"y": 1},{"y": 1}]

Matching products are then similarly listed for this bounding box (the entire image).

Multiple-product images (multi-detection) & response

The following search image contains multiple products.

image with 3 dresses in cloud storage bucket
gs://cloud-samples-data/product_search/multiple_dresses.jpg

Using Vision API Product Search to search for similar products returns a response that identifies multiple bounding boxes in the image, one for the entire image and also three boxes for each of the three dresses.

Expand the section below to view the entire response.

Bounding boxes & matching products

The following image shows all bounding boxes identified by Vision API Product Search in this multi-product image.

image with 3 dresses in cloud storage bucket
gs://cloud-samples-data/product_search/multiple_dresses.jpg

Similar to a single-product image, a multi-product image's response lists bounding boxes and matching products results in the productGroupedResults field. Each product is identified by a bounding poly, and matches are grouped by each product detected.

The following section from the response corresponds to the bounding box around the middle dress.

The following image shows some of the matches for the middle dress returned by Vision API Product Search.

image of 5 matching products
Several matching products for the middle dress

The following other bounding boxes are found in the image, each with their own list of matching products (not included here):

  • left dress: {"x":0.03, "y":0.07},{"x":0.29, "y":0.07},{"x":0.29, "y":0.95},{"x":0.03, "y":0.95}
  • right dress: {"x":0.63, "y":0.09},{"x":0.89, "y":0.09},{"x":0.89, "y":0.93},{"x":0.63, "y":0.93}
  • whole image:
    • {},{"x": 1},{"x": 1,"y": 1},{"y": 1}, also expressed as:
    • {"x": 0, "y": 0},{"x": 1,"y:0"},{"x": 1,"y": 1},{"x": 0,"y": 1}