Product Search Tutorial

This tutorial demonstrates how to create a product set which contains a group of products with reference images for those products. The tutorial shows users how to create a product set in two ways: via batch import and via online (individual) import. After the product set has been indexed, you can query the product set using Vision Product Search.

In this tutorial, you will learn how to:

  1. Create a product set through batch and online (individual) import
  2. Create an individual product
  3. Add a product to a product set
  4. Update a product
  5. Create a reference image
  6. Search for similar products

Installation

Python

The Python client library has been updated to reflect the latest changes to the Cloud Vision API Beta. These changes are not backwards-compatible with client libraries older than version 0.30.1.

To install the Python client library:

pip install --upgrade google-cloud-vision

Node.js

To install the Node.js client library:

npm install --save @google-cloud/vision

Java

If you are using Maven, add this to your pom.xml file:

<dependency>
  <groupId>com.google.cloud</groupId>
  <artifactId>google-cloud-storage</artifactId>
  <version>1.37.1</version>
</dependency>

If you are using Gradle, add this to your dependencies:

compile 'com.google.cloud:google-cloud-storage:1.37.1'

If you are using SBT, add this to your dependencies:

libraryDependencies += "com.google.cloud" % "google-cloud-storage" % "1.37.1"

If you're using IntelliJ or Eclipse, you can add client libraries to your project using these IDE plugins:

Besides adding client libraries, the plugins provide additional functionality, such as service account key management. Refer to the documentation for each plugin for more details.

Set the following environment variables where /path/to/service_account.json is the path to your service account key files (see Before you begin), project-id is the ID of your Google Cloud Platform (GCP) project, and region-name is the GCP location that will run your tutorial, for example, "us-east1". Valid location identifiers are: "us-west1", "us-east1", "europe-west1", and "asia-east1".

export GOOGLE_APPLICATION_CREDENTIALS=/path/to/service_account.json
export PROJECT_ID=project-id
export REGION_NAME=region-name

Import Libraries

To use the Cloud Vision Product Search, download and install the client library and import the following modules:

Python

from google.cloud import vision_v1p3beta1 as vision

Java

import com.google.api.gax.longrunning.OperationFuture;
import com.google.cloud.vision.v1p3beta1.BatchOperationMetadata;
import com.google.cloud.vision.v1p3beta1.ImportProductSetsGcsSource;
import com.google.cloud.vision.v1p3beta1.ImportProductSetsGcsSource.Builder;
import com.google.cloud.vision.v1p3beta1.ImportProductSetsInputConfig;
import com.google.cloud.vision.v1p3beta1.ImportProductSetsResponse;
import com.google.cloud.vision.v1p3beta1.LocationName;
import com.google.cloud.vision.v1p3beta1.ProductSearchClient;
import com.google.cloud.vision.v1p3beta1.ReferenceImage;

Running the Application

Step 1: Create a Product Catalog

Users have two options for creating a product catalog, either via batch import using a CSV file, which allows an entire product catalog to be imported in a single API call, or via online import, which offers you control over your product sets and allows for management of one resource or relationship at a time. This primarily means individual creation of product sets, products, and reference images. Online import also allows you to incrementally update a product catalog you have already created via batch import.

Option 1: Batch Import

You must initially pass the list of products belonging to a dataset (Product Set) and create an index in Vision Product Search. The import_product_sets method creates one or more product sets by creating one reference image for each line of the CSV, as well as all required products for those images as indicated by the appropriate CSV columns. The system will also create the required product sets and automatically add each product to the appropriate product set.

Product Images CSV File

You can provide image data to Vision Product Search using a CSV file. The CSV file contains the list of image locations of all products to be indexed for your product set. The file must be uploaded to a Google Cloud Storage bucket. The CSV format is described here.

The following shows an example of rows from a CSV file:

gs://cloud-ai-vision-data/product-search-tutorial/dress-shoe-dataset/469a896b70ba11e8be97d20059124800.jpg,image-id1,12000002,469a896b,apparel,"product name","style=women,category=dress","0.0,0.0,0.4,0.5",
gs://cloud-ai-vision-data/product-search-tutorial/dress-shoe-dataset/469ac87870ba11e88fe4d20059124800.jpg,image-id2,12000002,469ac878,apparel,"product name","style=women,category=dress,kids=true",,
gs://cloud-ai-vision-data/product-search-tutorial/dress-shoe-dataset/469ae13370ba11e894f4d20059124800.jpg,image-id3,12000002,469ae133,apparel,"product name","style=women,category=shoe","0.0,0.1,0.2,0.3",

The import_product_sets method creates a product set with a product set id of “12000002” in this example, which may correspond to a business' internal code for a particular department. The method then adds all the products (with product ids "469a896b", "469ac878", and "469ae133") and reference images. Labels are created for columns such as ‘style’, ‘category’, and ‘kids’.

Product images are imported from the Google Cloud Storage bucket location as specified in the ‘ImageUri’ column in the CSV file.

Request

Make a request to import_product_sets to create a product set and products, add those products to the product set, and import the product reference images. Pass the file path of the CSV that has the list of image locations of all products as an argument.

Code

The import_product_sets() function takes in the following parameters, creates a product set and products, and then adds those products to the product set. The function then adds image references to the products in the product set.

  • projectId: Id of the project.
  • computeRegion: Region name.
  • gcs_uri: Product image file path (Google Cloud Storage). This csv file contains list of image locations of all products that is to be indexed.

Python

def import_product_sets(project_id, location, gcs_uri):
    """Import images of different products in the product set.
    Args:
        project_id: Id of the project.
        location: A compute region name.
        gcs_uri: Google Cloud Storage URI.
            Target files must be in Product Search CSV format.
    """
    client = vision.ProductSearchClient()

    # A resource that represents Google Cloud Platform location.
    location_path = client.location_path(
        project=project_id, location=location)

    # Set the input configuration along with Google Cloud Storage URI
    gcs_source = vision.types.ImportProductSetsGcsSource(
        csv_file_uri=gcs_uri)
    input_config = vision.types.ImportProductSetsInputConfig(
        gcs_source=gcs_source)

    # Import the product sets from the input URI.
    response = client.import_product_sets(
        parent=location_path, input_config=input_config)

    print('Processing operation name: {}'.format(response.operation.name))
    # synchronous check of operation status
    result = response.result()
    print('Processing done.')

    for i, status in enumerate(result.statuses):
        print('Status of processing line {} of the csv: {}'.format(
            i, status))
        # Check the status of reference image
        # `0` is the code for OK in google.rpc.Code.
        if status.code == 0:
            reference_image = result.reference_images[i]
            print(reference_image)
        else:
            print('Status code not OK: {}'.format(status.message))

Java

/**
 * Import images of different products in the product set.
 *
 * @param projectId - Id of the project.
 * @param computeRegion - Region name.
 * @param gcsUri - Google Cloud Storage URI.Target files must be in Product Search CSV format.
 * @throws Exception - on client errors.
 */
public static void importProductSets(String projectId, String computeRegion, String gcsUri)
    throws Exception {
  ProductSearchClient client = ProductSearchClient.create();

  // A resource that represents Google Cloud Platform location.
  LocationName projectLocation = LocationName.of(projectId, computeRegion);
  Builder gcsSource = ImportProductSetsGcsSource.newBuilder().setCsvFileUri(gcsUri);

  // Set the input configuration along with Google Cloud Storage URI
  ImportProductSetsInputConfig inputConfig =
      ImportProductSetsInputConfig.newBuilder().setGcsSource(gcsSource).build();

  // Import the product sets from the input URI.
  OperationFuture<ImportProductSetsResponse, BatchOperationMetadata> response =
      client.importProductSetsAsync(projectLocation, inputConfig);

  System.out.println(String.format("Processing operation name: %s", response.getName()));
  ImportProductSetsResponse results = response.get();
  System.out.println("Processing done.");
  System.out.println("Results of the processing:");

  for (int i = 0; i < results.getStatusesCount(); i++) {
    System.out.println(
        String.format("Status of processing line %s of the csv: %s", i, results.getStatuses(i)));
    // Check the status of reference image.
    if (results.getStatuses(i).getCode() == 0) {
      ReferenceImage referenceImage = results.getReferenceImages(i);
      System.out.println(referenceImage);
    } else {
      System.out.println("No reference image.");
    }
  }
}

Node.js

function importProductSets(projectId, location, gcsUri) {
  // Imports the Google Cloud client library
  const vision = require('@google-cloud/vision').v1p3beta1;
  // Creates a client
  const client = new vision.ProductSearchClient();

  /**
   * TODO(developer): Uncomment the following line before running the sample.
   */
  // const projectId = 'Your Google Cloud project Id';
  // const location = 'A compute region name';
  // const gcsUri = 'Google Cloud Storage path of the input image'';

  // A resource that represents Google Cloud Platform location.
  const projectLocation = client.locationPath(projectId, location);

  // Set the input configuration along with Google Cloud Storage URI
  const inputConfig = {
    gcsSource: {
      csvFileUri: gcsUri,
    },
  };

  // Import the product sets from the input URI.
  client
    .importProductSets({parent: projectLocation, inputConfig: inputConfig})
    .then(responses => {
      const response = responses[0];
      const operation = responses[1];
      console.log('Processing operation name: ', operation.name);

      // synchronous check of operation status
      return response.promise();
    })
    .then(responses => {
      console.log('Processing done.');
      console.log('Results of the processing:');

      for (const i in responses[0].statuses) {
        console.log(
          'Status of processing ',
          i,
          'of the csv:',
          responses[0].statuses[i]
        );

        // Check the status of reference image
        if (responses[0].statuses[i].code === 0) {
          console.log(responses[0].referenceImages[i]);
        } else {
          console.log('No reference image.');
        }
      }
    })
    .catch(err => {
      console.error(err);
    });

Response

Processing operation name: locations/us-east1/operations/bf639a4355989c9b Processing done.
Results of the processing:
....
Status of processing line 20 of the csv:
name: "projects/prj-prod-search-tutorials/locations/us-east1/products/p46916b401/referenceImages/46916b4070ba11e89dbed20059124800"
uri: "gs://product-search-tutorial/dress-shoe-dataset/46916b4070ba11e89dbed20059124800.jpg"

Status of processing line 21 of the csv:
name: "projects/prj-prod-search-tutorials/locations/us-east1/products/p4691aea11/referenceImages/4691aea170ba11e8a1b3d20059124800"
uri: "gs://product-search-tutorial/dress-shoe-dataset/4691aea170ba11e8a1b3d20059124800.jpg"
...

After indexing has automatically occurred (approximately every 30 minutes) you can then pass an image uploaded from a consumer to Vision Product Search to find a product id of an image from the catalog that closely matches the provided image.

Option 2: Online (Individual) Import

1. Create a Product Set

Create an empty Product Set which is a simple container for a group of products.

Request

Create an empty Product Set and name it "PS_CLOTH-SHOE_070318" by executing the following request using the method create_product_set(). Pass the product set id and display name as arguments.

Code

The create_product_set() function takes in the following parameters and returns the newly created product set object.

  • project_id: Id of the project.
  • compute_region: Region name.
  • product_set_id: Id of the product set.
  • product_set_display_name: Display name of the product set.

Python

from google.cloud import vision_v1p3beta1 as vision

def create_product_set(
        project_id, location, product_set_id, product_set_display_name):
    """Create a product set.
    Args:
        project_id: Id of the project.
        location: A compute region name.
        product_set_id: Id of the product set.
        product_set_display_name: Display name of the product set.
    """
    client = vision.ProductSearchClient()

    # A resource that represents Google Cloud Platform location.
    location_path = client.location_path(
        project=project_id, location=location)

    # Create a product set with the product set specification in the region.
    product_set = vision.types.ProductSet(
            display_name=product_set_display_name)

    # The response is the product set with `name` populated.
    response = client.create_product_set(
        parent=location_path,
        product_set=product_set,
        product_set_id=product_set_id)

    # Display the product set information.
    print('Product set name: {}'.format(response.name))

Java

/**
 * Create a product set
 *
 * @param projectId - Id of the project.
 * @param computeRegion - Region name.
 * @param productSetId - Id of the product set.
 * @param productSetDisplayName - Display name of the product set.
 * @throws IOException - on I/O errors.
 */
public static void createProductSet(
    String projectId, String computeRegion, String productSetId, String productSetDisplayName)
    throws IOException {
  ProductSearchClient client = ProductSearchClient.create();

  // A resource that represents Google Cloud Platform location.
  LocationName projectLocation = LocationName.of(projectId, computeRegion);

  // Create a product set with the product set specification in the region.
  ProductSet myProductSet = ProductSet.newBuilder().setDisplayName(productSetDisplayName).build();
  CreateProductSetRequest request =
      CreateProductSetRequest.newBuilder()
          .setParent(projectLocation.toString())
          .setProductSet(myProductSet)
          .setProductSetId(productSetId)
          .build();
  ProductSet productSet = client.createProductSet(request);

  // Display the product set information
  System.out.println(String.format("Product set name: %s", productSet.getName()));

}

Node.js

// Imports the Google Cloud client library
const vision = require('@google-cloud/vision').v1p3beta1;

// Creates a client
const client = new vision.ProductSearchClient();

/**
 * TODO(developer): Uncomment the following line before running the sample.
 */
// const projectId = 'Your Google Cloud project Id';
// const location = 'A compute region name';
// const productSetId = 'Id of the product set';
// const productSetDisplayName = 'Display name of the product set';

// Resource path that represents Google Cloud Platform location.
const locationPath = client.locationPath(projectId, location);

const productSet = {
  displayName: productSetDisplayName,
};

const request = {
  parent: locationPath,
  productSet: productSet,
  productSetId: productSetId,
};

client
  .createProductSet(request)
  .then(results => {
    // The response is the product set with the `name` field populated
    const createdProductSet = results[0];
    console.log(`Product Set name: ${createdProductSet.name}`);
  })
  .catch(err => {
    console.error('ERROR:', err);
  });

Response

Product set name: projects/prj-prod-search-tutorials/locations/us-east1/productSets/PS_CLOTH-SHOE_070318
Product set id: PS_CLOTH-SHOE_070318
Product set display name: CLOTH-SHOE

2. Create a Product

After a product set has been created, the next step is to create a product.

Request

Create a product by executing the following request.

Code

The create_product() function takes in the following parameters and returns the newly created product object.

  • project_id: Id of the project.
  • compute_region: Region name.
  • product_id: Id of the product.
  • display_name: Display name of the product.
  • product_category: Category of the product.
  • product_description: Description of the product.
  • product_labels: Labels of the product.

Python

from google.cloud import vision_v1p3beta1 as vision

def create_product(
        project_id, location, product_id, product_display_name,
        product_category):
    """Create one product.
    Args:
        project_id: Id of the project.
        location: A compute region name.
        product_id: Id of the product.
        product_display_name: Display name of the product.
        product_category: Category of the product.
    """
    client = vision.ProductSearchClient()

    # A resource that represents Google Cloud Platform location.
    location_path = client.location_path(project=project_id, location=location)

    # Create a product with the product specification in the region.
    # Set product name and product display name.
    product = vision.types.Product(
        display_name=product_display_name,
        product_category=product_category)

    # The response is the product with the `name` field populated.
    response = client.create_product(
        parent=location_path,
        product=product,
        product_id=product_id)

    # Display the product information.
    print('Product name: {}'.format(response.name))

Java

/**
 * Create one product.
 *
 * @param projectId - Id of the project.
 * @param computeRegion - Region name.
 * @param productId - Id of the product.
 * @param productDisplayName - Display name of the product.
 * @param productCategory - Category of the product.
 * @throws IOException - on I/O errors.
 */
public static void createProduct(
    String projectId,
    String computeRegion,
    String productId,
    String productDisplayName,
    String productCategory)
    throws IOException {
  ProductSearchClient client = ProductSearchClient.create();

  // A resource that represents Google Cloud Platform location.
  LocationName projectLocation = LocationName.of(projectId, computeRegion);

  // Create a product with the product specification in the region.
  // Multiple labels are also supported.
  Product myProduct =
      Product.newBuilder()
          .setName(productId)
          .setDisplayName(productDisplayName)
          .setProductCategory(productCategory)
          .build();
  Product product = client.createProduct(projectLocation.toString(), myProduct, productId);

  // Display the product information
  System.out.println(String.format("Product name: %s", product.getName()));
}

Node.js

// Imports the Google Cloud client library
const vision = require('@google-cloud/vision').v1p3beta1;

// Creates a client
const client = new vision.ProductSearchClient();

/**
 * TODO(developer): Uncomment the following line before running the sample.
 */
// const projectId = 'Your Google Cloud project Id';
// const location = 'A compute region name';
// const productId = 'Id of the product';
// const productDisplayName = 'Display name of the product';
// const productCategory = 'Catoegory of the product';

// Resource path that represents Google Cloud Platform location.
const locationPath = client.locationPath(projectId, location);

const product = {
  displayName: productDisplayName,
  productCategory: productCategory,
};

const request = {
  parent: locationPath,
  product: product,
  productId: productId,
};

client
  .createProduct(request)
  .then(results => {
    // The response is the product with the `name` field populated
    const createdProduct = results[0];
    console.log(`Product name: ${createdProduct.name}`);
  })
  .catch(err => {
    console.error('ERROR:', err);
  });

Response

Product name: projects/prj-prod-search-tutorials/locations/us-east1/products/P_CLOTH-SHOE_46903668_070318
Product id: P_CLOTH-SHOE_46903668_070318
Product display name: Blue Dress
Product category: apparel
Product description: Short sleeved and 1950s style satin dress
Product labels:
  Product label 1:
        key: style
        value: women
  Product label 2:
        key: category
        value: dress
  Product label 3:
        key: color
        value: dark-blue

3. Add a Product to a Product Set

After a product set and a product have been created, you can add the product to the product set.

Request

Add a product to a product set by executing the following request with the method add_product_to_product_set. Pass the product id and product set id as arguments.

Code

The add_product_to_product_set() function takes in the following parameters and add the product to the product set.

  • project_id: Id of the project.
  • compute_region: Region name.
  • product_id: Id of the product
  • product_set_id: Id of the product set.

Python

from google.cloud import vision_v1p3beta1 as vision

def add_product_to_product_set(
        project_id, location, product_id, product_set_id):
    """Add a product to a product set.
    Args:
        project_id: Id of the project.
        location: A compute region name.
        product_id: Id of the product.
        product_set_id: Id of the product set.
    """
    client = vision.ProductSearchClient()

    # Get the full path of the product set.
    product_set_path = client.product_set_path(
        project=project_id, location=location,
        product_set=product_set_id)

    # Get the full path of the product.
    product_path = client.product_path(
        project=project_id, location=location, product=product_id)

    # Add the product to the product set.
    client.add_product_to_product_set(
        name=product_set_path, product=product_path)
    print('Product added to product set.')

Java

/**
 * Add a product to a product set.
 *
 * @param projectId - Id of the project.
 * @param computeRegion - Region name.
 * @param productId - Id of the product.
 * @param productSetId - Id of the product set.
 * @throws IOException - on I/O errors.
 */
public static void addProductToProductSet(
    String projectId, String computeRegion, String productId, String productSetId)
    throws IOException {
  ProductSearchClient client = ProductSearchClient.create();

  // Get the full path of the product set.
  ProductSetName productSetPath = ProductSetName.of(projectId, computeRegion, productSetId);

  // Get the full path of the product.
  String productPath = ProductName.of(projectId, computeRegion, productId).toString();

  // Add the product to the product set.
  client.addProductToProductSet(productSetPath, productPath);

  System.out.println(String.format("Product added to product set."));
}

Node.js

const vision = require('@google-cloud/vision').v1p3beta1;

const client = new vision.ProductSearchClient();

/**
 * TODO(developer): Uncomment the following line before running the sample.
 */
// const projectId = 'Your Google Cloud project Id';
// const location = 'A compute region name';
// const productId = 'Id of the product';
// const productSetId = 'Id of the product set';

const productPath = client.productPath(projectId, location, productId);
const productSetPath = client.productSetPath(
  projectId,
  location,
  productSetId
);

const request = {
  name: productSetPath,
  product: productPath,
};

client
  .addProductToProductSet(request)
  .then(() => {
    console.log(`Product added to product set.`);
  })
  .catch(err => {
    console.error(err);
  });

Response

Product added to product set.

4. Update product

If you need to update a product or product set after it has been created, you can use our update methods. Here, we provide an example of a product update where the labels are changed.

Request

Update product labels by executing the following request with the method update_product_labels. Pass the product id and product labels as arguments.

Code

The update_product_labels() function takes in the following parameters and returns the updated product object.

  • project_id: Id of the project.
  • compute_region: Region name.
  • product_id: Id of the product.
  • product_labels: Labels of the product.

Python

from google.cloud import vision_v1p3beta1 as vision

def update_product_labels(
        project_id, location, product_id, key, value):
    """Update the product labels.
    Args:
        project_id: Id of the project.
        location: A compute region name.
        product_id: Id of the product.
        key: The key of the label.
        value: The value of the label.
    """
    client = vision.ProductSearchClient()

    # Get the name of the product.
    product_path = client.product_path(
        project=project_id, location=location, product=product_id)

    # Set product name, product label and product display name.
    # Multiple labels are also supported.
    key_value = vision.types.Product.KeyValue(key=key, value=value)
    product = vision.types.Product(
        name=product_path,
        product_labels=[key_value])

    # Updating only the product_labels field here.
    update_mask = vision.types.FieldMask(paths=['product_labels'])

    # This overwrites the product_labels.
    updated_product = client.update_product(
        product=product, update_mask=update_mask)

    # Display the updated product information.
    print('Product name: {}'.format(updated_product.name))
    print('Updated product labels: {}'.format(product.product_labels))

Java

/**
 * Update the product labels.
 *
 * @param projectId - Id of the project.
 * @param computeRegion - Region name.
 * @param productId -Id of the product.
 * @param productLabels - Labels of the product.
 * @throws IOException - on I/O errors.
 */
public static void updateProductLabels(
    String projectId, String computeRegion, String productId, String productLabels)
    throws IOException {
  ProductSearchClient client = ProductSearchClient.create();

  // Get the full path of the product.
  String productPath = ProductName.of(projectId, computeRegion, productId).toString();

  // Set product name, product labels and product display name.
  // Multiple labels are also supported.
  Product product =
      Product.newBuilder()
          .setName(productPath)
          .addProductLabels(
              KeyValue.newBuilder()
                  .setKey(productLabels.split(",")[0].split("=")[0])
                  .setValue(productLabels.split(",")[0].split("=")[1])
                  .build())
          .build();

  // Set product update field name.
  FieldMask updateMask = FieldMask.newBuilder().addPaths("product_labels").build();

  // Update the product.
  Product updatedProduct = client.updateProduct(product, updateMask);

  // Display the product information
  System.out.println(String.format("Product name: %s", updatedProduct.getName()));
  System.out.println(String.format("Updated product labels: "));
  for (Product.KeyValue element : updatedProduct.getProductLabelsList()) {
    System.out.println(String.format("%s: %s", element.getKey(), element.getValue()));
  }
}

Node.js

// Imports the Google Cloud client library
const vision = require('@google-cloud/vision').v1p3beta1;

// Creates a client
const client = new vision.ProductSearchClient();

/**
 * TODO(developer): Uncomment the following line before running the sample.
 */
// const projectId = 'Your Google Cloud project Id';
// const location = 'A compute region name';
// const productId = 'Id of the product';
// const key = 'The key of the label';
// const value = 'The value of the label';

// Resource path that represents full path to the product.
const productPath = client.productPath(projectId, location, productId);

const product = {
  name: productPath,
  productLabels: [
    {
      key: key,
      value: value,
    },
  ],
};

const updateMask = {
  paths: ['product_labels'],
};

const request = {
  product: product,
  updateMask: updateMask,
};

client
  .updateProduct(request)
  .then(results => {
    const product = results[0];
    console.log(`Product name: ${product.name}`);
    console.log(`Product display name: ${product.displayName}`);
    console.log(`Product description: ${product.description}`);
    console.log(`Product category: ${product.productCategory}`);
    console.log(
      `Product Labels: ${product.productLabels[0].key}: ${
        product.productLabels[0].value
      }`
    );
  })
  .catch(err => {
    console.error('ERROR:', err);
  });

Response

Product name: projects/prj-prod-search-tutorials/locations/us-east1/products/P_CLOTH-SHOE_46903668_070318
Product id: P_CLOTH-SHOE_46903668_070318
Product display name: Blue Dress
Updated product labels:
  Product label 1:
        key: style
        value: women
  Product label 2:
        key: category
        value: dress
  Product label 3:
        key: color
        value: blue
Product description: Short sleeved and 1950s style satin dress

5. Create Reference Image

Creating a reference image for an individual product allows Vision Product Search to search for the product by this image after it is indexed. You can have multiple reference images in a product, particularly if you desire a better match quality.

You can add a new reference image to a product at any time.

When creating a reference image you have the option to include bounding poly coordinates. A bounding poly identifies an area of interest in the reference image. For example, if you create a reference image for a product that is a jacket, you can provide the coordinates for the jacket in the bounding poly argument, and the system will only consider the jacket when looking for product matches. Note: You can provide multiple bounding polys at index time, though at query time the API supports only a single bounding poly.

A convenient way to get the bounding poly coordinates for an image is to use Vision API object localization. For more information on object localization, see Detecting Multiple Objects.

Request

Create a single reference image by executing the following request with the method create_image. Pass the product id, image id, Google Cloud Storage URI and bounding poly as arguments.

Code

The create_reference_image() function takes in the following parameters and returns the newly created image reference object.

  • project_id: Id of the project.
  • compute_region: Region name.
  • product_id: Id of the product.
  • reference_image_id: Id of the reference image.
  • gcs_uri: Google Cloud Storage path of the input image.
  • bounding_polys: Bounding polygon for image annotation.

Python

from google.cloud import vision_v1p3beta1 as vision

def create_reference_image(
        project_id, location, product_id, reference_image_id, gcs_uri):
    """Create a reference image.
    Args:
        project_id: Id of the project.
        location: A compute region name.
        product_id: Id of the product.
        reference_image_id: Id of the reference image.
        gcs_uri: Google Cloud Storage path of the input image.
    """
    client = vision.ProductSearchClient()

    # Get the full path of the product.
    product_path = client.product_path(
        project=project_id, location=location, product=product_id)

    # Create a reference image.
    reference_image = vision.types.ReferenceImage(uri=gcs_uri)

    # The response is the reference image with `name` populated.
    image = client.create_reference_image(
        parent=product_path,
        reference_image=reference_image,
        reference_image_id=reference_image_id)

    # Display the reference image information.
    print('Reference image name: {}'.format(image.name))
    print('Reference image uri: {}'.format(image.uri))

Java

/**
 * Create a reference image.
 *
 * @param projectId - Id of the project.
 * @param computeRegion - Region name.
 * @param productId - Id of the product.
 * @param referenceImageId - Id of the image.
 * @param gcsUri - Google Cloud Storage path of the input image.
 * @throws IOException - on I/O errors.
 */
public static void createReferenceImage(
    String projectId,
    String computeRegion,
    String productId,
    String referenceImageId,
    String gcsUri)
    throws IOException {
  ProductSearchClient client = ProductSearchClient.create();

  // Get the full path of the product.
  ProductName productPath = ProductName.of(projectId, computeRegion, productId);

  // Create a reference image.
  ReferenceImage referenceImage = ReferenceImage.newBuilder().setUri(gcsUri).build();
  ReferenceImage image =
      client.createReferenceImage(productPath, referenceImage, referenceImageId);

  // Display the reference image information.
  System.out.println(String.format("Reference image name: %s", image.getName()));
  System.out.println(String.format("Reference image uri: %s", image.getUri()));
}

Node.js

const vision = require('@google-cloud/vision').v1p3beta1;

const client = new vision.ProductSearchClient();

/**
 * TODO(developer): Uncomment the following line before running the sample.
 */
// const projectId = 'Your Google Cloud project Id';
// const location = 'A compute region name';
// const productId = 'Id of the product';
// const referenceImageId = 'Id of the reference image';
// const gcsUri = 'Google Cloud Storage path of the input image';

const formattedParent = client.productPath(projectId, location, productId);

const referenceImage = {
  uri: gcsUri,
};

const request = {
  parent: formattedParent,
  referenceImage: referenceImage,
  referenceImageId: referenceImageId,
};

client
  .createReferenceImage(request)
  .then(responses => {
    const response = responses[0];
    console.log(`response.name: ${response.name}`);
    console.log(`response.uri: ${response.uri}`);
  })
  .catch(err => {
    console.error(err);
  });

Response

Reference image name: projects/prj-prod-search-tutorials/locations/us-east1/products/P_CLOTH-SHOE_46903668_070318/referenceImages/I_469a896b70ba11e8be97d20059124800_070418
Reference image id: I_469a896b70ba11e8be97d20059124800_070418
Reference image uri: gs://product-search-tutorial/dress-shoe-dataset/469a896b70ba11e8be97d20059124800.jpg
Reference image bounding polygons:
vertices {
  x: 80
  y: 50
}
vertices {
  x: 80
  y: 660
}
vertices {
  x: 300
  y: 50
}
vertices {
  x: 430
  y: 660
}

Step 2: Search For Matching Products

This interface allows you to query the product catalog you created by taking a new image as input and searching for the best matching product.

Similar to creating a reference image, when searching for matching images you have the option to include bounding poly coordinates. A bounding poly identifies the area of interest in the source image that you want to find matches for. For example, if your source image contains both a dress and a purse, and you only want to find matches for the dress, you can identify the bounding poly coordinates for the region of the picture that contains only the dress. By default, if no bounding poly is specified the API determines the largest bounding poly and queries it automatically.

A convenient way to get the bounding poly coordinates for an image is to use Vision API object localization. For more information on object localization, see Detecting Multiple Objects.

Request

Search for matching products from the image catalog by executing the following request with the method get_similar_product. Pass the product set id, product category, image file path (or URI), empty filter and bounding poly (optional) as arguments.

Code

The get_similar_products_file() function takes in the following parameters and returns an API response that includes the best matching product for an image with the score and the matching image. This image is returned using the highest confidence value.

  • project_id: Id of the project.
  • compute_region: Region name.
  • product_set_id: Id of the product set.
  • product_category: Category of the product.
  • file_path: Local file path of the image to be searched.
  • filter_str: Condition to be applied on the labels.
  • bounding_polys: Bounding polygon for image annotation.

Python

from google.cloud import vision_v1p3beta1 as vision

def get_similar_products_file(
        project_id, location, product_set_id, product_category,
        file_path, filter):
    """Search similar products to image.
    Args:
        project_id: Id of the project.
        location: A compute region name.
        product_set_id: Id of the product set.
        product_category: Category of the product.
        file_path: Local file path of the image to be searched.
        filter: Condition to be applied on the labels.
        Example for filter: (color = red OR color = blue) AND style = kids
        It will search on all products with the following labels:
        color:red AND style:kids
        color:blue AND style:kids
    """
    # product_search_client is needed only for its helper methods.
    product_search_client = vision.ProductSearchClient()
    image_annotator_client = vision.ImageAnnotatorClient()

    # Read the image as a stream of bytes.
    with open(file_path, 'rb') as image_file:
        content = image_file.read()

    # Create annotate image request along with product search feature.
    image = vision.types.Image(content=content)

    # product search specific parameters
    product_set_path = product_search_client.product_set_path(
        project=project_id, location=location,
        product_set=product_set_id)
    product_search_params = vision.types.ProductSearchParams(
        product_set=product_set_path,
        product_categories=[product_category],
        filter=filter)
    image_context = vision.types.ImageContext(
        product_search_params=product_search_params)

    # Search products similar to the image.
    response = image_annotator_client.product_search(
        image, image_context=image_context)

    index_time = response.product_search_results.index_time
    print('Product set index time:')
    print('  seconds: {}'.format(index_time.seconds))
    print('  nanos: {}\n'.format(index_time.nanos))

    results = response.product_search_results.results

    print('Search results:')
    for result in results:
        product = result.product

        print('Score(Confidence): {}'.format(result.score))
        print('Image name: {}'.format(result.image))

        print('Product name: {}'.format(product.name))
        print('Product display name: {}'.format(
            product.display_name))
        print('Product description: {}\n'.format(product.description))
        print('Product labels: {}\n'.format(product.product_labels))

Java

/**
 * Search similar products to image in local file.
 *
 * @param projectId - Id of the project.
 * @param computeRegion - Region name.
 * @param productSetId - Id of the product set.
 * @param productCategory - Category of the product.
 * @param filePath - Local file path of the image to be searched
 * @param filter - Condition to be applied on the labels. Example for filter: (color = red OR
 *     color = blue) AND style = kids It will search on all products with the following labels:
 *     color:red AND style:kids color:blue AND style:kids
 * @throws IOException - on I/O errors.
 */
public static void getSimilarProductsFile(
    String projectId,
    String computeRegion,
    String productSetId,
    String productCategory,
    String filePath,
    String filter)
    throws IOException {
  ImageAnnotatorClient queryImageClient = ImageAnnotatorClient.create();

  // Get the full path of the product set.
  String productSetPath = ProductSetName.of(projectId, computeRegion, productSetId).toString();

  // Read the image as a stream of bytes.
  File imgPath = new File(filePath);
  byte[] content = Files.readAllBytes(imgPath.toPath());

  // Create annotate image request along with product search feature.
  Feature featuresElement = Feature.newBuilder().setType(Type.PRODUCT_SEARCH).build();
  // The input image can be a HTTPS link or Raw image bytes.
  // Example:
  // To use HTTP link replace with below code
  //  ImageSource source = ImageSource.newBuilder().setImageUri(imageUri).build();
  //  Image image = Image.newBuilder().setSource(source).build();
  Image image = Image.newBuilder().setContent(ByteString.copyFrom(content)).build();
  ImageContext imageContext =
      ImageContext.newBuilder()
          .setProductSearchParams(
              ProductSearchParams.newBuilder()
                  .setProductSet(productSetPath)
                  .addProductCategories(productCategory)
                  .setFilter(filter))
          .build();

  AnnotateImageRequest annotateImageRequest =
      AnnotateImageRequest.newBuilder()
          .addFeatures(featuresElement)
          .setImage(image)
          .setImageContext(imageContext)
          .build();
  List<AnnotateImageRequest> requests = Arrays.asList(annotateImageRequest);

  // Search products similar to the image.
  BatchAnnotateImagesResponse response = queryImageClient.batchAnnotateImages(requests);

  List<Result> similarProducts =
      response.getResponses(0).getProductSearchResults().getResultsList();

  System.out.println("Similar Products: ");
  for (Result product : similarProducts) {
    System.out.println(String.format("\nProduct name: %s", product.getProduct().getName()));
    System.out.println(
        String.format("Product display name: %s", product.getProduct().getDisplayName()));
    System.out.println(
        String.format("Product description: %s", product.getProduct().getDescription()));
    System.out.println(String.format("Score(Confidence): %s", product.getScore()));
    System.out.println(String.format("Image name: %s", product.getImage()));
  }
}

Node.js

function getSimilarProductsFile(
  projectId,
  location,
  productSetId,
  productCategory,
  filePath,
  filter
) {
  // Imports the Google Cloud client library
  const vision = require('@google-cloud/vision').v1p3beta1;
  const fs = require('fs');
  // Creates a client
  const productSearchClient = new vision.ProductSearchClient();
  const imageAnnotatorClient = new vision.ImageAnnotatorClient();

  /**
   * TODO(developer): Uncomment the following line before running the sample.
   */
  // const projectId = 'java-docs-samples-testing';
  // const location = 'us-west1';
  // const productSetId = 'indexed_product_set_id_for_testing';
  // const productCategory = 'apparel';
  // const filePath = './resources/shoes_1.jpg';
  // const filter = '';
  const productSetPath = productSearchClient.productSetPath(
    projectId,
    location,
    productSetId
  );
  const content = fs.readFileSync(filePath, 'base64');
  const request = {
    // The input image can be a GCS link or HTTPS link or Raw image bytes.
    // Example:
    // To use GCS link replace with below code
    // image: {source: {gcsImageUri: filePath}}
    // To use HTTP link replace with below code
    // image: {source: {imageUri: filePath}}
    image: {content: content},
    features: [{type: 'PRODUCT_SEARCH'}],
    imageContext: {
      productSearchParams: {
        productSet: productSetPath,
        productCategories: [productCategory],
        filter: filter,
      },
    },
  };
  imageAnnotatorClient
    .batchAnnotateImages({requests: [request]})
    .then(responses => {
      const response = responses[0];

      console.log('Search Image:', filePath);

      const results =
        response['responses'][0]['productSearchResults']['results'];
      console.log('\nSimilar product information:');
      results.forEach(result => {
        console.log('Product id:', result['product'].name.split('/').pop(-1));
        console.log('Product display name:', result['product'].displayName);
        console.log('Product description:', result['product'].description);
        console.log('Product category:', result['product'].productCategory);
      });
    })
    .catch(err => {
      console.error(err);
    });
}

Apparel Response Example

Search Image:
  D:/product/final/images-20180618T073733Z-01/images/469355b570ba11e88ff2d20059124800.jpg

Image search result

Similar product information:
 Product id: 46930b6b
 Product display name: Evening gown
 Product description: Blue evening gown in 1940s style
 Product category: apparel
 style: women
 category: dress
 color: blue

Searching with a Label

The following search example includes a filter based on color.

Request

Make a search request by executing the following request with method get_similar_products_file() or get_similar_products_uri(). Product set id, local image file path and filter are passed as arguments. This input image is also present in “resources/input/”.

Python

python product_search.py get_similar_products_file "12000002" "D:/product/final/images-20180618T073733Z-001/images/469355b570ba11e88ff2d20059124800.jpg" "color=white"

Response

Search Image:
  D:/product/final/images-20180618T073733Z-001/images/469355b570ba11e88ff2d20059124800.jpg

Image search result 2

Similar product information:
 Product id: p569d4e7a1
 Product display name: Wedding Dress
 Product description: Elegant Wedding Dress for women
 Product category: apparel
 style: women
 category: dress
 color: white

Was this page helpful? Let us know how we did:

Send feedback about...

Cloud Vision API
Need help? Visit our support page.