Generate and search multimodal embeddings

This tutorial guides you through the end-to-end process of creating multimodal embeddings for images and text, and then performing a cross-modality text-to-image search.

This tutorial covers the following tasks:

• Creating a BigQuery object table over image data in a Cloud Storage bucket.
• Exploring the image data by using a Colab Enterprise notebook in BigQuery.
• Creating a BigQuery ML remote model that targets the Vertex AI multimodalembedding foundation model.
• Using the remote model with the ML.GENERATE_EMBEDDING function to generate embeddings from the images in the object table.
• Correct any embedding generation errors.
• Optionally, creating a vector index to index the image embeddings.
• Creating a text embedding for a given search string.
• Using the VECTOR_SEARCH function to search for image embeddings that are similar to the text embedding.
• Visualizing the results by using a notebook.

This tutorial uses the public domain art images from The Metropolitan Museum of Art that are available in the public Cloud Storage gcs-public-data--met bucket.

Required permissions

To run this tutorial, you need the following Identity and Access Management (IAM) permissions:

• To create a connection, you need membership in the BigQuery Connection Admin (roles/bigquery.connectionAdmin) role.

• To grant permissions to the connection's service account, you need the resourcemanager.projects.setIamPolicy permission.

• To create and run notebooks, you need the following IAM permissions:

  • resourcemanager.projects.get
  • resourcemanager.projects.list
  • bigquery.config.get
  • bigquery.jobs.create
  • bigquery.readsessions.create
  • bigquery.readsessions.getData
  • bigquery.readsessions.update
  • resourcemanager.projects.get
  • resourcemanager.projects.list
  • dataform.locations.get
  • dataform.locations.list

  • dataform.repositories.create

    Note: Users who have the dataform.repositories.create permission can execute code using the default Dataform service account and all permissions granted to that service account. For more information, see Security considerations for Dataform permissions.

  • dataform.repositories.list

  • dataform.collections.create

  • dataform.collections.list

  • aiplatform.notebookRuntimeTemplates.apply

  • aiplatform.notebookRuntimeTemplates.get

  • aiplatform.notebookRuntimeTemplates.list

  • aiplatform.notebookRuntimeTemplates.getIamPolicy

  • aiplatform.notebookRuntimes.assign

  • aiplatform.notebookRuntimes.get

  • aiplatform.notebookRuntimes.list

  • aiplatform.operations.list

  You can get these permissions from the following IAM roles:

  • BigQuery Read Session User (roles/bigquery.readSessionUser)
  • BigQuery Studio User (roles/bigquery.studioUser)

• The IAM permissions needed in this tutorial for the remaining BigQuery operations are included in the following two roles:

  • BigQuery Data Editor (roles/bigquery.dataEditor) to create models, tables, and indexes.
  • BigQuery User (roles/bigquery.user) to run BigQuery jobs.

Costs

In this document, you use the following billable components of Google Cloud:

• BigQuery ML: You incur costs for the data that you process in BigQuery.
• Vertex AI: You incur costs for calls to the Vertex AI service that's represented by the remote model.

To generate a cost estimate based on your projected usage, use the pricing calculator. New Google Cloud users might be eligible for a free trial.

For more information about BigQuery pricing, see BigQuery pricing in the BigQuery documentation.

For more information about Vertex AI pricing, see the Vertex AI pricing page.

Before you begin

1. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

   Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

   Go to project selector

2. Make sure that billing is enabled for your Google Cloud project.

3. Enable the BigQuery, BigQuery Connection, and Vertex AI APIs.

   Enable the APIs

4. Enable BigQuery Studio.

Create a dataset

Create a BigQuery dataset to store your ML model:

1. In the Google Cloud console, go to the BigQuery page.

   Go to the BigQuery page

2. In the Explorer pane, click your project name.

3. Click View actions > Create dataset.

   

4. On the Create dataset page, do the following:

   • For Dataset ID, enter bqml_tutorial.

   • For Location type, select Multi-region, and then select US (multiple regions in United States).

     The public datasets are stored in the US multi-region. For simplicity, store your dataset in the same location.

   • Leave the remaining default settings as they are, and click Create dataset.

     

Create a connection

Create a Cloud resource connection and get the connection's service account ID. Create the connection in the same location as the dataset that you created in the previous step.

Select one of the following options:

Console

1. Go to the BigQuery page.

   Go to BigQuery

2. To create a connection, click Add, and then click Connections to external data sources.

3. In the Connection type list, select Vertex AI remote models, remote functions and BigLake (Cloud Resource).

4. In the Connection ID field, enter a name for your connection.

5. Click Create connection.

6. Click Go to connection.

7. In the Connection info pane, copy the service account ID for use in a later step.

bq

1. In a command-line environment, create a connection:

   bq mk --connection --location=REGION --project_id=PROJECT_ID \
       --connection_type=CLOUD_RESOURCE CONNECTION_ID

   The --project_id parameter overrides the default project.

   Replace the following:

   • REGION: your connection region
   • PROJECT_ID: your Google Cloud project ID
   • CONNECTION_ID: an ID for your connection

   When you create a connection resource, BigQuery creates a unique system service account and associates it with the connection.

   Troubleshooting: If you get the following connection error, update the Google Cloud SDK:

   Flags parsing error: flag --connection_type=CLOUD_RESOURCE: value should be one of...
   

2. Retrieve and copy the service account ID for use in a later step:

   bq show --connection PROJECT_ID.REGION.CONNECTION_ID

   The output is similar to the following:

   name                          properties
   1234.REGION.CONNECTION_ID     {"serviceAccountId": "[e-mail]"}
   

Terraform

Append the following section into your main.tf file.

 ## This creates a cloud resource connection.
 ## Note: The cloud resource nested object has only one output only field - serviceAccountId.
 resource "google_bigquery_connection" "connection" {
    connection_id = "CONNECTION_ID"
    project = "PROJECT_ID"
    location = "REGION"
    cloud_resource {}
}        

Replace the following:

• CONNECTION_ID: an ID for your connection
• PROJECT_ID: your Google Cloud project ID
• REGION: your connection region

Grant permissions to the connection's service account

Grant the connection's service account the appropriate roles to access the Cloud Storage and Vertex AI services. You must grant these roles in the same project you created or selected in the Before you begin section. Granting the roles in a different project results in the error [e-mail] does not have the permission to access resource.

To grant the appropriate roles, follow these steps:

1. Go to the IAM & Admin page.

   Go to IAM & Admin

2. Click Grant Access.

3. In the New principals field, enter the service account ID that you copied earlier.

4. In the Select a role field, choose Vertex AI, and then select Vertex AI User.

5. Click Add another role.

6. In the Select a role field, choose Cloud Storage, and then select Storage Object Viewer.

7. Click Save.

Create the object table

Create an object table over the art images in the public Cloud Storage gcs-public-data--met bucket. The object table makes it possible to analyze the images without moving them from Cloud Storage.

1. In the Google Cloud console, go to the BigQuery page.

   Go to BigQuery

2. In the query editor, run the following query:

   CREATE OR REPLACE EXTERNAL TABLE `bqml_tutorial.met_images`
   WITH CONNECTION `LOCATION.CONNECTION_ID`
   OPTIONS
     ( object_metadata = 'SIMPLE',
       uris = ['gs://gcs-public-data--met/*']
     );

   Replace the following:

   • LOCATION: the connection location.
   • CONNECTION_ID: the ID of your BigQuery connection.

     When you view the connection details in the Google Cloud console, this is the value in the last section of the fully qualified connection ID that is shown in Connection ID, for example projects/myproject/locations/connection_location/connections/myconnection.

Explore the image data

Create a Colab Enterprise notebook in BigQuery to explore the image data.

1. In the Google Cloud console, go to the BigQuery page.

   Go to BigQuery

2. Create a notebook by using the BigQuery editor.

3. Connect the notebook to the default runtime.

4. Set up the notebook:

   1. Add a code cell to the notebook.

   2. Copy and paste the following code into the code cell:

      #@title Set up credentials
      
      from google.colab import auth
      auth.authenticate_user()
      print('Authenticated')
      
      PROJECT_ID='PROJECT_ID'
      from google.cloud import bigquery
      client = bigquery.Client(PROJECT_ID)
      

      Replace PROJECT_ID with the name of the project that you are using for this tutorial.

   3. Run the code cell.

5. Enable table display:

   1. Add a code cell to the notebook.

   2. Copy and paste the following code into the code cell:

      #@title Enable data table display
      %load_ext google.colab.data_table
      

   3. Run the code cell.

6. Create a function to display the images:

   1. Add a code cell to the notebook.

   2. Copy and paste the following code into the code cell:

      #@title Util function to display images
      import io
      from PIL import Image
      import matplotlib.pyplot as plt
      import tensorflow as tf
      
      def printImages(results):
       image_results_list = list(results)
       amt_of_images = len(image_results_list)
      
       fig, axes = plt.subplots(nrows=amt_of_images, ncols=2, figsize=(20, 20))
       fig.tight_layout()
       fig.subplots_adjust(hspace=0.5)
       for i in range(amt_of_images):
         gcs_uri = image_results_list[i][0]
         text = image_results_list[i][1]
         f = tf.io.gfile.GFile(gcs_uri, 'rb')
         stream = io.BytesIO(f.read())
         img = Image.open(stream)
         axes[i, 0].axis('off')
         axes[i, 0].imshow(img)
         axes[i, 1].axis('off')
         axes[i, 1].text(0, 0, text, fontsize=10)
       plt.show()
      

   3. Run the code cell.

7. Display the images:

   1. Add a code cell to the notebook.

   2. Copy and paste the following code into the code cell:

      #@title Display Met images
      
      inspect_obj_table_query = """
      SELECT uri, content_type
      FROM bqml_tutorial.met_images
      WHERE content_type = 'image/jpeg'
      Order by uri
      LIMIT 10;
      """
      printImages(client.query(inspect_obj_table_query))
      

   3. Run the code cell.

      The results should look similar to the following:

      

8. Save the notebook as met-image-analysis.

Create the remote model

Create a remote model that represents a hosted Vertex AI multimodal embedding model:

1. In the Google Cloud console, go to the BigQuery page.

   Go to BigQuery

2. In the query editor, run the following query:

   CREATE OR REPLACE MODEL `bqml_tutorial.multimodal_embedding_model`
     REMOTE WITH CONNECTION `LOCATION.CONNECTION_ID`
     OPTIONS (ENDPOINT = 'multimodalembedding@001');

   Replace the following:

   • LOCATION: the connection location.
   • CONNECTION_ID: the ID of your BigQuery connection.

     When you view the connection details in the Google Cloud console, this is the value in the last section of the fully qualified connection ID that is shown in Connection ID, for example projects/myproject/locations/connection_location/connections/myconnection.

   The query takes several seconds to complete, after which the multimodal_embedding_model model appears in the bqml_tutorial dataset in the Explorer pane. Because the query uses a CREATE MODEL statement to create a model, there are no query results.

Generate image embeddings

Generate embeddings from the images in the object table by using the ML.GENERATE_EMBEDDING function, and then write them to a table for use in a following step. Embedding generation is an expensive operation, so the query uses a subquery including the LIMIT clause to limit embedding generation to 10,000 images instead of embedding the full dataset of 601,294 images. This also helps keep the number of images under the 25,000 limit for the ML.GENERATE_EMBEDDING function. This query takes approximately 40 minutes to run.

1. In the Google Cloud console, go to the BigQuery page.

   Go to BigQuery

2. In the query editor, run the following query:

   CREATE OR REPLACE TABLE `bqml_tutorial.met_image_embeddings`
   AS
   SELECT *
   FROM
     ML.GENERATE_EMBEDDING(
       MODEL `bqml_tutorial.multimodal_embedding_model`,
       (SELECT * FROM `bqml_tutorial.met_images` WHERE content_type = 'image/jpeg' LIMIT 10000))

Correct any embedding generation errors

Check for and correct any embedding generation errors. Embedding generation can fail because of Generative AI on Vertex AI quotas or service unavailability.

The ML.GENERATE_EMBEDDING function returns error details in the ml_generate_embedding_status column. This column is empty if embedding generation was successful, or contains an error message if embedding generation failed.

1. In the Google Cloud console, go to the BigQuery page.

   Go to BigQuery

2. In the query editor, run the following query to see if there were any embedding generation failures:

   SELECT DISTINCT(ml_generate_embedding_status),
     COUNT(uri) AS num_rows
   FROM bqml_tutorial.met_image_embeddings
   GROUP BY 1;

3. If rows with errors are returned, drop any rows where embedding generation failed:

   DELETE FROM `bqml_tutorial.met_image_embeddings`
   WHERE ml_generate_embedding_status = 'A retryable error occurred: RESOURCE_EXHAUSTED error from remote service/endpoint.';

Create a vector index

You can optionally use the CREATE VECTOR INDEX statement to create the met_images_index vector index on the ml_generate_embedding_result column of the met_images_embeddings table. A vector index lets you perform a vector search more quickly, with the trade-off of reducing recall and so returning more approximate results.

1. In the Google Cloud console, go to the BigQuery page.

   Go to BigQuery

2. In the query editor, run the following query:

   CREATE OR REPLACE
     VECTOR INDEX `met_images_index`
   ON
     bqml_tutorial.met_image_embeddings(ml_generate_embedding_result)
     OPTIONS (
       index_type = 'IVF',
       distance_type = 'COSINE');

3. The vector index is created asynchronously. To check if the vector index has been created, query the INFORMATION_SCHEMA.VECTOR_INDEXES view and confirm that the coverage_percentage value is greater than 0, and the last_refresh_time value isn't NULL:

   SELECT table_name, index_name, index_status,
     coverage_percentage, last_refresh_time, disable_reason
   FROM bqml_tutorial.INFORMATION_SCHEMA.VECTOR_INDEXES
   WHERE index_name = 'met_images_index';

Generate an embedding for the search text

To search images that correspond to a specified text search string, you must first create a text embedding for that string. Use the same remote model to create the text embedding that you used to create the image embeddings, and then write the text embedding to a table for use in a following step. The search string is pictures of white or cream colored dress from victorian era.

1. In the Google Cloud console, go to the BigQuery page.

   Go to BigQuery

2. In the query editor, run the following query:

   CREATE OR REPLACE TABLE `bqml_tutorial.search_embedding`
   AS
   SELECT * FROM ML.GENERATE_EMBEDDING(
     MODEL `bqml_tutorial.multimodal_embedding_model`,
     (
       SELECT 'pictures of white or cream colored dress from victorian era' AS content
     )
   );

Perform a cross-modality text-to-image search

Use the VECTOR_SEARCH function to search for images that best correspond to the search string represented by the text embedding, and then write them to a table for use in a following step.

1. In the Google Cloud console, go to the BigQuery page.

   Go to BigQuery

2. In the query editor, run the following query:

   CREATE OR REPLACE TABLE `bqml_tutorial.vector_search_results` AS
   SELECT base.uri AS gcs_uri, distance
   FROM
     VECTOR_SEARCH(
       TABLE `bqml_tutorial.met_image_embeddings`,
       'ml_generate_embedding_result',
       TABLE `bqml_tutorial.search_embedding`,
       'ml_generate_embedding_result',
       top_k => 3);

Visualize the vector search results

Visualize the vector search results by using a notebook.

1. In the Google Cloud console, go to the BigQuery page.

   Go to BigQuery

2. Open the met-image-analysis notebook that you created earlier.

3. Visualize the vector search results:

   1. Add a code cell to the notebook.

   2. Copy and paste the following code into the code cell:

      query = """
        SELECT * FROM `bqml_tutorial.vector_search_results`
        ORDER BY distance;
      """
      
      printImages(client.query(query))
      

   3. Run the code cell.

      The results should look similar to the following:

      

Clean up

Caution: Deleting a project has the following effects:

• Everything in the project is deleted. If you used an existing project for the tasks in this document, when you delete it, you also delete any other work you've done in the project.
• Custom project IDs are lost. When you created this project, you might have created a custom project ID that you want to use in the future. To preserve the URLs that use the project ID, such as an appspot.com URL, delete selected resources inside the project instead of deleting the whole project.

If you plan to explore multiple architectures, tutorials, or quickstarts, reusing projects can help you avoid exceeding project quota limits.

1. In the Google Cloud console, go to the Manage resources page.

   Go to Manage resources

2. In the project list, select the project that you want to delete, and then click Delete.
3. In the dialog, type the project ID, and then click Shut down to delete the project.