Ground with your data

This page explains how you can ground responses by using your data from Vertex AI Search (Preview).

Ground Gemini to your data

Preview

This product or feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of the Service Specific Terms. Pre-GA products and features are available "as is" and might have limited support. For more information, see the launch stage descriptions.

If you want to do retrieval-augmented generation (RAG), connect your model to your website data or your sets of documents, then use Grounding with Vertex AI Search.

Grounding to your data supports a maximum of 10 Vertex AI Search data sources and can be combined with Grounding with Google Search.

Supported models

This section lists the models that support grounding with your data. To explore how each model generates grounded responses, follow these instructions:

1. Try a model listed in this table in the Google Cloud console.

2. Click the Grounding toggle to the on position.

3. Click Customize and a Customize Grounding pane displays.

4. Select Vertex AI Search.

5. In the Grounding with Vertex AI Search section, enter the path of the Vertex AI datastore. If you don't have a Vertex AI data store, create a data store. For more information, see Create a data store.

6. Click Save.

Model	Description	Try a model
Gemini 1.5 Pro	Text input only	Try the Gemini 1.5 Pro model
Gemini 1.5 Flash	Text input only	Try the Gemini 1.5 Flash model
Gemini 1.0 Pro	Text input only	Try the Gemini 1.0 Pro model
Gemini 2.0 Flash	Text, code, images, audio, video, video with audio, PDF	Try the Gemini 2.0 Flash model

Prerequisites

Before you can ground model output to your data, do the following:

1. Enable Vertex AI Agent Builder and activate the API.
2. Create a Vertex AI Agent Builder data source and application.

See the Introduction to Vertex AI Search for more.

Enable Vertex AI Agent Builder

1. In the Google Cloud console, go to the Agent Builder page.

   Agent Builder

2. Read and agree to the terms of service, then click Continue and activate the API.

   Important: You must accept the discovery solutions data use terms for every project that you want to use Vertex AI Agent Builder with.

Vertex AI Agent Builder is available in the global location, or the eu and us multi-region. To learn more, see Vertex AI Agent Builder locations

Create a data store in Vertex AI Agent Builder

To create a data store in Vertex AI Agent Builder, you can choose to ground with website data or documents.

Website

1. Open the Create Data Store page from the Google Cloud console.

2. In Website Content box, click Select.
   Specify the websites for your data store pane displays.

3. If Advanced website indexing isn't checked, then select the Advanced website indexing checkbox to turn it on.
   Configure your data store pane displays.

4. In the Specify URL patterns to index section, do the following:

   • Add URLs for Sites to include.
   • Optional: Add URLs for Sites to exclude.

5. Click Continue.

6. In the Configure your data store pane,

   1. Select a value from the Location of your data store list.
   2. Enter a name in the Your data store name field. The ID is generated. Use this ID when you generate your grounded responses with your data store. For more information, see Generate grounded responses with your data store.
   3. Click Create.

Documents

1. Open the Create Data Store page from the Google Cloud console.

2. In Cloud Storage box, click Select.
   Import data from Cloud Storage pane displays.

3. In the Unstructured documents (PDF, HTML, TXT and more) section, select Unstructured documents (PDF, HTML, TXT and more).

4. Select a Synchronization frequency option.

5. Select a Select a folder or a file you want to import option, and enter the path in the field.

6. Click Continue.
   Configure your data store pane displays.

7. In the Configure your data store pane,

   1. Select a value from the Location of your data store list.
   2. Enter a name in the Your data store name field. The ID is generated.
   3. To select parsing and chunking options for your documents, expand the Document Processing Options section. For more information about different parsers, see Parse documents.
   4. Click Create.

8. Click Create.

Generate grounded responses with your data store

Use the following instructions to ground a model with your data. A maximum of 10 data stores is supported.

If you don't know your data store ID, follow these steps:

1. In the Google Cloud console, go to the Vertex AI Agent Builder page and in the navigation menu, click Data stores.

   Go to the Data stores page

2. Click the name of your data store.

3. On the Data page for your data store, get the data store ID.

Console

To ground your model output to Vertex AI Agent Builder by using Vertex AI Studio in the Google Cloud console, follow these steps:

1. In the Google Cloud console, go to the Vertex AI Studio page, and click the Freeform tab.

   Go to Vertex AI Studio

2. In the side panel, click the Ground model responses toggle to enable grounding.
3. Click Customize.
   1. Set Vertex AI Search as your source.
   2. Using this path format, replace your data store's Project ID and the ID of the data store:
      

      projects/project_id/locations/global/collections/default_collection/dataStores/data_store_id.

4. Enter your prompt in the text box, and click Submit.

Your prompt responses are grounded to Vertex AI Agent Builder.

Python

Before trying this sample, follow the Python setup instructions in the Vertex AI quickstart using client libraries. For more information, see the Vertex AI Python API reference documentation.

To authenticate to Vertex AI, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.

import vertexai

from vertexai.preview.generative_models import (
    GenerationConfig,
    GenerativeModel,
    Tool,
    grounding,
)

# TODO(developer): Update and un-comment below lines
# PROJECT_ID = "your-project-id"
# data_store_id = "your-data-store-id"

vertexai.init(project=PROJECT_ID, location="us-central1")

model = GenerativeModel("gemini-1.5-flash-001")

tool = Tool.from_retrieval(
    grounding.Retrieval(
        grounding.VertexAISearch(
            datastore=data_store_id,
            project=PROJECT_ID,
            location="global",
        )
    )
)

prompt = "How do I make an appointment to renew my driver's license?"
response = model.generate_content(
    prompt,
    tools=[tool],
    generation_config=GenerationConfig(
        temperature=0.0,
    ),
)

print(response.text)

REST

To test a text prompt by using the Vertex AI API, send a POST request to the publisher model endpoint.

Before using any of the request data, make the following replacements:

• LOCATION: The region to process the request.
• PROJECT_ID: Your project ID.
• MODEL_ID: The model ID of the multimodal model.
• TEXT: The text instructions to include in the prompt.

HTTP method and URL:

POST https://LOCATION-aiplatform.googleapis.com/v1beta1/projects/PROJECT_ID/locations/LOCATION/publishers/google/models/MODEL_ID:generateContent

Request JSON body:

{
  "contents": [{
    "role": "user",
    "parts": [{
      "text": "TEXT"
    }]
  }],
  "tools": [{
    "retrieval": {
      "vertexAiSearch": {
        "datastore": projects/PROJECT_ID/locations/global/collections/default_collection/dataStores/DATA_STORE_ID
      }
    }
  }],
  "model": "projects/PROJECT_ID/locations/LOCATION/publishers/google/models/MODEL_ID"
}

To send your request, expand one of these options:

curl (Linux, macOS, or Cloud Shell)

Note: The following command assumes that you have logged in to the gcloud CLI with your user account by running gcloud init or gcloud auth login , or by using Cloud Shell, which automatically logs you into the gcloud CLI . You can check the currently active account by running gcloud auth list.

Save the request body in a file named request.json, and execute the following command:

curl -X POST \
     -H "Authorization: Bearer $(gcloud auth print-access-token)" \
     -H "Content-Type: application/json; charset=utf-8" \
     -d @request.json \
     "https://LOCATION-aiplatform.googleapis.com/v1beta1/projects/PROJECT_ID/locations/LOCATION/publishers/google/models/MODEL_ID:generateContent"

PowerShell (Windows)

Note: The following command assumes that you have logged in to the gcloud CLI with your user account by running gcloud init or gcloud auth login . You can check the currently active account by running gcloud auth list.

Save the request body in a file named request.json, and execute the following command:

$cred = gcloud auth print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
    -Method POST `
    -Headers $headers `
    -ContentType: "application/json; charset=utf-8" `
    -InFile request.json `
    -Uri "https://LOCATION-aiplatform.googleapis.com/v1beta1/projects/PROJECT_ID/locations/LOCATION/publishers/google/models/MODEL_ID:generateContent" | Select-Object -Expand Content

You should receive a JSON response similar to the following:

{
  "candidates": [
    {
      "content": {
        "role": "model",
        "parts": [
          {
            "text": "You can make an appointment on the website https://dmv.gov/"
          }
        ]
      },
      "finishReason": "STOP",
      "safetyRatings": [
        "..."
      ],
      "groundingMetadata": {
        "retrievalQueries": [
          "How to make appointment to renew driving license?"
        ],
        "groundingChunks": [
          {
            "retrievedContext": {
              "uri": "https://vertexaisearch.cloud.google.com/grounding-api-redirect/AXiHM.....QTN92V5ePQ==",
              "title": "dmv"
            }
          }
        ],
        "groundingSupport": [
          {
            "segment": {
              "startIndex": 25,
              "endIndex": 147
            },
            "segment_text": "ipsum lorem ...",
            "supportChunkIndices": [1, 2],
            "confidenceScore": [0.9541752, 0.97726375]
          },
          {
            "segment": {
              "startIndex": 294,
              "endIndex": 439
            },
            "segment_text": "ipsum lorem ...",
            "supportChunkIndices": [1],
            "confidenceScore": [0.9541752, 0.9325467]
          }
        ]
      }
    }
  ],
  "usageMetadata": {
    "..."
  }
}

Understand your response

If your model prompt successfully grounds to Vertex AI Search data stores using the Vertex AI Studio or the API, then the model's responses include metadata with citations and source links. If low-source relevance or incomplete information occurs within the model's response, then metadata might not be provided, and the prompt response won't be grounded.

What's next

• To learn how to send chat prompt requests, see Multiturn chat.
• To learn about responsible AI best practices and Vertex AI's safety filters, see Safety best practices.