Overview
With large search queries, updating your indexes is important to always having the most accurate information. Today you can update your Vector Search indexes using a batch update, which lets you to insert and delete data points through a batch schedule, or with streaming update, which lets you update and query your index within a few seconds.
Additionally, you can use UpdateIndex
to update your important metadata fields,
like display_name
, description
and labels
. You can also add optional tags
to your index to help with diversifying results or filtering pre index query.
Update index content with Batch Updates
To update the content of an existing Index
, use the IndexService.UpdateIndex
method.
To replace the existing content of an existing Index
:
- Set
Index.metadata.contentsDeltaUri
to the Cloud Storage URI that includes the vectors you want to update. - Set
isCompleteOverwrite
to true.
If you set the contentsDeltaUri
field when calling
IndexService.UpdateIndex
, then no other index fields (such as displayName
,
description
, or userLabels
) can be also updated as part of the same call.
gcloud
gcloud ai indexes update INDEX_ID \
--metadata-file=LOCAL_PATH_TO_METADATA_FILE \
--project=PROJECT_ID \
--region=LOCATION
Replace the following:
- INDEX_ID: The ID of the index.
- LOCAL_PATH_TO_METADATA_FILE: The local path to the metadata file.
- PROJECT_ID: The ID of the project.
- LOCATION: The region where you are using Vertex AI.
REST
Before using any of the request data, make the following replacements:
- LOCATION: Your region.
- PROJECT_ID: Your project ID.
- INPUT_DIR: The Cloud Storage directory path of the index content.
- PROJECT_NUMBER: Project number for your project
HTTP method and URL:
PATCH https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/indexes/INDEX_ID
Request JSON body:
{ "metadata": { "contentsDeltaUri": "INPUT_DIR", "isCompleteOverwrite": true } }
To send your request, expand one of these options:
You should receive a JSON response similar to the following:
{ "name": "projects/PROJECT_NUMBER/locations/LOCATION/indexes/INDEX_ID/operations/OPERATION_ID", "metadata": { "@type": "type.googleapis.com/google.cloud.aiplatform.v1.UpdateIndexOperationMetadata", "genericMetadata": { "createTime": "2022-01-12T23:56:14.480948Z", "updateTime": "2022-01-12T23:56:14.480948Z" } } }
"done": true
.
If the Index
has any associated deployments
(see the Index.deployed_indexes
field), then when certain changes to the
original Index
are done, the DeployedIndex
is automatically updated
asynchronously in the background to reflect these changes.
To check whether the change has been propagated, compare the update index
operation finish time and the DeployedIndex.index_sync_time
.
Update an index using Streaming Updates
With Streaming Updates, you can update and query your index within a few seconds. At this time, you can't use Streaming Updates on an existing index, you must create a new index. See Create an index for Streaming Update to learn more.
You are charged $0.45 per GB used for Streaming Updates. To learn more about pricing, see the Vertex AI pricing page. Streaming Updates are directly applied to the deployed indexes in memory, which are then reflected in query results after a short delay.
Upsert Data points
The throughput quota limit relates to the amount of data that is included in an upsert. If the data point ID exists in the index, the embedding is updated, otherwise, a new embedding is added.
DATAPOINT_ID_1=
DATAPOINT_ID_2=
curl -H "Content-Type: application/json" -H "Authorization: Bearer `gcloud auth print-access-token`" https://${LOCATION}-aiplatform.googleapis.com/v1/projects/${PROJECT_ID}/locations/${LOCATION}/indexes/${INDEX_ID}:upsertDatapoints \
-d '{datapoints: [{datapoint_id: "'${DATAPOINT_ID_1}'", feature_vector: [...]},
{datapoint_id: "'${DATAPOINT_ID_2}'", feature_vector: [...]}]}'
Remove datapoints
curl -H "Content-Type: application/json" -H "Authorization: Bearer `gcloud auth print-access-token`" https://${LOCATION}-aiplatform.googleapis.com/v1/projects/${PROJECT_ID}/locations/${REGION}/indexes/${INDEX_ID}:removeDatapoints -d '{datapoint_ids: ["'${DATAPOINT_ID_1}'", "'${DATAPOINT_ID_2}'"]}'
Compaction
Periodically, your index is rebuilt to account for all new updates since your last rebuild. This rebuild, or "compaction", improves query performance and reliability. Compactions occur for both Streaming Updates and Batch Updates.
Streaming Update: Occurs when the uncompacted data size is > 1 GB or the oldest uncompacted data is at least three days old. You are billed for the cost of rebuilding the index at the same rate of a batch update, in addition to the Streaming Update costs.
Batch Update: Occurs when the incremental dataset size is > 20% of the base dataset size.
Rebuild and query your index
You can send Match/BatchMatch requests as usual with the grpc cli, the client library or the python SDK. When you rebuild the query you can expect to see your updates within a few seconds. To learn how to query an index, see Query indexes to get nearest neighbors.
Optional fields
When you create an index, there are some optional fields you can use to fine-tune your queries.
Upsert with restricts
Upserting your index and adding a restrict is a way of tagging your data points so they are already identified for filtering at query time. You might want to add restrict tags to limit the results that presents on your data before a query is sent. For example, a customer wants to run a query on an index, but wants to make sure the results only display items that match "red" in a search for footwear. In the example below, the index is being upserted and is filtering in all red shoes, but denying blue ones. This ensures the search filters in the best specific options from a large and varied index before running.
To learn more about filtering, see Vector Search for Indexing.
curl -H "Content-Type: application/json" -H "Authorization: Bearer `gcloud auth print-access-token`" https://${ENDPOINT}/v1/projects/${PROJECT_ID}/locations/us-central1/indexes/${INDEX_ID}:upsertDatapoints \
-d '{
datapoints: [
{
datapoint_id: "'${DATAPOINT_ID_1}'",
feature_vector: [...],
restricts: { namespace: "color", allow_list: ["red"], deny_list: ["blue"]}
}
]}'
Upsert with crowding
The crowding tag limits similar results by improving result diversity.
Crowding is a constraint on a neighbor list produced by a nearest neighbor
search requiring that no more than some value, of a group of results, return the
same value of crowding_attribute
. As an example, let's say you were back online
shopping for shoes. You want to see a wide variety of colors in the results, but maybe
want them in a single style, like soccer cleats. You can ask that no more than 3 pairs of shoes with
the same color is returned by setting per_crowding_attribute_num_neighbors
= 3
in your query, assuming you set crowding_attribute to the color of the shoes
when inserting the data point.
This field represents the allowed maximum number of matches with the same crowding tag.
curl -H "Content-Type: application/json" -H "Authorization: Bearer `gcloud auth print-access-token`" https://${ENDPOINT}/v1/projects/${PROJECT_ID}/locations/us-central1/indexes/${INDEX_ID}:upsertDatapoints \
-d '{
datapoints: [
{
datapoint_id: "'${DATAPOINT_ID_1}'",
feature_vector: [...],
restricts: { namespace: "type", allow_list: ["cleats"]}
crowding_tag: { crowding_attribute: "red-shoe"},
}
]}'