Index and refresh web pages using sitemaps

If advanced website indexing is enabled in your data store, you can submit and use sitemaps to index and refresh the web pages in your data store. This feature supports only XML sitemaps and sitemap indexes.

This page describes how to submit a sitemap or sitemap index to trigger sitemap-based indexing and refresh. To understand and implement automatic and manual refresh without a sitemap, see Refresh web pages.

Additionally, this page describes how to view the sitemaps in your data store or delete a sitemap.

Sitemap-based refresh concepts

Here are some key concepts and terms that'll help you get started:

  • Sitemap protocol: All sitemaps and sitemap indexes that Vertex AI Search supports must follow the sitemap protocol.

  • Sitemap: A sitemap is a UTF-8 encoded, XML file that contains a list of URLs of the web pages and files in your website with other important but optional information such as the web page's last modified date and the priority of the web page for a crawler against other web pages in your website. According to the sitemap protocol, a single sitemap can contain a maximum of 50,000 URLs and it can be a maximum of 50 MB.

  • Sitemap index: When your sitemap exceeds the maximum URLs or maximum size, you can create multiple sitemaps and list these sitemaps in a sitemap index file. According to the sitemap protocol, a single sitemap index can nest a maximum of 50,000 sitemaps and it can be a maximum of 50 MB.

You can submit one or more sitemaps, one or more sitemap indexes, or a combination of sitemaps and sitemap indexes to Vertex AI Search.

When you submit your sitemap or sitemap index to the Vertex AI Search data store, you trigger the following actions:

  • Indexing of the URLs in your sitemap or sitemap index that fit the URL pattern included in your data store
  • Daily refresh of the included URLs

Website data store refresh methods

You can choose one of the following ways to incorporate sitemap-based refresh in your data store:

  • Sitemap-only refresh: Use sitemap-based refresh exclusively by turning off initial indexing and automatic refresh.
  • Combination refresh: Use sitemap-based refresh with initial indexing and automatic refresh.

You can manually refresh specific web pages in your data store index at any time, regardless of the refresh method that you choose.

Sitemap-only refresh

When you create a website data store, you must provide URL patterns for the web pages that you want to include in your data store's index. By default, when you finish creating a website data store, Vertex AI Search generates an initial index for these included web pages.

For website data stores with advanced website indexing, the initial indexing process is a part of automatic refresh. The initial indexing process indexes all the included URLs that are available in Google Search. The initial freshness of these URLs mirrors the freshness available in Google Search. After the initial indexing, the automatic refresh process discovers new pages and refreshes them on a best-effort basis. This can result in relatively stale pages and a bulkier index because this process discovers URLs that might be beyond what are required.

Instead, you can choose to perform sitemap-only refresh, which is useful in the following scenarios:

  • You have a well-maintained sitemap that's up-to-date.
  • You have a large website and you need a tighter control over which web pages are indexed. This results in a leaner, manageable index.
  • You need to refresh your indexed pages daily. This results in a fresher index.

The following table compares the different methods that refresh the data store index:

Refresh method Precision Manual intervention Frequency Discovery
Sitemap-based refresh Exact. Indexes only the URLs in the sitemaps. Not required after submitting the sitemap or sitemap index Daily Not beyond those specified in the sitemap.
Manual refresh (also known as recrawl) Exact. Indexes only the URLs specified in the recrawl request. Required On demand No.
Automatic refresh Not exact. The data store is refreshed on a best effort basis. Not required Random and on best-effort basis Yes. Discovers URLs beyond what's available in Google Search.

Before you begin

Before you submit a sitemap or sitemap index to the Vertex AI Search data store:

  • Create an XML sitemap or a sitemap index that references all the sitemaps for your website according to the sitemap protocol.
  • Understand that to submit a sitemap or a sitemap index to the Vertex AI Search data store, it's not necessary to them to Google Search.
  • All the URLs in your sitemap that you want indexed must belong to public domains that are verified in your data store. For more information, see Verify website domains.
  • The sitemap URI or the sitemap index URI with the nested sitemap URIs must be publicly available.

Submit a sitemap or sitemap index to a data store

To index and refresh the included web pages in your data store, follow these steps:

  1. Decide whether you want to perform a sitemap-only refresh or combination refresh with other methods.

    • If you want to perform a sitemap-only refresh, you can't use an existing data store which has initial indexing and refreshing. You must create a new data store by turning off initial indexing and automatic refresh using the AdvancedSiteSearchConfig configuration.

      REST

      Enable sitemap-only refresh in a data store

      • Create a data store by turning off initial index and automatic refresh.
      curl -X POST\
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "Content-Type: application/json"\
-H "X-Goog-User-Project: PROJECT_ID" \
"https://us-discoveryengine.googleapis.com/v1alpha/projects/PROJECT_ID/locations/us/collections/default_collection/dataStores?dataStoreId=DATA_STORE_ID" \
-d '{
 "displayName": "DATA_STORE_DISPLAY_NAME",
 "industryVertical": "GENERIC",
 "content_config": "PUBLIC_WEBSITE",
 "searchTier": "ENTERPRISE",
 "advancedSiteSearchConfig": {
    "disableInitialIndex": true,
    "disableAutomaticRefresh": true,
 }
}'

      Replace the following:

      • PROJECT_ID: the ID of your Google Cloud project.
      • DATA_STORE_ID: the ID of the Vertex AI Search data store that you want to create. This ID can contain only lowercase letters, digits, underscores, and hyphens.
      • DATA_STORE_DISPLAY_NAME: the display name of the Vertex AI Search data store that you want to create.

  2. Update the URL patterns of the sites to include and exclude in your data store. For more information, Create a data store using website content.

  3. Verify the domains of the web pages included in your data store.

  4. Whether you choose sitemap-only refresh or a combination refresh, submit a sitemap or sitemap index URI to a data store using the sitemaps.create method.

    REST

    Submit a sitemap or sitemap index.

    curl -X POST \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "Content-Type: application/json" \
-H "X-Goog-User-Project: PROJECT_NUMBER" \
"https://discoveryengine.googleapis.com/v1alpha/projects/PROJECT_NUMBER/locations/global/dataStores/DATA_STORE_ID/siteSearchEngine/sitemaps?sitemap.uri=SITEMAP_URI" \

    Replace the following:

    • PROJECT_ID: the ID of your Google Cloud project.
    • DATA_STORE_ID: the ID of the Vertex AI Search data store.
    • SITEMAP_URI: the public URI for the individual sitemap or the sitemap index that you want to submit. When you submit a sitemap index, it's sufficient to submit the URI of the sitemap index. Vertex AI Search automatically indexes the included URLs in all the sitemaps nested within the sitemap index.

    After you submit the sitemap or sitemap index to the data store, Vertex AI Search triggers the following:

    • An indexing of the eligible URLs in the sitemap—those which are included in your data store. This process can take a few hours to complete. Larger sitemaps can take longer to index.
    • A daily refresh of the web pages with eligible URLs.

    To know how modifications to the sitemap or sitemap index effect the refresh, see Changes to sitemap and sitemap index.

  5. View the sitemaps in your data store.

Changes to sitemap and sitemap index

After the initial submit, Vertex AI Search detects modifications in your sitemap or sitemap index on a daily basis and handles these modifications in the following manner:

  • Changes to a sitemap:
    • When you add URLs: URLs that match the included URLs pattern for the data store are added to the index and refreshed on a daily basis.
    • When you remove URLs: If the removed URLs are in the index, they're removed from the index and are no longer refreshed.
    • When you update the existing URLs—for example, when you update the lastmod field for a URL in the sitemap: Any updated URLs that match the included URLs pattern for the data store are refreshed. The refresh usually occurs within 24 hours after the update.
  • Changes to a sitemap index:
    • When you add a sitemap: URLs in the new sitemap that match the included URLs pattern for the data store are added to the index and refreshed on a daily basis.
    • When you remove a sitemap: The URLs that match the included URLs pattern for the data store are no longer refreshed. However, they still remain in the index. To remove the sitemap and its URLs from the index, see Remove a sitemap and its URLs from the index.

List the sitemaps and sitemap indexes in a data store

To list all the sitemaps and sitemap indexes in a data store, use the sitemaps.fetch method. If you have submitted a sitemap index, then this method returns the sitemap index and not the individual, nested sitemaps. If there are no sitemaps in the data store, this request returns an empty JSON file.

REST

curl -X GET \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "X-Goog-User-Project: PROJECT_ID" \
"https://discoveryengine.googleapis.com/v1alpha/projects/PROJECT_ID/locations/global/dataStores/DATA_STORE_ID/siteSearchEngine/sitemaps:fetch" \

Replace the following:

  • PROJECT_ID: the ID of your Google Cloud project.
  • DATA_STORE_ID: the ID of the Vertex AI Search data store.

Check whether a sitemap or sitemap index is present in a data store

To check whether a sitemap or sitemap index is present in a data store, use the sitemaps.fetch method. If the sitemap or sitemap index for which you're checking is submitted to the data store, then the response contains the sitemap name and URI of the sitemap. If you have submitted a sitemap index, then checking for individual sitemaps within the sitemap index won't return the correct results.

REST

curl -X GET \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "X-Goog-User-Project: PROJECT_ID" \
"https://discoveryengine.googleapis.com/v1alpha/projects/PROJECT_ID/locations/global/dataStores/DATA_STORE_ID/siteSearchEngine/sitemaps:fetch?matcher.uris_matcher.uris=SITEMAP_URI_1&matcher.uris_matcher.uris=SITEMAP_URI_2" \

Replace the following:

  • PROJECT_ID: the ID of your Google Cloud project.
  • DATA_STORE_ID: the ID of the Vertex AI Search data store.
  • SITEMAP_URI_N: the public URI of the sitemap or the sitemap index for which you want to check in the data store.

Delete a sitemap or sitemap index from the data store

To delete a sitemap from the data store, use the sitemap.delete method. Deleting a sitemap doesn't remove its URLs from the index. To remove the sitemap and its URLs from the index, see Remove a sitemap and its URLs from the index.

REST

1. Delete a sitemap or sitemap index.

curl -X DELETE \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "X-Goog-User-Project: PROJECT_ID" \
"https://discoveryengine.googleapis.com/v1alpha/projects/PROJECT_ID/locations/global/collections/default_collection/dataStores/DATA_STORE_ID/siteSearchEngine/sitemaps/SITEMAP_ID" \

Replace the following:

  • PROJECT_ID: the ID of your Google Cloud project.
  • DATA_STORE_ID: the ID of the Vertex AI Search data store.
  • SITEMAP_ID: a unique ID that identifies a sitemap or sitemap index. You can find this ID in the name field of the response when when you [submit a sitemap or a sitemap index](#submit-sitemap), or [list the sitemaps and sitemap indexes in your data store](#fetch-sitemap).

Remove a sitemap or a sitemap index and its URLs from the data store index

To remove a sitemap or a sitemap index and its URLs from the index, follow these steps:

  1. Empty the sitemap or the sitemap index that was submitted to the data store by removing all its URLs.

    If you submitted a sitemap index to the data store, empty the nested sitemaps by removing all the URLs and remove the sitemap from the sitemap index.

  2. Wait for a period of 48 hours for Vertex AI Search to process these changes and remove the URLs from the data store's index.

  3. Delete the sitemap or the sitemap index.