Method: projects.locations.obtainCrawlRate

Obtains the time series data of organic or dedicated crawl rate for monitoring. When dedicated crawl rate is not set, it will return vertex AI's organic crawl rate time series. Organic crawl means Google automatically crawl the internet at its own convenience. When dedicated crawl rate is set, it will return vertex AI's dedicated crawl rate time series.

HTTP request

POST https://discoveryengine.googleapis.com/v1beta/{location=projects/*/locations/*}:obtainCrawlRate

The URL uses gRPC Transcoding syntax.

Path parameters

Parameters
location

string

Required. The location resource where crawl rate management will be performed. Format: projects/{project}/locations/{location}

Request body

The request body contains data with the following structure:

JSON representation 
{
  "crawlRateScope": string
}
Fields
crawlRateScope

string

Required. The scope of the crawl rate that the user wants to monitor. Currently, only domain and host name are supported. A domain name example: example.com. A host name example: www.example.com. Please do not include / in the domain or host name.

Response body

Response message for CrawlRateManagementService.ObtainCrawlRate method. The response contains organcic or dedicated crawl rate time series data for monitoring, depending on whether dedicated crawl rate is set.

If successful, the response body contains data with the following structure:

JSON representation 
{
  "state": enum (State),
  "error": {
    object (Status)
  },

  // Union field crawl_rate_time_series can be only one of the following:
  "organicCrawlRateTimeSeries": {
    object (OrganicCrawlRateTimeSeries)
  },
  "dedicatedCrawlRateTimeSeries": {
    object (DedicatedCrawlRateTimeSeries)
  }
  // End of list of possible types for union field crawl_rate_time_series.
}
Fields
state

enum (State)

Output only. The state of the response.
error

object (Status)

Errors from service when handling the request.
Union field crawl_rate_time_series. Once the user sets the dedicated crawl rate, it will return the dedicated crawl rate time series, otherwise it will return the organic crawl rate time series. crawl_rate_time_series can be only one of the following:
organicCrawlRateTimeSeries

object (OrganicCrawlRateTimeSeries)

The historical organic crawl rate timeseries data, used for monitoring.
dedicatedCrawlRateTimeSeries

object (DedicatedCrawlRateTimeSeries)

The historical dedicated crawl rate timeseries data, used for monitoring.

Authorization scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.

OrganicCrawlRateTimeSeries

The historical organic crawl rate timeseries data, used for monitoring. Organic crawl is auto-determined by Google to crawl the user's website when dedicate crawl is not set. Crawl rate is the QPS of crawl request Google sends to the user's website.

JSON representation 
{
  "googleOrganicCrawlRate": {
    object (CrawlRateTimeSeries)
  },
  "vertexAiOrganicCrawlRate": {
    object (CrawlRateTimeSeries)
  }
}
Fields
googleOrganicCrawlRate

object (CrawlRateTimeSeries)

Google's organic crawl rate time series, which is the sum of all googlebots' crawl rate. Please refer to https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers for more details about googlebots.
vertexAiOrganicCrawlRate

object (CrawlRateTimeSeries)

Vertex AI's organic crawl rate time series, which is the crawl rate of Google-CloudVertexBot when dedicate crawl is not set. Please refer to https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers#google-cloudvertexbot for more details about Google-CloudVertexBot.

CrawlRateTimeSeries

The historical crawl rate timeseries data, used for monitoring.

JSON representation 
{
  "qpsTimeSeries": {
    object (TimeSeries)
  }
}
Fields
qpsTimeSeries

object (TimeSeries)

The QPS of the crawl rate.

DedicatedCrawlRateTimeSeries

The historical dedicated crawl rate timeseries data, used for monitoring. Dedicated crawl is used by Vertex AI to crawl the user's website when dedicate crawl is set.

JSON representation 
{
  "userTriggeredCrawlRate": {
    object (CrawlRateTimeSeries)
  },
  "autoRefreshCrawlRate": {
    object (CrawlRateTimeSeries)
  }
}
Fields
userTriggeredCrawlRate

object (CrawlRateTimeSeries)

Vertex AI's dedicated crawl rate time series of user triggered crawl, which is the crawl rate of Google-CloudVertexBot when dedicate crawl is set, and user triggered crawl rate is for deterministic use cases like crawling urls or sitemaps specified by users.
autoRefreshCrawlRate

object (CrawlRateTimeSeries)

Vertex AI's dedicated crawl rate time series of auto-refresh, which is the crawl rate of Google-CloudVertexBot when dedicate crawl is set, and the crawl rate is for best effort use cases like refreshing urls periodically.

State

Different states of the response.

Enums
STATE_UNSPECIFIED The state is unspecified.
SUCCEEDED The state is successful.
FAILED The state is failed.