- HTTP request
- Path parameters
- Request body
- Response body
- Authorization scopes
- OrganicCrawlRateTimeSeries
- CrawlRateTimeSeries
- DedicatedCrawlRateTimeSeries
- State
Obtains the time series data of organic or dedicated crawl rate for monitoring. When dedicated crawl rate is not set, it will return vertex AI's organic crawl rate time series. Organic crawl means Google automatically crawl the internet at its own convenience. When dedicated crawl rate is set, it will return vertex AI's dedicated crawl rate time series.
HTTP request
POST https://discoveryengine.googleapis.com/v1beta/{location=projects/*/locations/*}:obtainCrawlRate
The URL uses gRPC Transcoding syntax.
Path parameters
Parameters | |
---|---|
location |
Required. The location resource where crawl rate management will be performed. Format: |
Request body
The request body contains data with the following structure:
JSON representation |
---|
{ "crawlRateScope": string } |
Fields | |
---|---|
crawl |
Required. The scope of the crawl rate that the user wants to monitor. Currently, only domain and host name are supported. A domain name example: |
Response body
Response message for CrawlRateManagementService.ObtainCrawlRate
method. The response contains organcic or dedicated crawl rate time series data for monitoring, depending on whether dedicated crawl rate is set.
If successful, the response body contains data with the following structure:
JSON representation |
---|
{ "state": enum ( |
Fields | |
---|---|
state |
Output only. The state of the response. |
error |
Errors from service when handling the request. |
Union field crawl_rate_time_series . Once the user sets the dedicated crawl rate, it will return the dedicated crawl rate time series, otherwise it will return the organic crawl rate time series. crawl_rate_time_series can be only one of the following: |
|
organic |
The historical organic crawl rate timeseries data, used for monitoring. |
dedicated |
The historical dedicated crawl rate timeseries data, used for monitoring. |
Authorization scopes
Requires the following OAuth scope:
https://www.googleapis.com/auth/cloud-platform
For more information, see the Authentication Overview.
OrganicCrawlRateTimeSeries
The historical organic crawl rate timeseries data, used for monitoring. Organic crawl is auto-determined by Google to crawl the user's website when dedicate crawl is not set. Crawl rate is the QPS of crawl request Google sends to the user's website.
JSON representation |
---|
{ "googleOrganicCrawlRate": { object ( |
Fields | |
---|---|
google |
Google's organic crawl rate time series, which is the sum of all googlebots' crawl rate. Please refer to https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers for more details about googlebots. |
vertex |
Vertex AI's organic crawl rate time series, which is the crawl rate of Google-CloudVertexBot when dedicate crawl is not set. Please refer to https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers#google-cloudvertexbot for more details about Google-CloudVertexBot. |
CrawlRateTimeSeries
The historical crawl rate timeseries data, used for monitoring.
JSON representation |
---|
{
"qpsTimeSeries": {
object ( |
Fields | |
---|---|
qps |
The QPS of the crawl rate. |
DedicatedCrawlRateTimeSeries
The historical dedicated crawl rate timeseries data, used for monitoring. Dedicated crawl is used by Vertex AI to crawl the user's website when dedicate crawl is set.
JSON representation |
---|
{ "userTriggeredCrawlRate": { object ( |
Fields | |
---|---|
user |
Vertex AI's dedicated crawl rate time series of user triggered crawl, which is the crawl rate of Google-CloudVertexBot when dedicate crawl is set, and user triggered crawl rate is for deterministic use cases like crawling urls or sitemaps specified by users. |
auto |
Vertex AI's dedicated crawl rate time series of auto-refresh, which is the crawl rate of Google-CloudVertexBot when dedicate crawl is set, and the crawl rate is for best effort use cases like refreshing urls periodically. |
State
Different states of the response.
Enums | |
---|---|
STATE_UNSPECIFIED |
The state is unspecified. |
SUCCEEDED |
The state is successful. |
FAILED |
The state is failed. |