REST Resource: projects.locations.evaluations

Resource: Evaluation

An evaluation is a single execution (or run) of an evaluation process. It encapsulates the state of the evaluation and the resulting data.

JSON representation 
{
  "name": string,
  "evaluationSpec": {
    object (EvaluationSpec)
  },
  "qualityMetrics": {
    object (QualityMetrics)
  },
  "state": enum (State),
  "error": {
    object (Status)
  },
  "createTime": string,
  "endTime": string,
  "errorSamples": [
    {
      object (Status)
    }
  ]
}
Fields
name

string

Identifier. The full resource name of the Evaluation, in the format of projects/{project}/locations/{location}/evaluations/{evaluation}.

This field must be a UTF-8 encoded string with a length limit of 1024 characters.
evaluationSpec

object (EvaluationSpec)

Required. The specification of the evaluation.
qualityMetrics

object (QualityMetrics)

Output only. The metrics produced by the evaluation, averaged across all SampleQuerys in the SampleQuerySet.

Only populated when the evaluation's state is SUCCEEDED.
state

enum (State)

Output only. The state of the evaluation.
error

object (Status)

Output only. The error that occurred during evaluation. Only populated when the evaluation's state is FAILED.
createTime

string (Timestamp format)

Output only. timestamp the Evaluation was created at.

A timestamp in RFC3339 UTC "Zulu" format, with nanosecond resolution and up to nine fractional digits. Examples: "2014-10-02T15:01:23Z" and "2014-10-02T15:01:23.045123456Z".
endTime

string (Timestamp format)

Output only. timestamp the Evaluation was completed at.

A timestamp in RFC3339 UTC "Zulu" format, with nanosecond resolution and up to nine fractional digits. Examples: "2014-10-02T15:01:23Z" and "2014-10-02T15:01:23.045123456Z".
errorSamples[]

object (Status)

Output only. A sample of errors encountered while processing the request.

EvaluationSpec

Describes the specification of the evaluation.

JSON representation 
{
  "querySetSpec": {
    object (QuerySetSpec)
  },

  // Union field search_spec can be only one of the following:
  "searchRequest": {
    object (SearchRequest)
  }
  // End of list of possible types for union field search_spec.
}
Fields
querySetSpec

object (QuerySetSpec)

Required. The specification of the query set.
Union field search_spec. The search specification. search_spec can be only one of the following:
searchRequest

object (SearchRequest)

Required. The search request that is used to perform the evaluation.

Only the following fields within SearchRequest are supported; if any other fields are provided, an UNSUPPORTED error will be returned:

ImageQuery

Specifies the image query input.

JSON representation 
{

  // Union field image can be only one of the following:
  "imageBytes": string
  // End of list of possible types for union field image.
}
Fields

Union field image.

image can be only one of the following:
imageBytes

string

Base64 encoded image bytes. Supported image formats: JPEG, PNG, and BMP.

DataStoreSpec

A struct to define data stores to filter on in a search call and configurations for those data stores. Otherwise, an INVALID_ARGUMENT error is returned.

JSON representation 
{
  "dataStore": string,
  "filter": string,
  "boostSpec": {
    object (BoostSpec)
  }
}
Fields
dataStore

string

Required. Full resource name of DataStore, such as projects/{project}/locations/{location}/collections/{collectionId}/dataStores/{dataStoreId}.
filter

string

Optional. Filter specification to filter documents in the data store specified by dataStore field. For more information on filtering, see Filtering
boostSpec

object (BoostSpec)

Optional. Boost specification to boost certain documents. For more information on boosting, see Boosting

BoostSpec

Boost specification to boost certain documents.

JSON representation 
{
  "conditionBoostSpecs": [
    {
      object (ConditionBoostSpec)
    }
  ]
}
Fields
conditionBoostSpecs[]

object (ConditionBoostSpec)

Condition boost specifications. If a document matches multiple conditions in the specifictions, boost scores from these specifications are all applied and combined in a non-linear way. Maximum number of specifications is 20.

ConditionBoostSpec

Boost applies to documents which match a condition.

JSON representation 
{
  "condition": string,
  "boost": number,
  "boostControlSpec": {
    object (BoostControlSpec)
  }
}
Fields
condition

string

An expression which specifies a boost condition. The syntax and supported fields are the same as a filter expression. See SearchRequest.filter for detail syntax and limitations.

Examples:

  • To boost documents with document ID "doc_1" or "doc_2", and color "Red" or "Blue": (documentId: ANY("doc_1", "doc_2")) AND (color: ANY("Red", "Blue"))
boost

number

Strength of the condition boost, which should be in [-1, 1]. Negative boost means demotion. Default is 0.0.

Setting to 1.0 gives the document a big promotion. However, it does not necessarily mean that the boosted document will be the top result at all times, nor that other documents will be excluded. Results could still be shown even when none of them matches the condition. And results that are significantly more relevant to the search query can still trump your heavily favored but irrelevant documents.

Setting to -1.0 gives the document a big demotion. However, results that are deeply relevant might still be shown. The document will have an upstream battle to get a fairly high ranking, but it is not blocked out completely.

Setting to 0.0 means no boost applied. The boosting condition is ignored. Only one of the (condition, boost) combination or the boostControlSpec below are set. If both are set then the global boost is ignored and the more fine-grained boostControlSpec is applied.
boostControlSpec

object (BoostControlSpec)

Complex specification for custom ranking based on customer defined attribute value.

BoostControlSpec

Specification for custom ranking based on customer specified attribute value. It provides more controls for customized ranking than the simple (condition, boost) combination above.

JSON representation 
{
  "fieldName": string,
  "attributeType": enum (AttributeType),
  "interpolationType": enum (InterpolationType),
  "controlPoints": [
    {
      object (ControlPoint)
    }
  ]
}
Fields
fieldName

string

The name of the field whose value will be used to determine the boost amount.
attributeType

enum (AttributeType)

The attribute type to be used to determine the boost amount. The attribute value can be derived from the field value of the specified fieldName. In the case of numerical it is straightforward i.e. attributeValue = numerical_field_value. In the case of freshness however, attributeValue = (time.now() - datetime_field_value).
interpolationType

enum (InterpolationType)

The interpolation type to be applied to connect the control points listed below.
controlPoints[]

object (ControlPoint)

The control points used to define the curve. The monotonic function (defined through the interpolationType above) passes through the control points listed here.

AttributeType

The attribute(or function) for which the custom ranking is to be applied.

Enums
ATTRIBUTE_TYPE_UNSPECIFIED Unspecified AttributeType.
NUMERICAL The value of the numerical field will be used to dynamically update the boost amount. In this case, the attributeValue (the x value) of the control point will be the actual value of the numerical field for which the boostAmount is specified.
FRESHNESS For the freshness use case the attribute value will be the duration between the current time and the date in the datetime field specified. The value must be formatted as an XSD dayTimeDuration value (a restricted subset of an ISO 8601 duration value). The pattern for this is: [nD][T[nH][nM][nS]]. For example, 5D, 3DT12H30M, T24H.

InterpolationType

The interpolation type to be applied. Default will be linear (Piecewise Linear).

Enums
INTERPOLATION_TYPE_UNSPECIFIED Interpolation type is unspecified. In this case, it defaults to Linear.
LINEAR Piecewise linear interpolation will be applied.

ControlPoint

The control points used to define the curve. The curve defined through these control points can only be monotonically increasing or decreasing(constant values are acceptable).

JSON representation 
{
  "attributeValue": string,
  "boostAmount": number
}
Fields
attributeValue

string

Can be one of: 1. The numerical field value. 2. The duration spec for freshness: The value must be formatted as an XSD dayTimeDuration value (a restricted subset of an ISO 8601 duration value). The pattern for this is: [nD][T[nH][nM][nS]].
boostAmount

number

The value between -1 to 1 by which to boost the score if the attributeValue evaluates to the value specified above.

FacetSpec

A facet specification to perform faceted search.

JSON representation 
{
  "facetKey": {
    object (FacetKey)
  },
  "limit": integer,
  "excludedFilterKeys": [
    string
  ],
  "enableDynamicPosition": boolean
}
Fields
facetKey

object (FacetKey)

Required. The facet key specification.
limit

integer

Maximum facet values that are returned for this facet. If unspecified, defaults to 20. The maximum allowed value is 300. Values above 300 are coerced to 300. For aggregation in healthcare search, when the [FacetKey.key] is "healthcare_aggregation_key", the limit will be overridden to 10,000 internally, regardless of the value set here.

If this field is negative, an INVALID_ARGUMENT is returned.
excludedFilterKeys[]

string

List of keys to exclude when faceting.

By default, FacetKey.key is not excluded from the filter unless it is listed in this field.

Listing a facet key in this field allows its values to appear as facet results, even when they are filtered out of search results. Using this field does not affect what search results are returned.

For example, suppose there are 100 documents with the color facet "Red" and 200 documents with the color facet "Blue". A query containing the filter "color:ANY("Red")" and having "color" as FacetKey.key would by default return only "Red" documents in the search results, and also return "Red" with count 100 as the only color facet. Although there are also blue documents available, "Blue" would not be shown as an available facet value.

If "color" is listed in "excludedFilterKeys", then the query returns the facet values "Red" with count 100 and "Blue" with count 200, because the "color" key is now excluded from the filter. Because this field doesn't affect search results, the search results are still correctly filtered to return only "Red" documents.

A maximum of 100 values are allowed. Otherwise, an INVALID_ARGUMENT error is returned.
enableDynamicPosition

boolean

Enables dynamic position for this facet. If set to true, the position of this facet among all facets in the response is determined automatically. If dynamic facets are enabled, it is ordered together. If set to false, the position of this facet in the response is the same as in the request, and it is ranked before the facets with dynamic position enable and all dynamic facets.

For example, you may always want to have rating facet returned in the response, but it's not necessarily to always display the rating facet at the top. In that case, you can set enableDynamicPosition to true so that the position of rating facet in response is determined automatically.

Another example, assuming you have the following facets in the request:

  • "rating", enableDynamicPosition = true

  • "price", enableDynamicPosition = false

  • "brands", enableDynamicPosition = false

And also you have a dynamic facets enabled, which generates a facet gender. Then the final order of the facets in the response can be ("price", "brands", "rating", "gender") or ("price", "brands", "gender", "rating") depends on how API orders "gender" and "rating" facets. However, notice that "price" and "brands" are always ranked at first and second position because their enableDynamicPosition is false.

FacetKey

Specifies how a facet is computed.

JSON representation 
{
  "key": string,
  "intervals": [
    {
      object (Interval)
    }
  ],
  "restrictedValues": [
    string
  ],
  "prefixes": [
    string
  ],
  "contains": [
    string
  ],
  "caseInsensitive": boolean,
  "orderBy": string
}
Fields
key

string

Required. Supported textual and numerical facet keys in Document object, over which the facet values are computed. Facet key is case-sensitive.
intervals[]

object (Interval)

Set only if values should be bucketed into intervals. Must be set for facets with numerical values. Must not be set for facet with text values. Maximum number of intervals is 30.
restrictedValues[]

string

Only get facet for the given restricted values. Only supported on textual fields. For example, suppose "category" has three values "Action > 2022", "Action > 2021" and "Sci-Fi > 2022". If set "restrictedValues" to "Action > 2022", the "category" facet only contains "Action > 2022". Only supported on textual fields. Maximum is 10.
prefixes[]

string

Only get facet values that start with the given string prefix. For example, suppose "category" has three values "Action > 2022", "Action > 2021" and "Sci-Fi > 2022". If set "prefixes" to "Action", the "category" facet only contains "Action > 2022" and "Action > 2021". Only supported on textual fields. Maximum is 10.
contains[]

string

Only get facet values that contain the given strings. For example, suppose "category" has three values "Action > 2022", "Action > 2021" and "Sci-Fi > 2022". If set "contains" to "2022", the "category" facet only contains "Action > 2022" and "Sci-Fi > 2022". Only supported on textual fields. Maximum is 10.
caseInsensitive

boolean

True to make facet keys case insensitive when getting faceting values with prefixes or contains; false otherwise.
orderBy

string

The order in which documents are returned.

Allowed values are:

If not set, textual values are sorted in natural order; numerical intervals are sorted in the order given by FacetSpec.FacetKey.intervals.

Interval

A floating point interval.

JSON representation 
{

  // Union field min can be only one of the following:
  "minimum": number,
  "exclusiveMinimum": number
  // End of list of possible types for union field min.

  // Union field max can be only one of the following:
  "maximum": number,
  "exclusiveMaximum": number
  // End of list of possible types for union field max.
}
Fields

Union field min. The lower bound of the interval. If neither of the min fields are set, then the lower bound is negative infinity.

This field must be not larger than max. Otherwise, an INVALID_ARGUMENT error is returned. min can be only one of the following:
minimum

number

Inclusive lower bound.
exclusiveMinimum

number

Exclusive lower bound.

Union field max. The upper bound of the interval. If neither of the max fields are set, then the upper bound is positive infinity.

This field must be not smaller than min. Otherwise, an INVALID_ARGUMENT error is returned. max can be only one of the following:
maximum

number

Inclusive upper bound.
exclusiveMaximum

number

Exclusive upper bound.

QueryExpansionSpec

Specification to determine under which conditions query expansion should occur.

JSON representation 
{
  "condition": enum (Condition),
  "pinUnexpandedResults": boolean
}
Fields
condition

enum (Condition)

The condition under which query expansion should occur. Default to Condition.DISABLED.
pinUnexpandedResults

boolean

Whether to pin unexpanded results. If this field is set to true, unexpanded products are always at the top of the search results, followed by the expanded results.

Condition

Enum describing under which condition query expansion should occur.

Enums
CONDITION_UNSPECIFIED Unspecified query expansion condition. In this case, server behavior defaults to Condition.DISABLED.
DISABLED Disabled query expansion. Only the exact search query is used, even if SearchResponse.total_size is zero.
AUTO Automatic query expansion built by the Search API.

SpellCorrectionSpec

The specification for query spell correction.

JSON representation 
{
  "mode": enum (Mode)
}
Fields
mode

enum (Mode)

The mode under which spell correction replaces the original search query. Defaults to Mode.AUTO.

Mode

Enum describing under which mode spell correction should occur.

Enums
MODE_UNSPECIFIED Unspecified spell correction mode. In this case, server behavior defaults to Mode.AUTO.
SUGGESTION_ONLY Search API tries to find a spelling suggestion. If a suggestion is found, it is put in the SearchResponse.corrected_query. The spelling suggestion won't be used as the search query.
AUTO Automatic spell correction built by the Search API. Search will be based on the corrected query if found.

EmbeddingSpec

The specification that uses customized query embedding vector to do semantic document retrieval.

JSON representation 
{
  "embeddingVectors": [
    {
      object (EmbeddingVector)
    }
  ]
}
Fields
embeddingVectors[]

object (EmbeddingVector)

The embedding vector used for retrieval. Limit to 1.

EmbeddingVector

Embedding vector.

JSON representation 
{
  "fieldPath": string,
  "vector": [
    number
  ]
}
Fields
fieldPath

string

Embedding field path in schema.
vector[]

number

Query embedding vector.

NaturalLanguageQueryUnderstandingSpec

Specification to enable natural language understanding capabilities for search requests.

JSON representation 
{
  "filterExtractionCondition": enum (FilterExtractionCondition),
  "geoSearchQueryDetectionFieldNames": [
    string
  ]
}
Fields
filterExtractionCondition

enum (FilterExtractionCondition)

The condition under which filter extraction should occur. Default to [Condition.DISABLED][].
geoSearchQueryDetectionFieldNames[]

string

Field names used for location-based filtering, where geolocation filters are detected in natural language search queries. Only valid when the FilterExtractionCondition is set to ENABLED.

If this field is set, it overrides the field names set in ServingConfig.geo_search_query_detection_field_names.

FilterExtractionCondition

Enum describing under which condition filter extraction should occur.

Enums
CONDITION_UNSPECIFIED Server behavior defaults to [Condition.DISABLED][].
DISABLED Disables NL filter extraction.
ENABLED Enables NL filter extraction.

SearchAsYouTypeSpec

Specification for search as you type in search requests.

JSON representation 
{
  "condition": enum (Condition)
}
Fields
condition

enum (Condition)

The condition under which search as you type should occur. Default to Condition.DISABLED.

Condition

Enum describing under which condition search as you type should occur.

Enums
CONDITION_UNSPECIFIED Server behavior defaults to Condition.DISABLED.
DISABLED Disables Search As You Type.
ENABLED Enables Search As You Type.

SessionSpec

Session specification.

Multi-turn Search feature is currently at private GA stage. Please use v1alpha or v1beta version instead before we launch this feature to public GA. Or ask for allowlisting through Google Support team.

JSON representation 
{
  "queryId": string,
  "searchResultPersistenceCount": integer
}
Fields
queryId

string

If set, the search result gets stored to the "turn" specified by this query ID.

Example: Let's say the session looks like this: session { name: ".../sessions/xxx" turns { query { text: "What is foo?" queryId: ".../questions/yyy" } answer: "Foo is ..." } turns { query { text: "How about bar then?" queryId: ".../questions/zzz" } } }

The user can call /search API with a request like this:

session: ".../sessions/xxx" sessionSpec { queryId: ".../questions/zzz" }

Then, the API stores the search result, associated with the last turn. The stored search result can be used by a subsequent /answer API call (with the session ID and the query ID specified). Also, it is possible to call /search and /answer in parallel with the same session ID & query ID.
searchResultPersistenceCount

integer

The number of top search results to persist. The persisted search results can be used for the subsequent /answer api call.

This field is simliar to the summaryResultCount field in SearchRequest.ContentSearchSpec.SummarySpec.summary_result_count.

At most 10 results for documents mode, or 50 for chunks mode.

RelevanceThreshold

The relevance threshold of the search results. The higher relevance threshold is, the higher relevant results are shown and the less number of results are returned.

Enums
RELEVANCE_THRESHOLD_UNSPECIFIED Default value. In this case, server behavior defaults to Google defined threshold.
LOWEST Lowest relevance threshold.
LOW Low relevance threshold.
MEDIUM Medium relevance threshold.
HIGH High relevance threshold.

QuerySetSpec

Describes the specification of the query set.

JSON representation 
{
  "sampleQuerySet": string
}
Fields
sampleQuerySet

string

Required. The full resource name of the SampleQuerySet used for the evaluation, in the format of projects/{project}/locations/{location}/sampleQuerySets/{sampleQuerySet}.

QualityMetrics

Describes the metrics produced by the evaluation.

JSON representation 
{
  "docRecall": {
    object (TopkMetrics)
  },
  "docPrecision": {
    object (TopkMetrics)
  },
  "docNdcg": {
    object (TopkMetrics)
  },
  "pageRecall": {
    object (TopkMetrics)
  },
  "pageNdcg": {
    object (TopkMetrics)
  }
}
Fields
docRecall

object (TopkMetrics)

Recall per document, at various top-k cutoff levels.

Recall is the fraction of relevant documents retrieved out of all relevant documents.

Example (top-5): * For a single SampleQuery, If 3 out of 5 relevant documents are retrieved in the top-5, recall@5 = 3/5 = 0.6
docPrecision

object (TopkMetrics)

Precision per document, at various top-k cutoff levels.

Precision is the fraction of retrieved documents that are relevant.

Example (top-5): * For a single SampleQuery, If 4 out of 5 retrieved documents in the top-5 are relevant, precision@5 = 4/5 = 0.8
docNdcg

object (TopkMetrics)

Normalized discounted cumulative gain (NDCG) per document, at various top-k cutoff levels.

NDCG measures the ranking quality, giving higher relevance to top results.

Example (top-3): Suppose SampleQuery with three retrieved documents (D1, D2, D3) and binary relevance judgements (1 for relevant, 0 for not relevant):

Retrieved: [D3 (0), D1 (1), D2 (1)] Ideal: [D1 (1), D2 (1), D3 (0)]

Calculate NDCG@3 for each SampleQuery: * DCG@3: 0/log2(1+1) + 1/log2(2+1) + 1/log2(3+1) = 1.13 * Ideal DCG@3: 1/log2(1+1) + 1/log2(2+1) + 0/log2(3+1) = 1.63 * NDCG@3: 1.13/1.63 = 0.693
pageRecall

object (TopkMetrics)

Recall per page, at various top-k cutoff levels.

Recall is the fraction of relevant pages retrieved out of all relevant pages.

Example (top-5): * For a single SampleQuery, if 3 out of 5 relevant pages are retrieved in the top-5, recall@5 = 3/5 = 0.6
pageNdcg

object (TopkMetrics)

Normalized discounted cumulative gain (NDCG) per page, at various top-k cutoff levels.

NDCG measures the ranking quality, giving higher relevance to top results.

Example (top-3): Suppose SampleQuery with three retrieved pages (P1, P2, P3) and binary relevance judgements (1 for relevant, 0 for not relevant):

Retrieved: [P3 (0), P1 (1), P2 (1)] Ideal: [P1 (1), P2 (1), P3 (0)]

Calculate NDCG@3 for SampleQuery: * DCG@3: 0/log2(1+1) + 1/log2(2+1) + 1/log2(3+1) = 1.13 * Ideal DCG@3: 1/log2(1+1) + 1/log2(2+1) + 0/log2(3+1) = 1.63 * NDCG@3: 1.13/1.63 = 0.693

TopkMetrics

Stores the metric values at specific top-k levels.

JSON representation 
{
  "top1": number,
  "top3": number,
  "top5": number,
  "top10": number
}
Fields
top1

number

The top-1 value.
top3

number

The top-3 value.
top5

number

The top-5 value.
top10

number

The top-10 value.

State

Describes the state of an evaluation.

Enums
STATE_UNSPECIFIED The evaluation is unspecified.
PENDING The service is preparing to run the evaluation.
RUNNING The evaluation is in progress.
SUCCEEDED The evaluation completed successfully.
FAILED The evaluation failed.

Methods

create

 Creates a Evaluation.

get

 Gets a Evaluation.

list

 Gets a list of Evaluations.

listResults

 Gets a list of results for a given a Evaluation.