REST Resource: projects.locations.models.evaluations

Resource: ModelEvaluation

A collection of metrics calculated by comparing Model's predictions on all of the test data against annotations from the test data.

JSON representation
{
  "name": string,
  "metricsSchemaUri": string,
  "metrics": value,
  "createTime": string,
  "sliceDimensions": [
    string
  ],
  "modelExplanation": {
    object (ModelExplanation)
  },
  "explanationSpecs": [
    {
      object (ModelEvaluationExplanationSpec)
    }
  ]
}
Fields
name

string

Output only. The resource name of the ModelEvaluation.

metricsSchemaUri

string

Output only. Points to a YAML file stored on Google Cloud Storage describing the metrics of this ModelEvaluation. The schema is defined as an OpenAPI 3.0.2 Schema Object.

metrics

value (Value format)

Output only. Evaluation metrics of the Model. The schema of the metrics is stored in metricsSchemaUri

createTime

string (Timestamp format)

Output only. Timestamp when this ModelEvaluation was created.

A timestamp in RFC3339 UTC "Zulu" format, with nanosecond resolution and up to nine fractional digits. Examples: "2014-10-02T15:01:23Z" and "2014-10-02T15:01:23.045123456Z".

sliceDimensions[]

string

Output only. All possible [dimensions][ModelEvaluationSlice.slice.dimension] of ModelEvaluationSlices. The dimensions can be used as the filter of the ModelService.ListModelEvaluationSlices request, in the form of slice.dimension = <dimension>.

modelExplanation

object (ModelExplanation)

Output only. Aggregated explanation metrics for the Model's prediction output over the data this ModelEvaluation uses. This field is populated only if the Model is evaluated with explanations, and only for AutoML tabular Models.

explanationSpecs[]

object (ModelEvaluationExplanationSpec)

Output only. Describes the values of ExplanationSpec that are used for explaining the predicted values on the evaluated data.

ModelExplanation

Aggregated explanation metrics for a Model over a set of instances.

JSON representation
{
  "meanAttributions": [
    {
      object (Attribution)
    }
  ]
}
Fields
meanAttributions[]

object (Attribution)

Output only. Aggregated attributions explaining the Model's prediction outputs over the set of instances. The attributions are grouped by outputs.

For Models that predict only one output, such as regression Models that predict only one score, there is only one attibution that explains the predicted output. For Models that predict multiple outputs, such as multiclass Models that predict multiple classes, each element explains one specific item. Attribution.output_index can be used to identify which output this attribution is explaining.

The baselineOutputValue, instanceOutputValue and featureAttributions fields are averaged over the test data.

NOTE: Currently AutoML tabular classification Models produce only one attribution, which averages attributions over all the classes it predicts. Attribution.approximation_error is not populated.

Attribution

Attribution that explains a particular prediction output.

JSON representation
{
  "baselineOutputValue": number,
  "instanceOutputValue": number,
  "featureAttributions": value,
  "outputIndex": [
    integer
  ],
  "outputDisplayName": string,
  "approximationError": number,
  "outputName": string
}
Fields
baselineOutputValue

number

Output only. Model predicted output if the input instance is constructed from the baselines of all the features defined in ExplanationMetadata.inputs. The field name of the output is determined by the key in ExplanationMetadata.outputs.

If the Model's predicted output has multiple dimensions (rank > 1), this is the value in the output located by outputIndex.

If there are multiple baselines, their output values are averaged.

instanceOutputValue

number

Output only. Model predicted output on the corresponding [explanation instance][ExplainRequest.instances]. The field name of the output is determined by the key in ExplanationMetadata.outputs.

If the Model predicted output has multiple dimensions, this is the value in the output located by outputIndex.

featureAttributions

value (Value format)

Output only. Attributions of each explained feature. Features are extracted from the prediction instances according to explanation metadata for inputs.

The value is a struct, whose keys are the name of the feature. The values are how much the feature in the instance contributed to the predicted result.

The format of the value is determined by the feature's input format:

  • If the feature is a scalar value, the attribution value is a floating number.

  • If the feature is an array of scalar values, the attribution value is an array.

  • If the feature is a struct, the attribution value is a struct. The keys in the attribution value struct are the same as the keys in the feature struct. The formats of the values in the attribution struct are determined by the formats of the values in the feature struct.

The ExplanationMetadata.feature_attributions_schema_uri field, pointed to by the ExplanationSpec field of the Endpoint.deployed_models object, points to the schema file that describes the features and their attribution values (if it is populated).

outputIndex[]

integer

Output only. The index that locates the explained prediction output.

If the prediction output is a scalar value, outputIndex is not populated. If the prediction output has multiple dimensions, the length of the outputIndex list is the same as the number of dimensions of the output. The i-th element in outputIndex is the element index of the i-th dimension of the output vector. Indices start from 0.

outputDisplayName

string

Output only. The display name of the output identified by outputIndex. For example, the predicted class name by a multi-classification Model.

This field is only populated iff the Model predicts display names as a separate field along with the explained output. The predicted display name must has the same shape of the explained output, and can be located using outputIndex.

approximationError

number

Output only. Error of featureAttributions caused by approximation used in the explanation method. Lower value means more precise attributions.

See this introduction for more information.

outputName

string

Output only. Name of the explain output. Specified as the key in ExplanationMetadata.outputs.

ModelEvaluationExplanationSpec

JSON representation
{
  "explanationType": string,
  "explanationSpec": {
    object (ExplanationSpec)
  }
}
Fields
explanationType

string

Explanation type.

For AutoML Image Classification models, possible values are:

  • image-integrated-gradients
  • image-xrai
explanationSpec

object (ExplanationSpec)

Explanation spec details.

Methods

get

Gets a ModelEvaluation.

list

Lists ModelEvaluations in a Model.