Method: projects.locations.evaluateInstances

Evaluates instances based on a given metric.

Endpoint

post https://{endpoint}/v1/{location}:evaluateInstances

Where {service-endpoint} is one of the supported service endpoints.

Path parameters

location string

Required. The resource name of the Location to evaluate the instances. Format: projects/{project}/locations/{location}

Request body

The request body contains data with the following structure:

Fields
metric_inputs Union type
Instances and specs for evaluation metric_inputs can be only one of the following:
exactMatchInput object (ExactMatchInput)

Auto metric instances. Instances and metric spec for exact match metric.

bleuInput object (BleuInput)

Instances and metric spec for bleu metric.

rougeInput object (RougeInput)

Instances and metric spec for rouge metric.

fluencyInput object (FluencyInput)

LLM-based metric instance. General text generation metrics, applicable to other categories. Input for fluency metric.

coherenceInput object (CoherenceInput)

Input for coherence metric.

safetyInput object (SafetyInput)

Input for safety metric.

groundednessInput object (GroundednessInput)

Input for groundedness metric.

fulfillmentInput object (FulfillmentInput)

Input for fulfillment metric.

summarizationQualityInput object (SummarizationQualityInput)

Input for summarization quality metric.

pairwiseSummarizationQualityInput object (PairwiseSummarizationQualityInput)

Input for pairwise summarization quality metric.

summarizationHelpfulnessInput object (SummarizationHelpfulnessInput)

Input for summarization helpfulness metric.

summarizationVerbosityInput object (SummarizationVerbosityInput)

Input for summarization verbosity metric.

questionAnsweringQualityInput object (QuestionAnsweringQualityInput)

Input for question answering quality metric.

pairwiseQuestionAnsweringQualityInput object (PairwiseQuestionAnsweringQualityInput)

Input for pairwise question answering quality metric.

questionAnsweringRelevanceInput object (QuestionAnsweringRelevanceInput)

Input for question answering relevance metric.

questionAnsweringHelpfulnessInput object (QuestionAnsweringHelpfulnessInput)

Input for question answering helpfulness metric.

questionAnsweringCorrectnessInput object (QuestionAnsweringCorrectnessInput)

Input for question answering correctness metric.

toolCallValidInput object (ToolCallValidInput)

Tool call metric instances. Input for tool call valid metric.

toolNameMatchInput object (ToolNameMatchInput)

Input for tool name match metric.

toolParameterKeyMatchInput object (ToolParameterKeyMatchInput)

Input for tool parameter key match metric.

toolParameterKvMatchInput object (ToolParameterKVMatchInput)

Input for tool parameter key value match metric.

cometInput object (CometInput)

Translation metrics. Input for Comet metric.

trajectoryExactMatchInput object (TrajectoryExactMatchInput)

Input for trajectory exact match metric.

trajectoryInOrderMatchInput object (TrajectoryInOrderMatchInput)

Input for trajectory in order match metric.

trajectoryAnyOrderMatchInput object (TrajectoryAnyOrderMatchInput)

Input for trajectory match any order metric.

trajectoryPrecisionInput object (TrajectoryPrecisionInput)

Input for trajectory precision metric.

trajectoryRecallInput object (TrajectoryRecallInput)

Input for trajectory recall metric.

trajectorySingleToolUseInput object (TrajectorySingleToolUseInput)

Input for trajectory single tool use metric.

Response body

Response message for EvaluationService.EvaluateInstances.

If successful, the response body contains data with the following structure:

Fields
evaluation_results Union type
Evaluation results will be served in the same order as presented in EvaluationRequest.instances. evaluation_results can be only one of the following:
exactMatchResults object (ExactMatchResults)

Auto metric evaluation results. Results for exact match metric.

bleuResults object (BleuResults)

Results for bleu metric.

rougeResults object (RougeResults)

Results for rouge metric.

fluencyResult object (FluencyResult)

LLM-based metric evaluation result. General text generation metrics, applicable to other categories. result for fluency metric.

coherenceResult object (CoherenceResult)

result for coherence metric.

safetyResult object (SafetyResult)

result for safety metric.

groundednessResult object (GroundednessResult)

result for groundedness metric.

fulfillmentResult object (FulfillmentResult)

result for fulfillment metric.

summarizationQualityResult object (SummarizationQualityResult)

Summarization only metrics. result for summarization quality metric.

pairwiseSummarizationQualityResult object (PairwiseSummarizationQualityResult)

result for pairwise summarization quality metric.

summarizationHelpfulnessResult object (SummarizationHelpfulnessResult)

result for summarization helpfulness metric.

summarizationVerbosityResult object (SummarizationVerbosityResult)

result for summarization verbosity metric.

questionAnsweringQualityResult object (QuestionAnsweringQualityResult)

Question answering only metrics. result for question answering quality metric.

pairwiseQuestionAnsweringQualityResult object (PairwiseQuestionAnsweringQualityResult)

result for pairwise question answering quality metric.

questionAnsweringRelevanceResult object (QuestionAnsweringRelevanceResult)

result for question answering relevance metric.

questionAnsweringHelpfulnessResult object (QuestionAnsweringHelpfulnessResult)

result for question answering helpfulness metric.

questionAnsweringCorrectnessResult object (QuestionAnsweringCorrectnessResult)

result for question answering correctness metric.

toolCallValidResults object (ToolCallValidResults)

Tool call metrics. Results for tool call valid metric.

toolNameMatchResults object (ToolNameMatchResults)

Results for tool name match metric.

toolParameterKeyMatchResults object (ToolParameterKeyMatchResults)

Results for tool parameter key match metric.

toolParameterKvMatchResults object (ToolParameterKVMatchResults)

Results for tool parameter key value match metric.

cometResult object (CometResult)

Translation metrics. result for Comet metric.

trajectoryExactMatchResults object (TrajectoryExactMatchResults)

result for trajectory exact match metric.

trajectoryInOrderMatchResults object (TrajectoryInOrderMatchResults)

result for trajectory in order match metric.

trajectoryAnyOrderMatchResults object (TrajectoryAnyOrderMatchResults)

result for trajectory any order match metric.

trajectoryPrecisionResults object (TrajectoryPrecisionResults)

result for trajectory precision metric.

trajectoryRecallResults object (TrajectoryRecallResults)

Results for trajectory recall metric.

trajectorySingleToolUseResults object (TrajectorySingleToolUseResults)

Results for trajectory single tool use metric.

JSON representation 
{

  // evaluation_results
  "exactMatchResults": {
    object (ExactMatchResults)
  },
  "bleuResults": {
    object (BleuResults)
  },
  "rougeResults": {
    object (RougeResults)
  },
  "fluencyResult": {
    object (FluencyResult)
  },
  "coherenceResult": {
    object (CoherenceResult)
  },
  "safetyResult": {
    object (SafetyResult)
  },
  "groundednessResult": {
    object (GroundednessResult)
  },
  "fulfillmentResult": {
    object (FulfillmentResult)
  },
  "summarizationQualityResult": {
    object (SummarizationQualityResult)
  },
  "pairwiseSummarizationQualityResult": {
    object (PairwiseSummarizationQualityResult)
  },
  "summarizationHelpfulnessResult": {
    object (SummarizationHelpfulnessResult)
  },
  "summarizationVerbosityResult": {
    object (SummarizationVerbosityResult)
  },
  "questionAnsweringQualityResult": {
    object (QuestionAnsweringQualityResult)
  },
  "pairwiseQuestionAnsweringQualityResult": {
    object (PairwiseQuestionAnsweringQualityResult)
  },
  "questionAnsweringRelevanceResult": {
    object (QuestionAnsweringRelevanceResult)
  },
  "questionAnsweringHelpfulnessResult": {
    object (QuestionAnsweringHelpfulnessResult)
  },
  "questionAnsweringCorrectnessResult": {
    object (QuestionAnsweringCorrectnessResult)
  },
  "toolCallValidResults": {
    object (ToolCallValidResults)
  },
  "toolNameMatchResults": {
    object (ToolNameMatchResults)
  },
  "toolParameterKeyMatchResults": {
    object (ToolParameterKeyMatchResults)
  },
  "toolParameterKvMatchResults": {
    object (ToolParameterKVMatchResults)
  },
  "cometResult": {
    object (CometResult)
  },
  "trajectoryExactMatchResults": {
    object (TrajectoryExactMatchResults)
  },
  "trajectoryInOrderMatchResults": {
    object (TrajectoryInOrderMatchResults)
  },
  "trajectoryAnyOrderMatchResults": {
    object (TrajectoryAnyOrderMatchResults)
  },
  "trajectoryPrecisionResults": {
    object (TrajectoryPrecisionResults)
  },
  "trajectoryRecallResults": {
    object (TrajectoryRecallResults)
  },
  "trajectorySingleToolUseResults": {
    object (TrajectorySingleToolUseResults)
  }
  // Union type
}

ExactMatchInput

Input for exact match metric.

Fields
metricSpec object (ExactMatchSpec)

Required. Spec for exact match metric.

instances[] object (ExactMatchInstance)

Required. Repeated exact match instances.

JSON representation 
{
  "metricSpec": {
    object (ExactMatchSpec)
  },
  "instances": [
    {
      object (ExactMatchInstance)
    }
  ]
}

ExactMatchSpec

This type has no fields.

Spec for exact match metric - returns 1 if prediction and reference exactly matches, otherwise 0.

ExactMatchInstance

Spec for exact match instance.

Fields
prediction string

Required. Output of the evaluated model.

reference string

Required. Ground truth used to compare against the prediction.

JSON representation 
{
  "prediction": string,
  "reference": string
}

BleuInput

Input for bleu metric.

Fields
metricSpec object (BleuSpec)

Required. Spec for bleu score metric.

instances[] object (BleuInstance)

Required. Repeated bleu instances.

JSON representation 
{
  "metricSpec": {
    object (BleuSpec)
  },
  "instances": [
    {
      object (BleuInstance)
    }
  ]
}

BleuSpec

Spec for bleu score metric - calculates the precision of n-grams in the prediction as compared to reference - returns a score ranging between 0 to 1.

Fields
useEffectiveOrder boolean

Optional. Whether to useEffectiveOrder to compute bleu score.

JSON representation 
{
  "useEffectiveOrder": boolean
}

BleuInstance

Spec for bleu instance.

Fields
prediction string

Required. Output of the evaluated model.

reference string

Required. Ground truth used to compare against the prediction.

JSON representation 
{
  "prediction": string,
  "reference": string
}

RougeInput

Input for rouge metric.

Fields
metricSpec object (RougeSpec)

Required. Spec for rouge score metric.

instances[] object (RougeInstance)

Required. Repeated rouge instances.

JSON representation 
{
  "metricSpec": {
    object (RougeSpec)
  },
  "instances": [
    {
      object (RougeInstance)
    }
  ]
}

RougeSpec

Spec for rouge score metric - calculates the recall of n-grams in prediction as compared to reference - returns a score ranging between 0 and 1.

Fields
rougeType string

Optional. Supported rouge types are rougen[1-9], rougeL, and rougeLsum.

useStemmer boolean

Optional. Whether to use stemmer to compute rouge score.

splitSummaries boolean

Optional. Whether to split summaries while using rougeLsum.

JSON representation 
{
  "rougeType": string,
  "useStemmer": boolean,
  "splitSummaries": boolean
}

RougeInstance

Spec for rouge instance.

Fields
prediction string

Required. Output of the evaluated model.

reference string

Required. Ground truth used to compare against the prediction.

JSON representation 
{
  "prediction": string,
  "reference": string
}

FluencyInput

Input for fluency metric.

Fields
metricSpec object (FluencySpec)

Required. Spec for fluency score metric.

instance object (FluencyInstance)

Required. Fluency instance.

JSON representation 
{
  "metricSpec": {
    object (FluencySpec)
  },
  "instance": {
    object (FluencyInstance)
  }
}

FluencySpec

Spec for fluency score metric.

Fields
version integer

Optional. Which version to use for evaluation.

JSON representation 
{
  "version": integer
}

FluencyInstance

Spec for fluency instance.

Fields
prediction string

Required. Output of the evaluated model.

JSON representation 
{
  "prediction": string
}

CoherenceInput

Input for coherence metric.

Fields
metricSpec object (CoherenceSpec)

Required. Spec for coherence score metric.

instance object (CoherenceInstance)

Required. Coherence instance.

JSON representation 
{
  "metricSpec": {
    object (CoherenceSpec)
  },
  "instance": {
    object (CoherenceInstance)
  }
}

CoherenceSpec

Spec for coherence score metric.

Fields
version integer

Optional. Which version to use for evaluation.

JSON representation 
{
  "version": integer
}

CoherenceInstance

Spec for coherence instance.

Fields
prediction string

Required. Output of the evaluated model.

JSON representation 
{
  "prediction": string
}

SafetyInput

Input for safety metric.

Fields
metricSpec object (SafetySpec)

Required. Spec for safety metric.

instance object (SafetyInstance)

Required. Safety instance.

JSON representation 
{
  "metricSpec": {
    object (SafetySpec)
  },
  "instance": {
    object (SafetyInstance)
  }
}

SafetySpec

Spec for safety metric.

Fields
version integer

Optional. Which version to use for evaluation.

JSON representation 
{
  "version": integer
}

SafetyInstance

Spec for safety instance.

Fields
prediction string

Required. Output of the evaluated model.

JSON representation 
{
  "prediction": string
}

GroundednessInput

Input for groundedness metric.

Fields
metricSpec object (GroundednessSpec)

Required. Spec for groundedness metric.

instance object (GroundednessInstance)

Required. Groundedness instance.

JSON representation 
{
  "metricSpec": {
    object (GroundednessSpec)
  },
  "instance": {
    object (GroundednessInstance)
  }
}

GroundednessSpec

Spec for groundedness metric.

Fields
version integer

Optional. Which version to use for evaluation.

JSON representation 
{
  "version": integer
}

GroundednessInstance

Spec for groundedness instance.

Fields
prediction string

Required. Output of the evaluated model.

context string

Required. Background information provided in context used to compare against the prediction.

JSON representation 
{
  "prediction": string,
  "context": string
}

FulfillmentInput

Input for fulfillment metric.

Fields
metricSpec object (FulfillmentSpec)

Required. Spec for fulfillment score metric.

instance object (FulfillmentInstance)

Required. Fulfillment instance.

JSON representation 
{
  "metricSpec": {
    object (FulfillmentSpec)
  },
  "instance": {
    object (FulfillmentInstance)
  }
}

FulfillmentSpec

Spec for fulfillment metric.

Fields
version integer

Optional. Which version to use for evaluation.

JSON representation 
{
  "version": integer
}

FulfillmentInstance

Spec for fulfillment instance.

Fields
prediction string

Required. Output of the evaluated model.

instruction string

Required. Inference instruction prompt to compare prediction with.

JSON representation 
{
  "prediction": string,
  "instruction": string
}

SummarizationQualityInput

Input for summarization quality metric.

Fields
metricSpec object (SummarizationQualitySpec)

Required. Spec for summarization quality score metric.

instance object (SummarizationQualityInstance)

Required. Summarization quality instance.

JSON representation 
{
  "metricSpec": {
    object (SummarizationQualitySpec)
  },
  "instance": {
    object (SummarizationQualityInstance)
  }
}

SummarizationQualitySpec

Spec for summarization quality score metric.

Fields
useReference boolean

Optional. Whether to use instance.reference to compute summarization quality.

version integer

Optional. Which version to use for evaluation.

JSON representation 
{
  "useReference": boolean,
  "version": integer
}

SummarizationQualityInstance

Spec for summarization quality instance.

Fields
prediction string

Required. Output of the evaluated model.

reference string

Optional. Ground truth used to compare against the prediction.

context string

Required. Text to be summarized.

instruction string

Required. Summarization prompt for LLM.

JSON representation 
{
  "prediction": string,
  "reference": string,
  "context": string,
  "instruction": string
}

PairwiseSummarizationQualityInput

Input for pairwise summarization quality metric.

Fields
metricSpec object (PairwiseSummarizationQualitySpec)

Required. Spec for pairwise summarization quality score metric.

instance object (PairwiseSummarizationQualityInstance)

Required. Pairwise summarization quality instance.

JSON representation 
{
  "metricSpec": {
    object (PairwiseSummarizationQualitySpec)
  },
  "instance": {
    object (PairwiseSummarizationQualityInstance)
  }
}

PairwiseSummarizationQualitySpec

Spec for pairwise summarization quality score metric.

Fields
useReference boolean

Optional. Whether to use instance.reference to compute pairwise summarization quality.

version integer

Optional. Which version to use for evaluation.

JSON representation 
{
  "useReference": boolean,
  "version": integer
}

PairwiseSummarizationQualityInstance

Spec for pairwise summarization quality instance.

Fields
prediction string

Required. Output of the candidate model.

baselinePrediction string

Required. Output of the baseline model.

reference string

Optional. Ground truth used to compare against the prediction.

context string

Required. Text to be summarized.

instruction string

Required. Summarization prompt for LLM.

JSON representation 
{
  "prediction": string,
  "baselinePrediction": string,
  "reference": string,
  "context": string,
  "instruction": string
}

SummarizationHelpfulnessInput

Input for summarization helpfulness metric.

Fields
metricSpec object (SummarizationHelpfulnessSpec)

Required. Spec for summarization helpfulness score metric.

instance object (SummarizationHelpfulnessInstance)

Required. Summarization helpfulness instance.

JSON representation 
{
  "metricSpec": {
    object (SummarizationHelpfulnessSpec)
  },
  "instance": {
    object (SummarizationHelpfulnessInstance)
  }
}

SummarizationHelpfulnessSpec

Spec for summarization helpfulness score metric.

Fields
useReference boolean

Optional. Whether to use instance.reference to compute summarization helpfulness.

version integer

Optional. Which version to use for evaluation.

JSON representation 
{
  "useReference": boolean,
  "version": integer
}

SummarizationHelpfulnessInstance

Spec for summarization helpfulness instance.

Fields
prediction string

Required. Output of the evaluated model.

reference string

Optional. Ground truth used to compare against the prediction.

context string

Required. Text to be summarized.

instruction string

Optional. Summarization prompt for LLM.

JSON representation 
{
  "prediction": string,
  "reference": string,
  "context": string,
  "instruction": string
}

SummarizationVerbosityInput

Input for summarization verbosity metric.

Fields
metricSpec object (SummarizationVerbositySpec)

Required. Spec for summarization verbosity score metric.

instance object (SummarizationVerbosityInstance)

Required. Summarization verbosity instance.

JSON representation 
{
  "metricSpec": {
    object (SummarizationVerbositySpec)
  },
  "instance": {
    object (SummarizationVerbosityInstance)
  }
}

SummarizationVerbositySpec

Spec for summarization verbosity score metric.

Fields
useReference boolean

Optional. Whether to use instance.reference to compute summarization verbosity.

version integer

Optional. Which version to use for evaluation.

JSON representation 
{
  "useReference": boolean,
  "version": integer
}

SummarizationVerbosityInstance

Spec for summarization verbosity instance.

Fields
prediction string

Required. Output of the evaluated model.

reference string

Optional. Ground truth used to compare against the prediction.

context string

Required. Text to be summarized.

instruction string

Optional. Summarization prompt for LLM.

JSON representation 
{
  "prediction": string,
  "reference": string,
  "context": string,
  "instruction": string
}

QuestionAnsweringQualityInput

Input for question answering quality metric.

Fields
metricSpec object (QuestionAnsweringQualitySpec)

Required. Spec for question answering quality score metric.

instance object (QuestionAnsweringQualityInstance)

Required. Question answering quality instance.

JSON representation 
{
  "metricSpec": {
    object (QuestionAnsweringQualitySpec)
  },
  "instance": {
    object (QuestionAnsweringQualityInstance)
  }
}

QuestionAnsweringQualitySpec

Spec for question answering quality score metric.

Fields
useReference boolean

Optional. Whether to use instance.reference to compute question answering quality.

version integer

Optional. Which version to use for evaluation.

JSON representation 
{
  "useReference": boolean,
  "version": integer
}

QuestionAnsweringQualityInstance

Spec for question answering quality instance.

Fields
prediction string

Required. Output of the evaluated model.

reference string

Optional. Ground truth used to compare against the prediction.

context string

Required. Text to answer the question.

instruction string

Required. Question Answering prompt for LLM.

JSON representation 
{
  "prediction": string,
  "reference": string,
  "context": string,
  "instruction": string
}

PairwiseQuestionAnsweringQualityInput

Input for pairwise question answering quality metric.

Fields
metricSpec object (PairwiseQuestionAnsweringQualitySpec)

Required. Spec for pairwise question answering quality score metric.

instance object (PairwiseQuestionAnsweringQualityInstance)

Required. Pairwise question answering quality instance.

JSON representation 
{
  "metricSpec": {
    object (PairwiseQuestionAnsweringQualitySpec)
  },
  "instance": {
    object (PairwiseQuestionAnsweringQualityInstance)
  }
}

PairwiseQuestionAnsweringQualitySpec

Spec for pairwise question answering quality score metric.

Fields
useReference boolean

Optional. Whether to use instance.reference to compute question answering quality.

version integer

Optional. Which version to use for evaluation.

JSON representation 
{
  "useReference": boolean,
  "version": integer
}

PairwiseQuestionAnsweringQualityInstance

Spec for pairwise question answering quality instance.

Fields
prediction string

Required. Output of the candidate model.

baselinePrediction string

Required. Output of the baseline model.

reference string

Optional. Ground truth used to compare against the prediction.

context string

Required. Text to answer the question.

instruction string

Required. Question Answering prompt for LLM.

JSON representation 
{
  "prediction": string,
  "baselinePrediction": string,
  "reference": string,
  "context": string,
  "instruction": string
}

QuestionAnsweringRelevanceInput

Input for question answering relevance metric.

Fields
metricSpec object (QuestionAnsweringRelevanceSpec)

Required. Spec for question answering relevance score metric.

instance object (QuestionAnsweringRelevanceInstance)

Required. Question answering relevance instance.

JSON representation 
{
  "metricSpec": {
    object (QuestionAnsweringRelevanceSpec)
  },
  "instance": {
    object (QuestionAnsweringRelevanceInstance)
  }
}

QuestionAnsweringRelevanceSpec

Spec for question answering relevance metric.

Fields
useReference boolean

Optional. Whether to use instance.reference to compute question answering relevance.

version integer

Optional. Which version to use for evaluation.

JSON representation 
{
  "useReference": boolean,
  "version": integer
}

QuestionAnsweringRelevanceInstance

Spec for question answering relevance instance.

Fields
prediction string

Required. Output of the evaluated model.

reference string

Optional. Ground truth used to compare against the prediction.

context string

Optional. Text provided as context to answer the question.

instruction string

Required. The question asked and other instruction in the inference prompt.

JSON representation 
{
  "prediction": string,
  "reference": string,
  "context": string,
  "instruction": string
}

QuestionAnsweringHelpfulnessInput

Input for question answering helpfulness metric.

Fields
metricSpec object (QuestionAnsweringHelpfulnessSpec)

Required. Spec for question answering helpfulness score metric.

instance object (QuestionAnsweringHelpfulnessInstance)

Required. Question answering helpfulness instance.

JSON representation 
{
  "metricSpec": {
    object (QuestionAnsweringHelpfulnessSpec)
  },
  "instance": {
    object (QuestionAnsweringHelpfulnessInstance)
  }
}

QuestionAnsweringHelpfulnessSpec

Spec for question answering helpfulness metric.

Fields
useReference boolean

Optional. Whether to use instance.reference to compute question answering helpfulness.

version integer

Optional. Which version to use for evaluation.

JSON representation 
{
  "useReference": boolean,
  "version": integer
}

QuestionAnsweringHelpfulnessInstance

Spec for question answering helpfulness instance.

Fields
prediction string

Required. Output of the evaluated model.

reference string

Optional. Ground truth used to compare against the prediction.

context string

Optional. Text provided as context to answer the question.

instruction string

Required. The question asked and other instruction in the inference prompt.

JSON representation 
{
  "prediction": string,
  "reference": string,
  "context": string,
  "instruction": string
}

QuestionAnsweringCorrectnessInput

Input for question answering correctness metric.

Fields
metricSpec object (QuestionAnsweringCorrectnessSpec)

Required. Spec for question answering correctness score metric.

instance object (QuestionAnsweringCorrectnessInstance)

Required. Question answering correctness instance.

JSON representation 
{
  "metricSpec": {
    object (QuestionAnsweringCorrectnessSpec)
  },
  "instance": {
    object (QuestionAnsweringCorrectnessInstance)
  }
}

QuestionAnsweringCorrectnessSpec

Spec for question answering correctness metric.

Fields
useReference boolean

Optional. Whether to use instance.reference to compute question answering correctness.

version integer

Optional. Which version to use for evaluation.

JSON representation 
{
  "useReference": boolean,
  "version": integer
}

QuestionAnsweringCorrectnessInstance

Spec for question answering correctness instance.

Fields
prediction string

Required. Output of the evaluated model.

reference string

Optional. Ground truth used to compare against the prediction.

context string

Optional. Text provided as context to answer the question.

instruction string

Required. The question asked and other instruction in the inference prompt.

JSON representation 
{
  "prediction": string,
  "reference": string,
  "context": string,
  "instruction": string
}

ToolCallValidInput

Input for tool call valid metric.

Fields
metricSpec object (ToolCallValidSpec)

Required. Spec for tool call valid metric.

instances[] object (ToolCallValidInstance)

Required. Repeated tool call valid instances.

JSON representation 
{
  "metricSpec": {
    object (ToolCallValidSpec)
  },
  "instances": [
    {
      object (ToolCallValidInstance)
    }
  ]
}

ToolCallValidSpec

This type has no fields.

Spec for tool call valid metric.