Metric

The metric used for running evaluations.

Fields
aggregationMetrics[] enum (AggregationMetric)

Optional. The aggregation metrics to use.

metric_spec Union type
The spec for the metric. It would be either a pre-defined metric, or a inline metric spec. metric_spec can be only one of the following:
predefinedMetricSpec object (PredefinedMetricSpec)

The spec for a pre-defined metric.

llmBasedMetricSpec object (LLMBasedMetricSpec)

Spec for an LLM based metric.

pointwiseMetricSpec object (PointwiseMetricSpec)

Spec for pointwise metric.

pairwiseMetricSpec object (PairwiseMetricSpec)

Spec for pairwise metric.

exactMatchSpec object (ExactMatchSpec)

Spec for exact match metric.

bleuSpec object (BleuSpec)

Spec for bleu metric.

rougeSpec object (RougeSpec)

Spec for rouge metric.

JSON representation 
{
  "aggregationMetrics": [
    enum (AggregationMetric)
  ],

  // metric_spec
  "predefinedMetricSpec": {
    object (PredefinedMetricSpec)
  },
  "llmBasedMetricSpec": {
    object (LLMBasedMetricSpec)
  },
  "pointwiseMetricSpec": {
    object (PointwiseMetricSpec)
  },
  "pairwiseMetricSpec": {
    object (PairwiseMetricSpec)
  },
  "exactMatchSpec": {
    object (ExactMatchSpec)
  },
  "bleuSpec": {
    object (BleuSpec)
  },
  "rougeSpec": {
    object (RougeSpec)
  }
  // Union type
}

LLMBasedMetricSpec

Specification for an LLM based metric.

Fields
rubrics_source Union type
Source of the rubrics to be used for evaluation. rubrics_source can be only one of the following:
rubricGroupKey string

Use a pre-defined group of rubrics associated with the input. Refers to a key in the rubricGroups map of EvaluationInstance.

rubricGenerationSpec object (RubricGenerationSpec)

Dynamically generate rubrics using this specification.

predefinedRubricGenerationSpec object (PredefinedMetricSpec)

Dynamically generate rubrics using a predefined spec.

metricPromptTemplate string

Required. Template for the prompt sent to the judge model.

systemInstruction string

Optional. System instructions for the judge model.

judgeAutoraterConfig object (AutoraterConfig)

Optional. Optional configuration for the judge LLM (Autorater).

additionalConfig object (Struct format)

Optional. Optional additional configuration for the metric.

JSON representation 
{

  // rubrics_source
  "rubricGroupKey": string,
  "rubricGenerationSpec": {
    object (RubricGenerationSpec)
  },
  "predefinedRubricGenerationSpec": {
    object (PredefinedMetricSpec)
  }
  // Union type
  "metricPromptTemplate": string,
  "systemInstruction": string,
  "judgeAutoraterConfig": {
    object (AutoraterConfig)
  },
  "additionalConfig": {
    object
  }
}

AggregationMetric

The aggregation metrics supported by EvaluationService.EvaluateDataset.

Enums
AGGREGATION_METRIC_UNSPECIFIED Unspecified aggregation metric.
AVERAGE Average aggregation metric. Not supported for Pairwise metric.
MODE Mode aggregation metric.
STANDARD_DEVIATION Standard deviation aggregation metric. Not supported for pairwise metric.
VARIANCE Variance aggregation metric. Not supported for pairwise metric.
MINIMUM Minimum aggregation metric. Not supported for pairwise metric.
MAXIMUM Maximum aggregation metric. Not supported for pairwise metric.
MEDIAN Median aggregation metric. Not supported for pairwise metric.
PERCENTILE_P90 90th percentile aggregation metric. Not supported for pairwise metric.
PERCENTILE_P95 95th percentile aggregation metric. Not supported for pairwise metric.
PERCENTILE_P99 99th percentile aggregation metric. Not supported for pairwise metric.