Guía de inicio rápido: Flujo de trabajo de evaluación de IA generativa

En esta página, se muestra cómo realizar una evaluación basada en modelos con Gen AI Evaluation Service usando el SDK de Vertex AI para Python.

Antes de comenzar

Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.
In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

Go to project selector

Make sure that billing is enabled for your Google Cloud project.

In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

Go to project selector

Make sure that billing is enabled for your Google Cloud project.
Instala el SDK de Vertex AI para Python con la dependencia de Gen AI Evaluation Service:
```
!pip install google-cloud-aiplatform[evaluation]
```
Configura tus credenciales. Si ejecutas esta guía de inicio rápido en Colaboratory, ejecuta lo siguiente:
```
from google.colab import auth
auth.authenticate_user()
```
Para otros entornos, consulta Autentícate en Vertex AI.

Importa las bibliotecas

Importa tus bibliotecas y configura tu proyecto y ubicación.

import pandas as pd

import vertexai
from vertexai.evaluation import EvalTask, PointwiseMetric, PointwiseMetricPromptTemplate

PROJECT_ID = "PROJECT_ID"
LOCATION = "LOCATION"
EXPERIMENT_NAME = "EXPERIMENT_NAME"

vertexai.init(
    project=PROJECT_ID,
    location=LOCATION,
)

Configura métricas de evaluación según tus criterios

La siguiente definición de métrica evalúa la calidad del texto generado a partir de un modelo grande de lenguaje según dos criterios: Fluency y Entertaining. El código define una métrica llamada custom_text_quality con esos dos criterios:

custom_text_quality = PointwiseMetric(
    metric="custom_text_quality",
    metric_prompt_template=PointwiseMetricPromptTemplate(
        criteria={
            "fluency": (
                "Sentences flow smoothly and are easy to read, avoiding awkward"
                " phrasing or run-on sentences. Ideas and sentences connect"
                " logically, using transitions effectively where needed."
            ),
            "entertaining": (
                "Short, amusing text that incorporates emojis, exclamations and"
                " questions to convey quick and spontaneous communication and"
                " diversion."
            ),
        },
        rating_rubric={
            "1": "The response performs well on both criteria.",
            "0": "The response is somewhat aligned with both criteria",
            "-1": "The response falls short on both criteria",
        },
    ),
)

Prepara tu conjunto de datos

Agrega el siguiente código para preparar tu conjunto de datos:

responses = [
    # An example of good custom_text_quality
    "Life is a rollercoaster, full of ups and downs, but it's the thrill that keeps us coming back for more!",
    # An example of medium custom_text_quality
    "The weather is nice today, not too hot, not too cold.",
    # An example of poor custom_text_quality
    "The weather is, you know, whatever.",
]

eval_dataset = pd.DataFrame({
    "response" : responses,
})

Ejecuta la evaluación con tu conjunto de datos

Ejecuta la evaluación:

eval_task = EvalTask(
    dataset=eval_dataset,
    metrics=[custom_text_quality],
    experiment=EXPERIMENT_NAME
)

pointwise_result = eval_task.evaluate()

Observa los resultados de la evaluación para cada respuesta en metrics_table:

pointwise_result.metrics_table

Realiza una limpieza

Sigue estos pasos para evitar que se apliquen cargos a tu cuenta de Google Cloud por los recursos que usaste en esta página.

Borra el ExperimentRun que creó la evaluación:

vertexai.ExperimentRun(
    run_name=pointwise_result.metadata["experiment_run"],
    experiment=pointwise_result.metadata["experiment"],
).delete()