Evaluating models

After training a model, AutoML Video Intelligence uses items from the TEST set to evaluate the quality and accuracy of the new model.

AutoML Video Intelligence provides an aggregate set of evaluation metrics indicating how well the model performs overall, as well as evaluation metrics for each category label, indicating how well the model performs for that label.

  • AuPRC : Area under Precision/Recall curve, also referred to as "average precision." Generally between 0.5 and 1.0. Higher values indicate more accurate models.

  • The Confidence threshold curves show how different confidence thresholds would affect precision, recall, true and false positive rates. Read about the relationship of precision and recall.

Use this data to evaluate your model's readiness. Low AUC scores, or low precision and recall scores can indicate that your model needs additional training data or has inconsistent labels. A very high AUC score and perfect precision and recall can indicate that the data is too easy and may not generalize well.

Get model evaluation values

Web UI

  1. Open the Models page in the AutoML Video UI.

  2. Click the row for the model you want to evaluate.

  3. Click the Evaluate tab.

    If training has completed for the model, AutoML Video shows its evaluation metrics.

    Evaluate tab with model information
  4. To view metrics for a specific label, select the label name from the list of labels in the lower part of the page.

Command-line

  • Replace model-name with the full name of your model, from the response when you created the model. The full name has the format: projects/{project-id}/locations/us-central1/models/{model-id}
curl -H "Authorization: Bearer $(gcloud auth application-default print-access-token)" \
  -H "Content-Type: application/json" \
  https://automl.googleapis.com/v1beta1/model-name/modelEvaluations

The response includes a ModelEvaluation resource for the overall model.

{
  "modelEvaluation": [
    {
      "name": "projects/674030654376/locations/us-central1/models/VCN6212715685957599232/modelEvaluations/2308399322008336298",
      "annotationSpecId": "-1",
      "createTime": "2018-10-24T06:03:08.042671Z",
      "classificationEvaluationMetrics": {
        "auPrc": 0.79087126,
        "confidenceMetricsEntry": [
          {
            "recall": 1,
            "precision": 0.027764726
          },
          {
            "confidenceThreshold": 8.0779483e-07,
            "recall": 0.99538463,
            "precision": 0.027640123
          },
          {
            "confidenceThreshold": 9.632977e-07,
            "recall": 0.99538463,
            "precision": 0.027640713
          },
          …
          …
          {
            "confidenceThreshold": 0.9999999,
            "recall": 0.005384615,
            "precision": 1
          },
          {
            "confidenceThreshold": 1,
            "recall": 0.0038461538,
            "precision": 1
          },
          {
            "confidenceThreshold": 1,
            "precision": 1
          }
        ]
      }
    }
  ]
}

Java

/**
 * Demonstrates using the AutoML client to get model evaluations.
 *
 * @param projectId the Id of the project.
 * @param computeRegion the Region name. (e.g., "us-central1")
 * @param modelId the Id of the model.
 * @param modelEvaluationId the Id of your model evaluation.
 * @throws IOException
 */
public static void getModelEvaluation(
    String projectId, String computeRegion, String modelId, String modelEvaluationId)
    throws IOException {
  // Instantiates a client
  AutoMlClient client = AutoMlClient.create();

  // Get the full path of the model evaluation.
  ModelEvaluationName modelEvaluationFullId =
      ModelEvaluationName.of(projectId, computeRegion, modelId, modelEvaluationId);

  // Get complete detail of the model evaluation.
  ModelEvaluation response = client.getModelEvaluation(modelEvaluationFullId);

  // Display the model evaluations information.
  System.out.println(String.format("Model evaluation name: %s", response.getName()));
  System.out.println(
      String.format(
          "Model evaluation Id: %s",
          response.getName().split("/")[response.getName().split("/").length - 1]));
  System.out.println(
      String.format("Model evaluation annotation spec Id: %s", response.getAnnotationSpecId()));
  System.out.println(
      String.format("Model evaluation example count: %s", response.getEvaluatedExampleCount()));
  System.out.println(
      String.format("Model evaluation display name: %s", response.getDisplayName()));
  DateFormat dateFormat = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss.SSSZ");
  String createTime =
      dateFormat.format(new java.util.Date(response.getCreateTime().getSeconds() * 1000));
  System.out.println(String.format("Model create time: %s", createTime));

  System.out.println("Video classification evaluation metrics:");
  ClassificationEvaluationMetrics classMetrics = response.getClassificationEvaluationMetrics();
  System.out.println(String.format("\tModel au_prc: %f ", classMetrics.getAuPrc()));
  List<ConfidenceMetricsEntry> confidenceMetricsEntries =
      classMetrics.getConfidenceMetricsEntryList();

  // Showing video classification evaluation metrics
  System.out.println("\tConfidence metrics entries:");
  for (ConfidenceMetricsEntry confidenceMetricsEntry : confidenceMetricsEntries) {
    System.out.println(
        String.format(
            "\t\tModel confidence threshold: %f ",
            confidenceMetricsEntry.getConfidenceThreshold()));
    System.out.println(
        String.format("\t\tModel precision: %.2f ", confidenceMetricsEntry.getPrecision() * 100)
            + '%');
    System.out.println(
        String.format("\t\tModel recall: %.2f ", confidenceMetricsEntry.getRecall() * 100) + '%');
    System.out.println(
        String.format("\t\tModel f1 score: %.2f ", confidenceMetricsEntry.getF1Score() * 100)
            + '%');
    System.out.println(
        String.format(
                "\t\tModel precision@1: %.2f ", confidenceMetricsEntry.getPrecisionAt1() * 100)
            + '%');
    System.out.println(
        String.format("\t\tModel recall@1: %.2f ", confidenceMetricsEntry.getRecallAt1() * 100)
            + '%');
    System.out.println(
        String.format("\t\tModel f1 score@1: %.2f ", confidenceMetricsEntry.getF1ScoreAt1() * 100)
            + '%'
            + '\n');
  }
}

Node.js

const automl = require(`@google-cloud/automl`);
const math = require(`mathjs`);
const client = new automl.v1beta1.AutoMlClient();

/**
 * Demonstrates using the AutoML client to get model evaluations.
 * TODO(developer): Uncomment the following lines before running the sample.
 */
// const projectId = '[PROJECT_ID]' e.g., "my-gcloud-project";
// const computeRegion = '[REGION_NAME]' e.g., "us-central1";
// const modelId = '[MODEL_ID]' e.g., "VCN7209576908164431872";
// const modelEvaluationId = '[MODEL_EVALUATION_ID]'
// e.g., "3806191078210741236";

// Get the full path of the model evaluation.
const modelEvaluationFullId = client.modelEvaluationPath(
  projectId,
  computeRegion,
  modelId,
  modelEvaluationId
);

// Get complete detail of the model evaluation.
client
  .getModelEvaluation({name: modelEvaluationFullId})
  .then(responses => {
    const response = responses[0];

    const confidenceMetricsEntries =
      response.classificationEvaluationMetrics.confidenceMetricsEntry;

    // Display the model evaluations information.
    console.log(`\nModel evaluation name: ${response.name}`);
    console.log(
      `Model evaluation Id: ${response.name
        .split(`/`)
        .slice(-1)
        .pop()}`
    );
    console.log(
      `Model evaluation annotation spec Id: ${response.annotationSpecId}`
    );
    console.log(`Model evaluation display name: ${response.displayName}`);
    console.log(
      `Model evaluation example count: ${response.evaluatedExampleCount}`
    );
    console.log(`Video classification evaluation metrics:`);
    console.log(
      `\tModel auPrc: ${math.round(
        response.classificationEvaluationMetrics.auPrc,
        6
      )}`
    );
    console.log(`\tConfidence metrics entries:`);

    for (const confidenceMetricsEntry of confidenceMetricsEntries) {
      console.log(
        `\t\tModel confidenceThreshold: ${math.round(
          confidenceMetricsEntry.confidenceThreshold,
          6
        )}`
      );
      console.log(
        `\t\tModel recall: ${math.round(
          confidenceMetricsEntry.recall * 100,
          2
        )} %`
      );
      console.log(
        `\t\tModel precision: ${math.round(
          confidenceMetricsEntry.precision * 100,
          2
        )} %`
      );
      console.log(
        `\t\tModel f1 score: ${math.round(
          confidenceMetricsEntry.f1Score * 100,
          2
        )} %`
      );
      console.log(
        `\t\tModel recall@1: ${math.round(
          confidenceMetricsEntry.recallAt1 * 100,
          2
        )} %`
      );
      console.log(
        `\t\tModel precision@1: ${math.round(
          confidenceMetricsEntry.precisionAt1 * 100,
          2
        )} %`
      );
      console.log(
        `\t\tModel f1 score@1: ${math.round(
          confidenceMetricsEntry.f1ScoreAt1 * 100,
          2
        )} % \n`
      );
    }
  })
  .catch(err => {
    console.error(err);
  });

Python

# TODO(developer): Uncomment and set the following variables
# project_id = 'PROJECT_ID_HERE'
# compute_region = 'COMPUTE_REGION_HERE'
# model_id = 'MODEL_ID_HERE'
# model_evaluation_id = 'MODEL_EVALUATION_ID_HERE'

from google.cloud import automl_v1beta1 as automl

client = automl.AutoMlClient()

# Get the full path of the model evaluation.
model_evaluation_full_id = client.model_evaluation_path(
    project_id, compute_region, model_id, model_evaluation_id
)

# Get complete detail of the model evaluation.
response = client.get_model_evaluation(model_evaluation_full_id)

print(response)

Iterate on your model

If you're not happy with the quality levels, you can go back to earlier steps to improve the quality:

  • You may need to add different types of videos (e.g. wider angle, higher or lower resolution, different points of view).
  • Consider removing labels altogether if you don't have enough training videos.
  • Remember that machines can’t read your label name; it's just a random string of letters to them. If you have one label that says "door" and another that says "door_with_knob" the machine has no way of figuring out the nuance other than the videos you provide it.
  • Augment your data with more examples of true positives and negatives. Especially important examples are the ones that are close to the decision boundary.

Once you've made changes, train and evaluate a new model until you reach a high enough quality level.

Was this page helpful? Let us know how we did:

Send feedback about...

AutoML Video Intelligence Documentation