Resource: EvaluationJob
Defines an evaluation job that runs periodically to generate Evaluations
. Creating an evaluation job is the starting point for using continuous evaluation.
JSON representation | |
---|---|
{ "name": string, "description": string, "state": enum ( |
Fields | |
---|---|
name |
Output only. After you create a job, Data Labeling Service assigns a name to the job with the following format: "projects/{project_id}/evaluationJobs/{evaluation_job_id}" |
description |
Required. Description of the job. The description can be up to 25,000 characters long. |
state |
Output only. Describes the current state of the job. |
schedule |
Required. Describes the interval at which the job runs. This interval must be at least 1 day, and it is rounded to the nearest day. For example, if you specify a 50-hour interval, the job runs every 2 days. You can provide the schedule in crontab format or in an English-like format. Regardless of what you specify, the job will run at 10:00 AM UTC. Only the interval from this schedule is used, not the specific time of day. |
modelVersion |
Required. The AI Platform Prediction model version to be evaluated. Prediction input and output is sampled from this model version. When creating an evaluation job, specify the model version in the following format: "projects/{project_id}/models/{model_name}/versions/{version_name}" There can only be one evaluation job per model version. |
evaluationJobConfig |
Required. Configuration details for the evaluation job. |
annotationSpecSet |
Required. Name of the "projects/{project_id}/annotationSpecSets/{annotation_spec_set_id}" |
labelMissingGroundTruth |
Required. Whether you want Data Labeling Service to provide ground truth labels for prediction input. If you want the service to assign human labelers to annotate your data, set this to |
attempts[] |
Output only. Every time the evaluation job runs and an error occurs, the failed attempt is appended to this array. |
createTime |
Output only. Timestamp of when this evaluation job was created. A timestamp in RFC3339 UTC "Zulu" format, accurate to nanoseconds. Example: |
State
State of the job.
Enums | |
---|---|
STATE_UNSPECIFIED |
|
SCHEDULED |
The job is scheduled to run at the When the job is in this state, it samples prediction input and output from your model version into your BigQuery table as predictions occur. |
RUNNING |
The job is currently running. When the job runs, Data Labeling Service does several things:
If the job remains in this state for a long time, it continues to sample prediction data into your BigQuery table and will run again at the next interval, even if it causes the job to run multiple times in parallel. |
PAUSED |
The job is not sampling prediction input and output into your BigQuery table and it will not run according to its schedule. You can resume the job. |
STOPPED |
The job has this state right before it is deleted. |
EvaluationJobConfig
Configures specific details of how a continuous evaluation job works. Provide this configuration when you create an EvaluationJob.
JSON representation | |
---|---|
{ "inputConfig": { object ( |
Fields | ||
---|---|---|
inputConfig |
Rquired. Details for the sampled prediction input. Within this configuration, there are requirements for several fields:
|
|
evaluationConfig |
Required. Details for calculating evaluation metrics and creating |
|
humanAnnotationConfig |
Optional. Details for human annotation of your data. If you set Note that you must create an |
|
bigqueryImportKeys |
Required. Prediction keys that tell Data Labeling Service where to find the data for evaluation in your BigQuery table. When the service samples prediction input and output from your model version and saves it to BigQuery, the data gets stored as JSON strings in the BigQuery table. These keys tell Data Labeling Service how to parse the JSON. You can provide the following entries in this field:
|
|
exampleCount |
Required. The maximum number of predictions to sample and save to BigQuery during each |
|
exampleSamplePercentage |
Required. Fraction of predictions to sample and save to BigQuery during each |
|
evaluationJobAlertConfig |
Optional. Configuration details for evaluation job alerts. Specify this field if you want to receive email alerts if the evaluation job finds that your predictions have low mean average precision during a run. |
|
Union field human_annotation_request_config . Required. Details for how you want human reviewers to provide ground truth labels. human_annotation_request_config can be only one of the following: |
||
imageClassificationConfig |
Specify this field if your model version performs image classification or general classification.
|
|
boundingPolyConfig |
Specify this field if your model version performs image object detection (bounding box detection).
|
|
textClassificationConfig |
Specify this field if your model version performs text classification.
|
EvaluationJobAlertConfig
Provides details for how an evaluation job sends email alerts based on the results of a run.
JSON representation | |
---|---|
{ "email": string, "minAcceptableMeanAveragePrecision": number } |
Fields | |
---|---|
email |
Required. An email address to send alerts to. |
minAcceptableMeanAveragePrecision |
Required. A number between 0 and 1 that describes a minimum mean average precision threshold. When the evaluation job runs, if it calculates that your model version's predictions from the recent interval have |
Attempt
Records a failed evaluation job run.
JSON representation | |
---|---|
{
"attemptTime": string,
"partialFailures": [
{
object ( |
Fields | |
---|---|
attemptTime |
A timestamp in RFC3339 UTC "Zulu" format, accurate to nanoseconds. Example: |
partialFailures[] |
Details of errors that occurred. |
Methods |
|
---|---|
|
Creates an evaluation job. |
|
Stops and deletes an evaluation job. |
|
Gets an evaluation job by resource name. |
|
Lists all evaluation jobs within a project with possible filters. |
|
Updates an evaluation job. |
|
Pauses an evaluation job. |
|
Resumes a paused evaluation job. |