EvaluationJobConfig(mapping=None, *, ignore_unknown_fields=False, **kwargs)Configures specific details of how a continuous evaluation job works. Provide this configuration when you create an EvaluationJob.
This message has oneof_ fields (mutually exclusive fields).
For each oneof, at most one member field can be set at the same time.
Setting any member of the oneof automatically clears all other
members.
.. _oneof: https://proto-plus-python.readthedocs.io/en/stable/fields.html#oneofs-mutually-exclusive-fields
Attributes | |
|---|---|
| Name | Description |
image_classification_config |
google.cloud.datalabeling_v1beta1.types.ImageClassificationConfig
Specify this field if your model version performs image classification or general classification. annotationSpecSet in this configuration must match
EvaluationJob.annotationSpecSet.
allowMultiLabel in this configuration must match
classificationMetadata.isMultiLabel in
input_config.
This field is a member of oneof_ human_annotation_request_config.
|
bounding_poly_config |
google.cloud.datalabeling_v1beta1.types.BoundingPolyConfig
Specify this field if your model version performs image object detection (bounding box detection). annotationSpecSet in this configuration must match
EvaluationJob.annotationSpecSet.
This field is a member of oneof_ human_annotation_request_config.
|
text_classification_config |
google.cloud.datalabeling_v1beta1.types.TextClassificationConfig
Specify this field if your model version performs text classification. annotationSpecSet in this configuration must match
EvaluationJob.annotationSpecSet.
allowMultiLabel in this configuration must match
classificationMetadata.isMultiLabel in
input_config.
This field is a member of oneof_ human_annotation_request_config.
|
input_config |
google.cloud.datalabeling_v1beta1.types.InputConfig
Rquired. Details for the sampled prediction input. Within this configuration, there are requirements for several fields: - dataType must be one of IMAGE, TEXT, or
GENERAL_DATA.
- annotationType must be one of
IMAGE_CLASSIFICATION_ANNOTATION,
TEXT_CLASSIFICATION_ANNOTATION,
GENERAL_CLASSIFICATION_ANNOTATION, or
IMAGE_BOUNDING_BOX_ANNOTATION (image object
detection).
- If your machine learning model performs classification,
you must specify classificationMetadata.isMultiLabel.
- You must specify bigquerySource (not gcsSource).
|
evaluation_config |
google.cloud.datalabeling_v1beta1.types.EvaluationConfig
Required. Details for calculating evaluation metrics and creating Evaulations. If your model version performs image object detection, you must specify the boundingBoxEvaluationOptions field
within this configuration. Otherwise, provide an empty
object for this configuration.
|
human_annotation_config |
google.cloud.datalabeling_v1beta1.types.HumanAnnotationConfig
Optional. Details for human annotation of your data. If you set labelMissingGroundTruth to true for this evaluation job, then you must specify
this field. If you plan to provide your own ground truth
labels, then omit this field.
Note that you must create an
Instruction
resource before you can specify this field. Provide the name
of the instruction resource in the instruction field
within this configuration.
|
bigquery_import_keys |
Sequence[google.cloud.datalabeling_v1beta1.types.EvaluationJobConfig.BigqueryImportKeysEntry]
Required. Prediction keys that tell Data Labeling Service where to find the data for evaluation in your BigQuery table. When the service samples prediction input and output from your model version and saves it to BigQuery, the data gets stored as JSON strings in the BigQuery table. These keys tell Data Labeling Service how to parse the JSON. You can provide the following entries in this field: - data_json_key: the data key for prediction input. You
must provide either this key or reference_json_key.
- reference_json_key: the data reference key for
prediction input. You must provide either this key or
data_json_key.
- label_json_key: the label key for prediction output.
Required.
- label_score_json_key: the score key for prediction
output. Required.
- bounding_box_json_key: the bounding box key for
prediction output. Required if your model version perform
image object detection.
Learn `how to configure prediction
keys `__.
|
example_count |
int
Required. The maximum number of predictions to sample and save to BigQuery during each [evaluation interval][google.cloud.datalabeling.v1beta1.EvaluationJob.schedule]. This limit overrides example_sample_percentage: even if
the service has not sampled enough predictions to fulfill
example_sample_perecentage during an interval, it stops
sampling predictions when it meets this limit.
|
example_sample_percentage |
float
Required. Fraction of predictions to sample and save to BigQuery during each [evaluation interval][google.cloud.datalabeling.v1beta1.EvaluationJob.schedule]. For example, 0.1 means 10% of predictions served by your model version get saved to BigQuery. |
evaluation_job_alert_config |
google.cloud.datalabeling_v1beta1.types.EvaluationJobAlertConfig
Optional. Configuration details for evaluation job alerts. Specify this field if you want to receive email alerts if the evaluation job finds that your predictions have low mean average precision during a run. |
Classes
BigqueryImportKeysEntry
BigqueryImportKeysEntry(mapping=None, *, ignore_unknown_fields=False, **kwargs)The abstract base class for a message.
| Parameters | |
|---|---|
| Name | Description |
kwargs |
dict
Keys and values corresponding to the fields of the message. |
mapping |
Union[dict,
A dictionary or message to be used to determine the values for this message. |
ignore_unknown_fields |
Optional(bool)
If True, do not raise errors for unknown fields. Only applied if |