Method: annotationStores.evaluate

Full name: projects.locations.datasets.annotationStores.evaluate

Evaluate an Annotation store against a ground truth Annotation store. When the operation finishes successfully, a detailed response is returned of type EvaluateAnnotationStoreResponse, contained in the [response] [google.longrunning.Operation.result.response]. The metadata field type is OperationMetadata. Errors are logged to Cloud Logging (see Viewing error logs in Cloud Logging).

HTTP request

POST https://healthcare.googleapis.com/v1beta1/{name=projects/*/locations/*/datasets/*/annotationStores/*}:evaluate

The URL uses gRPC Transcoding syntax.

Path parameters

Parameters
name

string

Required. The Annotation store to compare against goldenStore, in the format of projects/{projectId}/locations/{locationId}/datasets/{datasetId}/annotationStores/{annotationStoreId}.

Authorization requires the following IAM permission on the specified resource name:

  • healthcare.annotationStores.evaluate

Request body

The request body contains data with the following structure:

JSON representation 
{
  "evalInfoTypeMapping": {
    string: string,
    ...
  },
  "goldenStore": string,
  "goldenInfoTypeMapping": {
    string: string,
    ...
  },
  "infoTypeConfig": {
    object(InfoTypeConfig)
  },

  // Union field report_destination can be only one of the following:
  "bigqueryDestination": {
    object(BigQueryDestination)
  }
  // End of list of possible types for union field report_destination.
}
Fields
evalInfoTypeMapping

map (key: string, value: string)

Optional. InfoType mapping for eval_store. Different resources can map to the same infoType. For example, PERSON_NAME, PERSON, NAME, and HUMAN are different. To map all of these into a single infoType (such as PERSON_NAME), specify the following mapping:

  info_type_mapping["PERSON"] = "PERSON_NAME"
  info_type_mapping["NAME"] = "PERSON_NAME"
  info_type_mapping["HUMAN"] = "PERSON_NAME"

Unmentioned infoTypes, such as DATE, are treated as identity mapping. For example:

  info_type_mapping["DATE"] = "DATE"

InfoTypes are case-insensitive.

An object containing a list of "key": value pairs. Example: { "name": "wrench", "mass": "1.3kg", "count": "3" }.
goldenStore

string

Required. The Annotation store to use as ground truth, in the format of projects/{projectId}/locations/{locationId}/datasets/{datasetId}/annotationStores/{annotationStoreId}.
goldenInfoTypeMapping

map (key: string, value: string)

Optional. Similar to evalInfoTypeMapping, infoType mapping for goldenStore.

An object containing a list of "key": value pairs. Example: { "name": "wrench", "mass": "1.3kg", "count": "3" }.
infoTypeConfig

object(InfoTypeConfig)
Union field report_destination. Specifies the report location. report_destination can be only one of the following:
bigqueryDestination

object(BigQueryDestination)

The BigQuery table where the server writes the output. BigQueryDestination requires the roles/bigquery.dataEditor and roles/bigquery.jobUser Cloud IAM roles.

Response body

If successful, the response body contains an instance of Operation.

Authorization scopes

Requires one of the following OAuth scopes:

  • https://www.googleapis.com/auth/cloud-healthcare
  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.

InfoTypeConfig

Specifies how to use infoTypes for evaluation. For example, a user might only want to evaluate PERSON, LOCATION, and AGE.

JSON representation 
{
  "strictMatching": boolean,

  // Union field filter can be only one of the following:
  "evaluateList": {
    object(FilterList)
  },
  "ignoreList": {
    object(FilterList)
  }
  // End of list of possible types for union field filter.
}
Fields
strictMatching

boolean

If TRUE, infoTypes described by filter are used for evaluation. Otherwise, infoTypes are not considered for evaluation. For example:

  • Annotated text: "Toronto is a location"
  • Finding 1: {"infoType": "PERSON", "quote": "Toronto", "start": 0, "end": 7}
  • Finding 2: {"infoType": "CITY", "quote": "Toronto", "start": 0, "end": 7}
  • Finding 3: {}
  • Ground truth: {"infoType": "LOCATION", "quote": "Toronto", "start": 0, "end": 7}

When strictMatching is TRUE:

  • Finding 1: 1 false positive
  • Finding 2: 1 false positive
  • Finding 3: 1 false negative

When strictMatching is FALSE:

  • Finding 1: 1 true positive
  • Finding 2: 1 true positive
  • Finding 3: 1 false negative
Union field filter. filter allows users to flexibly choose a subset of infoTypes for evaluation. If not specified, default behavior is to keep all infoTypes. filter can be only one of the following:
evaluateList

object(FilterList)
ignoreList

object(FilterList)

FilterList

List of infoTypes to be filtered.

JSON representation 
{
  "infoTypes": [
    string
  ]
}
Fields
infoTypes[]

string

These infoTypes are based on after the evalInfoTypeMapping and goldenInfoTypeMapping.