Quickstart: Complete a Review Task

Quickstart: Complete a Review Task

Caution: Document AI Human-in-the-Loop is deprecated and will no longer be available on Google Cloud after January 16, 2025. New customers are not allowlisted. If you want to use (HITL) but don't see the option available, contact your Google Account team.

To implement a human review and correction solution that meets your requirements, we recommend working with a Google Cloud certified partner like Devoteam, Searce, or Quantiphi. See Deprecations for details.

Each processor creates a "HITL task" that is assigned to a pool of human labelers (called "Labeler Pool") that review the documents processed by the processor. Once processed, these documents are queued up in the task for HITL review by the assigned Labeler Pool.

Before you begin

Complete the previous Quickstart: Configure Human Review before proceeding.

Add Labelers

1. After configuring Human Review, you should receive an email similar to the following:

From: AI Platform <[e-mail]>\
Subject: Added to AI Platform SpecialistPool Test Labeler Group

Hello AI Platform Customer,

You are added as a manager to specialist pool
cloudml_data_specialists_us_central1_785484070008756xxxx.
To view details, visit Manager console
https://datacompute.google.com/cm/

Sincerely,
The Google Cloud AI Team

1. Click on the link in the email (or cut and paste into your browser) to navigate to the Manager page:

2. The task created by the newly created Processor has the same name as the Processor (for example, "Department A Invoices") and is listed in the Tasks tab.

3. Click on Labelers tab

4. Click on Add Labeler, and add at least one labeler (Email). A manager can be a labeler, so you can add yourself.

5. For a newly created Labeler Pool, the task must be assigned to the individual labeler explicitly in the Assignments tab. That is, adding the Labeler Pool doesn't automatically assign the task to these new labelers. If the Labeler Pool has already provisioned labelers, the pool is automatically assigned to the task.

Post an Initial Document to Review

1. Send a document for extraction thru the created Processor. Refer to How-To Guides as needed. Both online (sync) and batch (async) calls currently support Human Review routing for supported processors.

   For test purposes, you can either use known poor extraction so as to trigger the Human Review by the confidence score being below the threshold, or you can set the threshold to 100%. Document limits are generally 5 pages, 20MB max, but check the specific Processor limits.

   Note: Uploading a document through the Google Cloud console to test the processor won't trigger a Review process.

   There is an API to track each document routed to Human Review. As part of the response from Processors, there is a humanReviewOperationstring that is the Operation ID (job name) of the document within Human Review. This Long Running Operation (also referred to as a LRO) can be queried for status.

   You can query the status of a document routed to Human Review with the projects.locations.operations.get method

   Note: The batchProcess method includes a skipHumanReview boolean: Whether the Human Review feature should be skipped for this request. Default to false.

2. You can force a document to be Human Reviewed with the reviewDocument method (note that it is Processor-specific):

   Using the reviewDocument method requires that the Human Review ENABLED checkbox be selected. Using this API will not override the config settings.

   You can follow the Request Human Review guide to use this method.

Review the document

1. If the document triggers the human review process, but there are no Labelers in the pool, the manager will get an email.

   > Hello AI Platform Customer,
   >
   > There is a running task for Project  with Job ID 404298342016955xxxx,\
   > Labeling Task ID cloudml_data_us_central1_prod_11b273b4dabdxxxx,\
   > Display name \
   > datalabeling-job-8005676103232389120-for-hitl-340697311810578xxxx,\
   > replication count 1\
   > Please see attachment for instruction.\
   > Please add labelers and have them finish the task at Labeler console \
   > [https://datacompute.google.com/w/cloudml_data_specialists_us_central1_7854840700087566336](https://datacompute.google.com/w/cloudml_data_specialists_us_central1_7854840700087566336){: target="external" class="external" track-type="quickstart" track-name="externalLink" track-metadata-position="body" }.\
   > To view details, visit Manager console \
   > [https://datacompute.google.com/cm/cloudml_data_specialists_us_central1_7854840700087566336/tasks](https://datacompute.google.com/cm/cloudml_data_specialists_us_central1_7854840700087566336/tasks){: target="external" class="external" track-type="quickstart" track-name="externalLink" track-metadata-position="body" }.
   >
   > Sincerely,\
   The Google Cloud AI Team
   

These links are for the Labeling Manager to allocate labeling work.

1. If the labelers aren't enrolled in the Labeling Group and are newly added, the Labeling Manager must send a similar email with instructions to the Labelers added.

   Note: The Manager may manually re-allocate routing of future new documents to labelling pools (tasks). See Manage Labeling Tasks.

Labeler Steps

Labeler Procedure

1. Labeler Email

   If a labeler is already enrolled to a Labeling Group in Step 2e, the system will auto-allocate, and they will get an email like this:

   

   The link in this email is the mechanism for the selected Labeler to do labeling work on the document.

2. Labeling Actions:

   When a Labeler clicks the link to do the labeling, they will see the Interactive AI Human Review User Interface shown below. The highlighted (in yellow) rows indicate the rows that are failing the validation (that is, they are below the configured confidence score threshold) and need review.

   

3. Review By hovering the cursor over a recognized text field, the recognized text and (In smaller font below it) the name of the field (schema) is shown. The row is also highlighted in the left panel.

   The user may zoom in/out with the magnifier icons at the top left of the toolbar.

   Note: **To scroll horizontally, hold down the shift key and use the scroll wheel on the mouse. **This is not documented elsewhere.\

   

4. Change Field

   By (left) clicking on the field, you can edit it:

   You may change the value or label in the left panel. Then click Apply.

   OR, by clicking on one of the resize corner icons on the document image, you may resize the bounding box around the document text:

   As shown, if different text is selected, it will change the text in the field's Value in the left panel. You can then further edit this text if needed. Then click Apply.

   Be sure to review all pages, using the page control in the upper right corner.

After making all needed label changes, click SUBMIT (at bottom left). The result json will be saved to the "Results location" folder.

If the document cannot be satisfactorily labeled, you may click REJECT (at bottom left), and then select a reason for rejecting the document: - Select a reason, then click Reject document.

Labelers may also click on the following icons in the upper right corner:

• Analytics - to show how many documents they have reviewed (Answers) and their total time.
• Notifications - To show any Notifications they have gotten.
• More (3 Dots) - To Skip the document, Find Answers, or provide Feedback (if enabled).
• Toggle Title bar (Diagonal Arrows) - This will hide (or show) the DataCompute title bar for more room.

Labeling Manager

Review Document from Cloud Storage

If you want to check the document, there are two options:

• Option 1:

  1. Locate the Cloud Storage from previous configured location
  2. Retrieve and download the documents

• Option 2:

  1. Remember the URI for retrieve the document from Prediction response
  2. Call URI to retrieve the reviewed document

Note: Human Corrections will be recorded in the TextChanges construct of the Document JSON.

What's next

• Manage labeling tasks using the labeling manager console.
• See a list and descriptions of currently available processors.