Creating Instructions for HITL Review

Caution: Document AI Human-in-the-Loop is deprecated and will no longer be available on Google Cloud after January 16, 2025. New customers are not allowlisted. If you want to use (HITL) but don't see the option available, contact your Google Account team.

To implement a human review and correction solution that meets your requirements, we recommend working with a Google Cloud certified partner like Devoteam, Searce, or Quantiphi. See Deprecations for details.

Note: This product is subject to the Data Processing and Security Terms.

While the HITL Labeler Workbench provides a What You See Is What You Get (WYSIWYG) interface that maps document entities to the extracted labels, which makes it easy for the labeler to compare and correct. An instructions document is needed to instruct the human labelers of which labels to look for and add, and in case it's missed by the Document AI model or validation filters of HITL. This includes:

• Which labels to review.
• Whether any fields are mandatory or optional.
• Any business logic to
  • Correct labels (such as add "USA" for United States addresses that don't specify USA).
  • Reject documents with the correct rejection field - such as reject invoices >$10,000.
• Special label names in the document that map to schema labels, so labeler can add these - such as "Client #" = "Account #".
• These can be set up as Filters in the HITL task configuration.

Design good instruction

Good instructions are the most important factor in getting good human labeling results. Good instructions are those that let human labelers know what you want them to do. Here are some guidelines for creating good instructions:

• The human labelers might not have your domain knowledge. The distinctions you ask labelers to make must be easy to understand for someone unfamiliar with your use case.
• Avoid making the instructions too long. It is best if a labeler can review and understand them within 20 minutes.
• Instructions must describe the concept of the task as well as details about how to label the data.
• If your instructions have a corresponding label set, they must cover all labels in that set. The label name in the instructions must match the name in the label set.
• It often takes several iterations to create good instructions. We recommend having a small dataset labeled first, then adjusting your instructions based on what you see in the results you receive.

A good instructions file must include the following sections:

• Label list and description: list all of the labels that are used and describe the meaning of each label.
• Examples: For each label, give at least three positive examples and one negative example. These examples must cover different cases.
• Cover edge cases. Clarify as many edge cases as you can, This reduces the need for the labeler to interpret the label. For example, if you need to draw a bounding box for a person, it is better to clarify:
  • If there are multiple people, do you need a box for each person?
  • If a person is occluded, do you need a box??
  • Do you need a box for a person who is partially shown in the image?
  • Do you need a box for a person in a picture or painting?
• Describe how to add annotations. For example:
  • For a bounding box, do you need a tight box or a loose box?
  • For text entity extraction, where should the interested entity start and end?
• Clarification on labels. If two labels are similar or easy to confuse, give examples to clarify the differences.

Visual Examples

The visual example provides clarification to the labelers where to expect different entities in the document and how they map to the extracted labels in the schema. Include visual examples in your instructions like the following: