Creating Instructions for HITL Review

While the HITL Labeler Workbench provides a WYSIWYG interface that maps document entities to the extracted labels and makes it easy for the labeler to compare and correct, an Instructions document is needed to instruct the human labelers what labels to look for and add, in case it's missed by the Document AI model or Validation filters of HITL. This includes:

  • Which labels to review
  • Whether any fields are mandatory or optional
  • Any business logic to
    • Correct labels (e.g. add "USA" for US addresses that don't specify USA)
    • Reject documents with the correct rejection field - e.g. reject invoices >$10,000
  • Special label names in the document that map to schema labels, so labeler can add these - e.g. "Client #" = "Account #"
*These can be set up as Filters in the HITL task configuration. HITL AI supports PDF instructions, which can provide sophisticated directions and examples for each case. You can create the instructions by creating a Google Slides and then exporting the slides as a PDF file.

Design good instruction

Good instructions are the most important factor in getting good human labeling results. Since you know your use case best, you need to let the human labelers know what you want them to do. Here are some guidelines for creating good instructions:

  • The human labelers may not have your domain knowledge. The distinctions you ask labelers to make should be easy to understand for someone unfamiliar with your use case.
  • Avoid making the instructions too long. It is best if a labeler can review and understand them within 20 minutes.
  • Instructions should describe the concept of the task as well as details about how to label the data.
  • If your instructions have a corresponding label set, they should cover all labels in that set. The label name in the instructions should match the name in the label set.
  • It often takes several iterations to create good instructions. We recommend having a small dataset labeled first, then adjusting your instructions based on what you see in the results you get back.

A good instructions file should include the following sections:

  • Label list and description: list all the labels you would like to use and describe the meaning of each label.
  • Examples: For each label, give at least 3 positive examples and 1 negative example. These examples should cover different cases.
  • Cover edge cases. Clarify as many edge cases as you can, This reduces the need for the labeler to interpret the label. For example, if you need to draw a bounding box for a person, it is better to clarify:
    • Do you need a box for each person if there are multiple people?
    • Do you need box if a person is occluded?
    • Do you need a box for a person who is partially shown in the image?
    • Do you need a box for a person in a picture or painting?
  • Describe how to add annotations. For example:
    • For a bounding box, do you need a tight box or loose box?
    • For text entity extraction, where should the interested entity start and end?
  • Clarification on labels. If two labels are similar or easy to mix up, give examples to clarify the difference.

Visual Examples

Below is a visual example explaining to labelers where to expect different entities in the document and how they map to the extracted labels in the schema. Include such visual examples in your Instructions: example-instruction