Custom extractor with generative AI

Generative AI training and extraction allows you to:

• Use zero-shot and few-shot technology to get a high performing model with little to no training data using the foundation model.
• Use fine-tuning to further boost accuracy as you provide more and more training data.

Generative AI training methods

The training method you choose depends on the amount of documents you have available and the amount of effort available to put into training your model. There are three ways to train a generative AI model:

Training method	Zero-shot	Few-shot	Fine-tuning
Accuracy	Medium	Medium-high	High
Effort	Low	Low	Medium
Recommended number of training documents	0	5 to 10	10 to 50+

Before getting started

If not done so already, enable billing and the Document AI APIs.

Note: Takes some time for each training to complete.

Build and evaluate a generative AI model

1. Create a processor and define fields you want to extract following best practices, which is important because it impacts extraction quality.

   1. Go to Workbench > Custom extractor > Create processor > Assign a name.

   

   1. Go to Get started > Create new field.

   

   Note: The field names with the foundation model can greatly affect model accuracy and performance. A descriptive name is recommended.

2. Import documents

   • Import documents with auto-labeling and assign documents to the training and test set.
   • For zero-shot, only the schema is required. To evaluate the accuracy of the model, only a testing set is needed.
   • For few-shot, we recommend five training documents.
   • The number of needed testing documents depend on the use case. Generally, more testing documents is better.
   • Confirm or edit the labels in the document.

3. Train model:

   • Select the Build tab and Create new version.
   • Enter a name and select Create.

   

4. Evaluation:

   • Go to Evaluate & test, select the version you just trained, then select View full evaluation.

   

   • You now see metrics such as f1, precision, and recall for the entire document and each field.
   • Decide if performance meets your production goals. If it does not then reevaluate training and testing sets.

5. Set a new version as default:

   • Navigate to Manage versions.
   • Select to expand the options, and then select Set as default.
   Note: Ensure the Status of the version is deployed. If it is not, select the deploy version first. You can now set the new version as the default.

   

   Your model is now deployed. Documents sent to this processor use your custom version. You can evaluate the model's performance to check if it requires further training.

Evaluation reference

The evaluation engine can do both exact match or fuzzy matching. For an exact match, the extracted value must exactly match the ground truth or is counted as a miss.

Fuzzy matching extractions that had slight differences such as capitalization differences still count as a match. This can be changed at the Evaluation screen.

Fine-tuning

With fine-tuning, you use hundreds or thousands of documents for your training.

1. Create a processor and define fields you want to extract following best practices, which is important because it impacts extraction quality.

   Note: The field names with the foundation model can greatly affect model accuracy and performance. Be sure to give a descriptive name.

2. Import documents with auto-labeling, and assign documents to the training and test set.

3. Confirm or edit the labels in the document.

4. Train model.

   1. Select the Build tab and select Create New Version.
   2. Try out the default training parameters or values provided. If the results are unsatisfactory, experiment with these advanced options:

   • Training steps (between 100 and 400): Controls how often the weights are optimized on a batch of data during the tuning.

     • Too low indicates a risk that the training ends before convergence (under-fitting).
     • Too high means the model might see the same batch of data multiple times during the training, which can lead to overfitting.
     • Fewer steps leads to faster training time. Higher counts can help for documents with little template variation (and lower ones for those with more variation).

   • Learning rate multiplier (between 0.1 and 10): Controls how quickly the model parameters are optimized on the training data. It roughly corresponds to the size of each training step.

     • Low rates mean small changes in the model weights at each training step. If too low, the model might not converge to a stable solution.
     • High rates indicate large changes, and too high can mean the model steps over the optimal solution and converges instead to a suboptimal solution.
     • Training time is not affected by the choice of learning rate.
   1. Give a name and select Create.

   

5. Evaluation: Go to Evaluate & test, then select the version you just trained and select View full evaluation.

   

   • You now see metrics such as f1, precision, and recall for the entire document and each field.
   • Decide if performance meets your production goals. If not, then further training documents might be required.

6. Set a new version as the default:

   1. Navigate to Manage versions.
   2. Select to expand the options, and select Set as default.
   Note: Ensure the status of the version is deployed. If it's not, then instead select Deploy version. You can now set the new version as the default when its status is changed to deployed.

   

   Your model is now deployed and documents sent to this processor now use your custom version. You want to evaluate the model's performance to check if it requires further training.

Auto-labeling with the foundation model

The foundation model can accurately extract fields for a variety of document types, but you can also provide additional training data to improve the accuracy of the model for specific document structures.

Document AI uses the label names you define and previous annotations to make it quicker and easier to label documents at scale with auto-labeling.

1. When you've created a custom processor, go to the Get Started tab.

2. Select Create New Field.

   

   Note: The label names with the foundation model can greatly affect model accuracy and performance. Be sure to give a descriptive name.

3. Navigate to the Build tab then select Import Documents.

   

4. Select the path of the documents and which set the documents should be imported into. Check the auto-labeling option and select the foundation model.

5. In the Build tab, select Manage Dataset.

6. When you see your imported documents, select one of them.

   

The predictions from the model are now shown highlighted in purple.

1. Review each label predicted by the model, and ensure it's correct.

2. If there are missing fields, add those as well.

   Note: It's important that all fields are as accurate as possible, or model performance will suffer. More details on labeling

   

3. When the document has been reviewed, select Mark as Labeled. The document is now ready to be used by the model.

4. Make sure the document is in either the testing or training set.

Three-level nesting

Custom Extractor now provides three levels of nesting. This feature provides better extraction for complex tables.

Note: Three levels of nesting is only supported if using generative AI ("Foundation") model types within the Custom Extractor. Extract, classify, and splitting models only support two levels of nesting.

You can determine the model type using the following API calls:

• processors.get
• processorVersions.get

The response of these is a ProcessorVersion, which contains the modelType field in v1beta3 preview.

Procedure and example

We are using this sample:

1. Select Get Started, and then create a field:

   • Create the top level.
   • In this sample, the officer_appointments is used.
   • Select This is a parent label.
   • Select Occurrence: Optional multiple.

   

2. Select Add child field. The second level label can now be created:

   • For this level label, create officer.
   • Select This is a parent label.
   • Select Occurrence: Optional multiple.

   

3. Select Add child field from second level officer. Create child labels for the third level of nesting.

4. When your schema is set, you can get predictions from documents with three levels of nesting using auto-labeling.

   

Dataset configuration

A document dataset is required to train, up-train, or evaluate a processor version. Document AI processors learn from examples, just like humans. Dataset fuels processor stability in terms of performance.

Train dataset

To improve the model and its accuracy, train a dataset on your documents. The model is made up of documents with ground-truth.

• For fine-tuning, you need a minimum of 10 documents to train a new model.
• For a few-shot, 5 documents is recommended.
• For zero-shot, only a schema is required.

Ground-truth is the correctly labeled data, as determined by humans.

Test dataset

The test dataset is what the model uses to generate an F1 score (accuracy). It is made up of documents with ground-truth. To see how often the model is right, the ground truth is used to compare the model's predictions (extracted fields from the model) with the correct answers. The test dataset should have at least 10 documents for fine-tuning.

What's next

Uptrain a specialized processor