Bulk upload with the Cloud Storage ingest pipeline

Caution: Document AI Warehouse is deprecated and will no longer be available on Google Cloud after January 16, 2025. To safeguard your data, migrate any documents currently saved in Document AI Warehouse to an alternative like Cloud Storage. Verify that your data migration is completed before the discontinuation date to prevent any data loss. See Deprecations for details.

Preview

This feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of the Service Specific Terms. Pre-GA features are available "as is" and might have limited support. For more information, see the launch stage descriptions.

This document describes how to perform bulk upload, which triggers the Cloud Storage ingest pipeline behind the scene.

Preprocessing options

Currently, the bulk upload provides three preprocessing options:

1. Bulk upload without preprocessing: This triggers runPipeline API with GcsIngestPipeline without processing the documents with Document AI processors.

2. Extract entities with Document AI processors: This triggers runPipeline API with GcsIngestWithDocAiProcessorsPipeline. The pipeline will call the given Document AI processor first, and then ingest the documents with the processed results.

3. Classify document types and extract entities for each type: This also triggers runPipeline API with GcsIngestWithDocAiProcessorsPipeline, which first calls a classifier. Then, for each document type, you can specify a corresponding schema and processor to process those particular document types. They're ingested with the results and set to this schema.

Each of the preprocessing types correspond to the following options in the UI:

Example: Trigger bulk upload with an OCR processor

This example illustrates the second use of the pipeline.

Create an OCR processor and get processor ID

If you have created an OCR processor before, just find it in the processor list, and go into the details page of the processor and get the processor ID.

If you have not created one, follow these steps:

1. At the top of the processor list, click the Processor Gallery:

   

2. Find the Document OCR processor in the gallery, and at the bottom of the card, click Create Processor:

   

3. Enter a processor display name:

   

4. Click Create and when you're redirected to the Processor Details page, find the ID:

   

   This is what you need to copy to the input fields in the bulk upload view.

Trigger bulk upload

1. Open the bulk upload view.

   Next to Add New, click Bulk Upload:

   

2. Find the correct processor.

   1. Select the second preprocessing option.

   2. Choose a schema and specify a processor and Cloud Storage bucket path for saving the extraction results in JSON format.

3. Find the processor ID through the link in the description text:

   

   Note: If you haven't used Document AI before, the processors link redirects to an API-enabling page, as shown below:

4. Trigger upload:

   1. With the processor ID copied from the last step, specify the input fields. The source file bucket path can be a bucket or a folder or subfolder in the bucket.

   2. When the input fields are valid, to trigger bulk upload, at the top right, click Upload.

Check progress in the status page

After the bulk upload is triggered, you are redirected to the status tracking page:

The first table shows any pending or processed documents. After they're ingested, the document is not listed in the first table anymore. Documents that failed to upload appear in the second table. On the right, the statistics shows the number of ingested, failed, and pending documents.

After the job is complete, the status page shows 100% complete without any pending documents:

Examine the uploaded documents

1. Find the newly ingested documents by going back to the search view. Click the Document AI Warehouse logo or Search on the top navigation bar:

   

2. Open any of the newly ingested documents by clicking the document name. In the document viewer, you can open the AI View.

   

3. Go to the Text block tab. The OCR results are stored in the document:

   

Next step

Update existing documents with the extract with Document AI pipeline.