Labeling Manager UI and Labeler Workbench

Caution: Document AI Human-in-the-Loop is deprecated and will no longer be available on Google Cloud after January 16, 2025. New customers are not allowlisted. If you want to use (HITL) but don't see the option available, contact your Google Account team.

To implement a human review and correction solution that meets your requirements, we recommend working with a Google Cloud certified partner like Devoteam, Searce, or Quantiphi. See Deprecations for details.

Manager UI

The Manager UI allows Managers¹ to administer the work of Labelers on Tasks². Managers can use this UI to add or remove Labelers, assign or unassign Tasks, and monitor the progress of Tasks and analytics about each Labeler's performance.

On the left is a list of all Google Cloud project enabled Labeler pools³ that the user is a manager of. The number on the right of the labeler pool name is the number of Labelers in that pool. To the right of that is the main content of the UI, which consists of four tabs:

• Tasks
• Labelers
• Assignments

Tasks

The tasks table is a view of all the Tasks in the Labeler pool's Google Cloud project. It contains some basic statistics about the Task, as well as providing the ability to pause the Task, and quick routes to assigning Labelers to the Tasks.

Filters

Above the data grid is a set of filter controls to determine what Tasks are shown in the data grid.

• Search tasks by name: Filters results to those with the entered name. Includes auto-complete functionality.
• Ongoing: If checked, the results will include all Tasks that are not paused, completed, or unassigned. This is enabled by default.
• Paused: If checked, the results will include all paused Tasks.
• Completed: If checked, the results will include all Tasks that are completed--that is, that have no available Questions (whether because no Questions have been enqueued yet or because all enqueued Questions have been answered).
• Unassigned: If checked, the results will include all Tasks that have no Labelers assigned to it.

If no filters are selected, this is an error and the page will tell the Manager to make some filter selections. If any of the checkboxes are unchecked, Tasks that match that checkbox's condition will be excluded unless the Task matches one of the checked box's conditions. For example, if a Task is both paused and unassigned, it will be shown if the Paused checkbox is selected, or the Unassigned checkbox is selected (or both). This result will not show if neither the Paused checkbox nor the Unassigned checkbox is selected.

The Data Grid

• The first column contains a checkbox for each row. If the Manager selects one or more of these checkboxes and then clicks the Manage Assignment button above the table, they will be taken to the Assignments page, with the selected Tasks already populated in the Tasks filter.
• The Task column displays the Task's name. This is set by the person who creates the Task and is a human-readable name (like "Résumé classification") as opposed to an internal identifier (like "cloud_data_67523fe89343").
• The Objective column contains the objective name--this describes the sort of task (image classification, text classification, etc.).
• The "Last enqueued" column contains the date on which the most recent Question was enqueued (empty if no questions have been enqueued to the task yet).
• The Assigned column displays the number of Labelers who are currently assigned to the task. It also provides access to a dialog from which the Manager can unassign Labelers. The Manager cannot assign from this dialog--in order to do that, they need to use the Assignment page.
• The Questions Available/Enqueued column displays the ratio between the number of Questions in the Task that are currently available to be worked on by Labelers and the total number of Questions currently enqueued in the Task.
• The Progress column shows a graphical and percentage-based display of the above ratio. It also shows whether the Task is paused (not serving any questions), as seen in the bottom row of the table in the above screenshot.
• The Actions column contains a menu with some additional actions to take on the Task:
  • Pause: Only available if the Task is not currently paused, this will pause the Task.
  • Resume: Only available if the Task is currently paused, this will resume the Task.
  • Assignment: This provides a shortcut to the Assignment page, with the Task initially selected in the filters.

Above the data table is a pagination control, with which the Manager can decide to show more results per page, and navigate between pages of results.

Labelers

The labelers table is a view of all the Labelers in the Labeler pool. It contains some basic statistics about the Labeler, as well as a way to quickly assign the Labeler to Tasks.

Filters

Above the data grid is a filter box, which filters results to those with the entered email(s). Multiple emails can be separated by commas.

Add Labeler

The Add button will open a dialog from which the Manager may add one or more Labelers to the pool. Multiple email addresses may be entered, separated by commas. Language codes (for example, "en", "de") may also be entered, separated by commas. The same person (email) can appear in multiple labeler pools, and have different assignments in each pool.

The dialog will show a summary of Labelers that were or were not added. A check mark means that the Labeler was successfully added, whereas the red X means that the Labeler could not be added (for example, because they already exist in the pool).

Data Grid

• The first column contains a checkbox for each row. If the Manager selects one or more of these checkboxes and then clicks the Manage Assignment button above the table, they will be taken to the Assignments page, with the selected Labelers already populated in the Labelers filter.
• The Email column displays the Labeler's email address.
• The Tasks assigned column displays the number of Tasks to which the Labeler is assigned. It also provides access to a dialog from which the Manager can unassign the Labeler from one or more Tasks. The Manager cannot assign from this dialog. To assign a Task the Manager needs to use the Assignment page.
• The Languages column displays which languages the Labeler is associated with. These values can affect which Questions a Labeler can be leased, depending on the configuration of the Question when it was enqueued.
• The Actions column contains a menu with some additional actions to take on the Task:
  • Assignment: This provides a shortcut to the Assignment page, with the Labeler initially selected in the filters.
  • Delete: Removes the Labeler from the current Labeler pool.

Above the data table is a pagination control, with which the Manager can decide to show more results per page, and navigate between pages of results.

Assignments

The assignments page allows Managers to manage assignments to Tasks for the Labelers in the Labeler pool.

Filters

The contents of the assignments grid are determined by two sets of filter criteria: the Labeler filters, which determine what shows up on the Y axis, and the Task filters, which determine what shows up on the X axis.

Labeler filters

• Task picker: using this field, the Manager can select one or more Tasks. The set of Labelers displayed in the grid will include all Labelers who are assigned to any of those tasks.
• Emails filter: using this field, the Manager can specify one or more email addresses, separated by commas. The set of Labelers displayed in the grid will include all Labelers with any of those email addresses.

Either or both of these filters may be used, but at least one must be used.

Task filters

• Task picker: using this field, the Manager can select one or more Tasks. The set of Tasks displayed in the grid will include all Tasks selected.
• Additional filters:
  • All Paused Tasks: If this is checked, the displayed Tasks will also include any Task that is paused.
  • All Completed Tasks: If this is checked, the displayed Tasks will also include any Task that is completed (has no questions available to work on).
  • All Unassigned Tasks: If this is checked, the displayed Tasks will also include any Task that is not assigned to any Labelers.
  • All Ongoing Tasks: If this is checked, the displayed Tasks will also include any Task that is "ongoing" (not paused, completed or unassigned).

Any combination of these filters may be used, but at least one must be used. These filters are independent of each other; the options chosen in the Labelers filter do not have any effect on the options chosen in the Tasks filter, and vice versa.

Assignments Grid

The grid shows the results of the selected filter criteria. Labelers constitute the Y axis, and Tasks constitute the X axis. For each cell, the Manager can check the checkbox to assign the corresponding Labeler to the corresponding Task, or uncheck the checkbox to unassign the corresponding Labeler from the corresponding Task.

There are also some shortcuts available in the menus in each row header and column header cell (the menu is opened by clicking the button with the three vertical dots). For instance, if the Manager opens the menu in a Labeler header cell, the Manager can assign or unassign that Labeler from all Tasks at once. The Manager can also hide the row or column with the Remove menu option, or view the Tasks to which a Labeler is currently assigned/the Labelers that are currently assigned to the Task. On populating the rows and columns, the Manager can choose to remove a particular Labeler or Task from the assignment grid using the available shortcut. This will not affect the assignment status of the Task or Labelers removed. The removed Tasks or Labelers can be shown again in the grid using the respective filters at the top. Only the Tasks and Labelers present in the assignment grid will be used to modify the assignment after committing the changes.

After changes to the assignments have been made, the Commit changes button at the top of the page will be enabled. The Discard changes button will also be available; clicking this button will reset any uncommitted assignment changes.

When the Commit changes button is clicked, the Manager will be shown a list of all the changes that will be made to the assignments. After the Manager reviews the changes, the Manager can abort the changes or commit them. If they abort the changes, they will be returned to the Assignments page to continue editing the assignments. If they Commit the changes, the assignments will be committed and they'll be brought back to the Assignments page.

Until the Manager clicks Commit on the change list page, the changes they made in the grid will not be saved.

Labeler workbench

The Labeler workbench is the interface that Labeler uses to label the Questions. The Questions they receive depend on the Worker Pool(s) of which they are a part, which are assigned by the Managers.

Getting Questions

When the workbench is loaded, it will automatically try to get Questions for the Labeler to label. This may fail for a few reasons:

• The Labeler is not assigned to any Tasks in the current Labeler Pool (the workbench uses one Labeler Pool at a time). There may be questions available for the Labeler in another Labeler Pool.
• The Labeler is assigned to one or more Tasks, but none of the Tasks currently has any questions available to that particular Labeler. This may be due to a number of reasons:
  • No Questions are enqueued
  • All Questions have met replication
  • All Questions with remaining replications have been leased to other Labelers who are working on them at the time
  • All available Questions have languages associated with them that the Labeler does not have.
  • The Labeler has already answered all Questions available to them. These Questions may still be available to other Labelers until replication is met.
  • The Labeler is blocked from answering available Questions because they submitted Answers to prior instances of this Questions. These Questions may still be available to other Labeler.

After the workbench is loaded, the Labeler may run out of Questions. When this happens, a button will appear that allows the Labeler to try to get Questions again (subject to the same potential failures as above).

Working on Questions

When the Labeler has a Question leased to them, it will appear in the workbench. The specific appearance depends on the Question enqueued and the UI that the Task is configured to use.

The workbench will start a timer until the question assignment expires. Once the timer has finished, the Question will expire and be removed from the workbench in order to be assigned to other raters and keep the queue going.

When the Labeler has submitted their answer to the Question, the workbench will automatically load the next question available. Otherwise, the Labeler will have to request more questions with the Get More Questions button, shown above.

Submission status

There are a couple of places in the workbench where a Labeler can view the status of their Answer submissions: whether the submission has succeeded, failed, or is still in progress.

Notifications dialog

This dialog shows the status of the Answers that the Labeler has submitted:

• The icon and number on the left (the orange clock) represents the Answers that are still in the process of being submitted.
• The middle icon (the red triangle) represents the Answers that have failed to submit. Submission failure could be due to a server error of some sort, or a problem with the client's internet connection.
• The icon on the right represents the Answers that have successfully been submitted.

Note that the count does not persist in the browser; if you close the workbench and open it again(or refresh the page) it will reset the counters to 0.

More tools

There are some additional tools available within a menu opened from the button at the end of the main toolbar.

The options are:

• Skip: Skips the current question, if the Task has skipping enabled. This is generally used if the Question as displayed in the UI is broken, but individual Managers may give instructions to Labeler on when to skip a question. Note that a skip is counted as an answer, saved with additional metadata indicating that it was skipped and why.
• Find Answers: Goes to the Answers page, which lets the Labeler view their Answers to any Question by searching for the Question ID. The Labeler can only view their own Answers, while a Manager in the Labeler Pool can view all answers.

Question queue

There is also a queue of Questions for which the Labeler has submitted answers:

In this case, the first Answer was submitted successfully, but the next Answer failed to submit. The next Answer submitted successfully, and the fourth bubble represents the currently loaded Question and its remaining time. The final bubble gives a count of how many Questions could be leased to the Labeler. This helps the Labeler to know when they're close to running out of Questions to label, and they can inform their Manager.

A Labeler can go back to a previous Question by clicking its corresponding bubble. If the Question has not expired, then the Labeler may change their Answer and submit it again. Only the latest answer will be stored; it will overwrite any previous answer. Note that some tasks may be instructed to not use this functionality if the answer submission is final.

Additional tools toolbar

There is a toolbar available that contains additional tools for the workbench. It is hidden by default, but can be expanded if needed.

There are three buttons available here:

• Fullscreen: Makes the plugin open in fullscreen mode. This will show only the workbench, and hide the browser UI, as well as any operating system toolbars. The Labeler can exit this mode by pressing Esc or by clicking the same button again.
• Reload: Refreshes the currently visible plugin. This is better than reloading the page from the browser in most cases since reloading the page may clear some state, such as time spent and answers submitted, and the Labeler is not guaranteed to get the same question again if they reload the page.
• Question info: This displays some information about the currently visible Question, such as question IDs and plugin name. Noting this information may be useful information when reporting problems with a specific question, depending on task manager instructions.

Analytics page

The Analytics page gives the Labeler a view of their recent contributions.

At the top is a graph of contribution rates. Below it is the same information displayed in tabular form. The Labeler can select from a few predefined time ranges to view the data. The Analytics page can be accessed by clicking the Analytics button in the toolbar.

Making space for the plugin

There are a couple of controls available to adjust how much space the plugin has available to it.

The top button (the diagonal arrows) toggles the visibility of the One Google Bar--the bar right above it which contains the Google account switcher. The bottom button (the downward arrow) toggles the visibility of the additional tools toolbar. When combined with Fullscreen mode, toggling both toolbars to be hidden will give the plugin almost the entire screen.

---

1. A Manager of a Labeler Pool is a user with the permission to add or remove Labelers and to change which tasks the labelers are assigned to. ↩

2. A Task is a collection of Questions that can be assigned to one or more Labelers from a Labeler Pool.. ↩

3. A Labeler Pool is a collection of labelers managed by one or more Managers. Labelers from a pool can be assigned to work on Tasks enabled for that pool. ↩