Prioritized Review

The Prioritized Review queue serves documents that are most likely to be coded on the positive choice (ex. Relevant) with a small set of documents included for index health. The documents included for index health are system-selected to help the model develop a better range of training. Any coded document contributes to the model’s learning.

This page contains the following information:

Adding Reviewers

From the project homepage, admins can add reviewers to each queue on a user-by-user basis or in bulk.

To add reviewers:

  1. Click Add Reviewers in the queue modal.
  2. Find and select an individual reviewer, or type to enter a reviewer name. You can also click All Users to select every user in the group.
  3. Click the green check mark to save your changes.
  4. Click the red X to cancel reviewer changes.

Note: We recommend no more than 20 concurrent reviewers per project. Concurrent reviewers are defined as reviewers making coding decisions in an Active Learning queue. There is no limit to how many reviewers you can add to a queue as long as the number of concurrent reviewers remains at 20 or fewer.

Starting the Prioritized Review queue

To start the Prioritized Review queue:

  1. Successfully add reviewers to the queue.
  2. Click Start Review.

The next time an active reviewer accesses the view for the Active Learning project, they will see a blue banner which indicates they have access to the queue. The view also displays any documents coded by the logged in user for that particular Active Learning project. The Prioritized Review queue serves documents that are most likely to be relevant with a small set of documents included for index health. However, if you start the Prioritized Review queue before a model build occurs, the queue initially serves up random documents.

After the project begins, you can see the following items in the Prioritized Review queue modal:

  • Coded - the number of documents coded and skipped in the Prioritized Review queue.
  • Active Reviewers - the number of reviewers added to the project.

Admins can pause the review at any point in the project by clicking the Pause Review button on the project modal. Once the review is paused, the access point on the document view is disabled. You can restart the review at any time. Documents coded on the review field while the review is paused are included the next time the model rebuilds. For more information, see Project Setup.