Creating a Review Center queue

Review Center is a review management tool that helps you build custom queues, use AI to prioritize relevant documents, and leverage a rich reporting dashboard to understand the state of your data and track productivity. With streamlined administrative features and flexible AI algorithms, you can tailor the review process to your needs.

Some of Review Center's key features include:

  • Templatization—set up best-practice structures ahead of time for easy re-use.

  • Customizable queues—replace batch administration with queues based on saved searches.

  • Powerful AI classifier—Review Center uses an all-new integrative learning classifier that provides even greater efficiency than previous AI classifiers.

  • Clear progress reporting—a rich dashboard features timeline-based visualizations that show relevance rates and progress in the context of other project milestones.

Integrative learning classifier

The integrative learning classifier used by Review Center is a scalable, secure, and efficient classification service that can support a variety of use cases and documents. It makes connections among concepts and decisions to serve up relevant documents to reviewers as early as possible.

You do not need to create an Analytics index for Review Center queues. Instead, when you prepare or start an AI-driven queue, the classifier automatically runs in the background to manage documents.

Because the classifier is language-agnostic, you can use Review Center for documents written in any language.

This page contains the following information:

See these related pages:

Installing Review Center

Review Center is available as a secured application from the Application Library.

To install it:

  1. Navigate to the Relativity Applications tab in your workspace.

  2. Select Install from application library.

  3. Select the ReviewCenter application.

  4. Click Install.

After installation completes, the following tabs will appear in your workspace:

  • Review Library—create and manage queue templates.

  • Review Center—create and manage queues and view the dashboard.

  • Review Queues—review documents using queues.

For more information on installing applications, see Relativity applications.

Choosing a queue type

Review Center offers two types of review queues. Based on the needs of your project, you can set up review queues that either focus on custom-sorted sets of documents, or focus on documents that the AI classifier predicts as relevant.

Saved search queues

Saved search queues tie your queue to a saved search. You can use saved searches to group documents based on nearly any criteria, including documents from any existing Active Learning project or other Review Center queue. With this queue type, documents are served up to reviewers based on the sort method used for the saved search. If the saved search does not have a sort method selected, documents will be served up based on Artifact ID.

These queues also have the option of setting a relational field. If this is set, the whole relational group of documents present in the queue will be checked out to the same reviewer. This keeps families, email threads, or other relational groupings together in one queue.

When a relational field is set, it takes priority over the sort method. For example, if you sort by size and set the relational field to Family, then the family of the largest document will be checked out to the first reviewer, even if it contains small documents.

Prioritized review queues

Prioritized review queues are also based on a saved search, but instead of serving up documents based on their sort order, they use the AI classifier to serve up documents that it predicts as relevant. These relevance rankings are stored in the Rank Output field, and the ranks automatically update every time the queue refreshes.

The AI classifier uses the extracted text of documents to make its predictions. Even if other fields are returned in the saved search, it will not affect the results.

If you choose a prioritized review queue, you must code at least two non-empty documents in your data source before preparing or starting the queue: one with the positive choice on your review field, and one with the negative choice. This gives the AI classifier the information it needs to start making its predictions.

Note: The more documents are coded, the more accurate the classifier’s predictions will be. We recommend starting with at least 10 positive-coded and 10 negative-coded documents when possible.

Setting up the reviewer group

To give reviewers access to a queue, set up a reviewer group. You can either create a brand new group, or modify the permissions for an existing user group.

To set up a reviewer group:

  1. Decide which user group should contain the reviewers for the queue. For information on creating and editing groups, see Groups.

    Note: You cannot set the Workspace Admin Group as the reviewer group.

  2. Add the group to the workspace.

  3. Assign the reviewer group the following permissions:

    1. Object Security: Document - View and Document - Edit

    2. Object Security: Review Center Queue - View

    3. Tab Visibility: Review Queues tab

  4. Add the reviewers to the group.

For more information about permissions, see Review Center security permissions.

Creating required queue fields

Before creating a prioritized review queue, create the following fields:

  • Review field—a single-choice field that serves as the coding field for your queue. This field should have at least one positive choice and one negative choice. Any other choices will be considered neutral.

  • Rank Output—a decimal field that will hold the document ranks. Each prioritized review queue needs a separate Rank Output field on the Document object.

    Note: You cannot use a reflected field as the Rank Output field.

If you are creating a saved search queue, you do not need a Rank Output field, and the review field is optional.

For more information about creating new fields, see Fields.

Creating a queue template

Templates are unassigned queues that can be used as the basis for building other queues quickly. The Is Template field should always be toggled to On for templates.

The Review Center application comes with several premade queue templates to choose from. However, we recommend tailoring them or creating your own to best suit your needs. These can also be saved as part of your workspace template.

To create a queue template:

  1. Navigate to the Review Library tab.

  2. Click the New Review Center Queue button.

  3. Enter the following information:

    1. Name—the queue name reviewers will see.

    2. Is Template—toggle this to On.

      Note: This field exists for all queues. If you toggle the Is Template setting to On for a regular queue, it disappears from the dashboard and becomes usable as a template for other queues. Toggling it off again returns the queue to the dashboard. The queue keeps all of its statistics and coding decisions, but the queue state resets to Not Started.
    3. Template Description—enter notes about the template such as its intended use, comments about field settings, etc.

    4. Queue Label—create and choose organizational labels that will apply to queues created from this template. Some label ideas include First Level Review, Second Level Review, or Quality Control.

    5. Reviewer Group—this is not recommended for templates.

    6. Queue Type—choose either Saved Search or Prioritized Review.

    7. Data Source—select the saved search that contains the documents for your queue. For file type considerations, see Data source considerations for Analytics.

    8. Rank Output (Prioritized Review only)—select the decimal field you created to hold the document rank scores.

    9. Review Field—select the single choice field you created for review. This field must have two or more choices.

      1. Positive Choice—select the choice that represents the positive or responsive designation.

      2. Negative Choice—Select the choice that represents the negative or non-responsive designation.

        Note: Any remaining choices are considered neutral.

    10. Relational Field (Saved Search only)—select a relational field for grouping documents in the queue. This makes reviewers receive related documents together, such as members of the same document family.

      Note: If you set a relational field on a template or queue, set the same field in the Related Items drop-down of the saved search Conditions tab. Only relational group members returned by the saved search will be included in the queue. For more information, see Creating or editing a saved search.
    11. Allow Coded in Review (Saved Search only)—controls whether documents coded outside of the queue will still be served up in the queue.

      1. Toggle this On to allow outside-coded documents to be served up.

      2. Toggle this Off to exclude outside-coded documents. These are found and removed every time the queue refreshes.

      Note: Prioritized review queues always exclude outside-coded documents from being served to reviewers, but the AI classifier still uses those documents to train its predictions.

    12. Queue Display Options—select which statistics you want reviewers to see on the queue card in the Review Queues tab.

    13. Reviewer Document View—select a view to control which fields reviewers see in the Documents panel of the Viewer. If you do not choose a view, this defaults to the lowest ordered view the reviewer has permission to access.

      • This panel shows reviewers a list of documents they previously reviewed in their queue. For more information, see Finding previously viewed documents.

      • If there are any conditions applied to the view, those conditions will also limit which documents appear in the panel.

    14. Queue Refresh—controls whether the queue automatically refreshes after coding activity. This refresh includes re-running the saved search and checking for outside-coded documents. For prioritized review queues, this also re-trains the classifier with the latest coding and re-ranks documents in order of predicted relevance.

      1. Toggle this On to refresh the queue automatically.

      2. Toggle this Off to prevent automatic refreshes. You will still be able to manually trigger refreshes using the Prepare Queue button on the dashboard.

    15. Email Notification Recipients—enter email addresses to receive notifications about the queue status. These emails tell users when a manually-triggered queue preparation completes, a queue is empty, or a queue encounters an error while populating. To enter multiple email addresses, separate them with a semicolon. Do not include a space.

  4. Click Save.

    The template now appears in the Review Library list.

Creating a new queue from a template

To create a new queue using a queue template, use the Add Queue button on the Review Center dashboard.

Add Queue button

To create a new queue from template using the dashboard:

  1. Navigate to the Review Center tab.

  2. Click the Add Queue button.

  3. Select the template you want to use, then click Next.

  4. Choose or modify the Reviewer Group.

  5. In the other fields, check the default values filled in by the template. Change any values that should be different for this queue.

  6. Click Save.

    The new queue appears as a tab in the banner at the top of the dashboard.

All queue settings can also be edited after creating the queue.

Note: After a queue has been created from a template, the two of them are no longer connected. You can edit the template without affecting the queue.

For information on starting, managing, and deleting queues, see Monitoring a Review Center queue.