Creating a Review Center queue

Review Center queues are flexible, customizable, and can be used for any stage of review. You can also create templates for common workflows, which shortens the setup time for a new queue to only a few clicks. These queue templates can be saved as part of workspace templates, making it easy to re-use them for other cases. You can also use the AI training from previous queues to improve the relevance predictions in new queues.

Even after creating a queue, you can still edit the settings or add new documents without interrupting reviewers.

See these related pages:

• Monitoring a Review Center queue
• Reviewing documents using Review Center
• Review Center security permissions

Installing Review Center

Review Center is available as a secured application from the Application Library.

To install it:

1. Navigate to the Relativity Applications tab in your workspace.
2. Select Install from application library.
3. Select the ReviewCenter application.
4. Click Install.

After installation completes, the following tabs will appear in your workspace:

• Review Library—create and manage queue templates.
• Review Center—create and manage queues and view the dashboard.
• Review Queues—review documents using queues.
• Saved Models—manage saved copies of the relevance training from previous queues.

For more information on installing applications, see Relativity applications.

Choosing a queue type

Review Center offers two types of review queues. Based on the needs of your project, you can set up review queues that either focus on custom-sorted sets of documents, or focus on documents that the AI classifier predicts as relevant.

Saved search queues

Saved search queues tie your queue to a saved search. You can use saved searches to group documents based on nearly any criteria, including documents from any existing Review Center queue. With this queue type, documents are served up to reviewers based on the sort method used for the saved search. If the saved search does not have a sort method selected, documents will be served up based on Artifact ID.

Prioritized review queues

Prioritized review queues are also based on a saved search, but instead of serving up documents based on their sort order, they use the AI classifier to serve up documents that it predicts as relevant. These relevance rankings are stored in the Rank Output field, and the ranks automatically update every time the queue refreshes.

The AI classifier uses either the extracted text of documents, or another text field you select to make its predictions. Even if other fields are returned in the saved search, they will not affect the results.

If you choose a prioritized review queue, we recommend coding at least two non-empty documents in your data source before preparing or starting the queue: one with the positive choice on your review field, and one with the negative choice. This gives the AI classifier the information it needs to start making its predictions. The more documents are coded, the more accurate the classifier’s predictions will be.

If you do not have any coding completed, you can start the prioritized review queue without any coding. The classifier model won't build until at least 50 documents have been coded, with at least one coded positive and one coded negative. After you reach 50 coded documents, your ranks will update upon the next auto-refresh or manual refresh. If you need it to build sooner, you can manually trigger a queue refresh at any point after at least one document has been coded positive and one has been coded negative.

If your prioritized review queue is similar to an older queue, you can copy the relevance training from the older queue and link it to the new queue to jump-start predictions. For more information, see Reusing saved models.

Including random documents in the queue

When you set up a prioritized review queue, you have the option to serve up randomly chosen documents alongside documents that are predicted relevant. This gives the AI classifier a broader variety of coding decisions to learn from, which improves its predictions in the early stages of a review. Having reviewers code a selection of random documents helps the classifier identify a wider range of relevant topics and prevents it from focusing on a limited subject area.

Under the queue setting Include Random Items, you can choose to include random documents as up to 20% of the total documents served to reviewers. You can change this setting at any time. We recommend including a high percent of random items during the early stages of review.

If you have Coverage Mode turned on, this overrides the Include Random Items setting.

Using Coverage Mode

When Coverage Mode is turned on for a prioritized review queue, the queue switches away from serving up the highest-ranking documents. Instead, it serves up documents that are better for training the model. These are documents with scores near 50, which usually have different content and topics from documents that the model has previously seen. Labeling these helps the model learn from a wider variety of documents and become more confident quickly.

When in Coverage Mode, the AI classifier sorts all documents by their scores’ distance from 50, but limits and spreads out the number of exactly 50-ranked documents. This intermixing diversifies the group of documents and lowers the chance of duplicates. The classifier then serves up these sorted documents to reviewers until the next refresh. After each refresh in Coverage Mode, it re-sorts the documents. Coverage Mode also overrides the Include Random Items setting.

You can turn the Coverage Mode setting on or off at any time during a review. For instructions, see Turning Coverage Mode on and off.

Whenever you turn Coverage Mode on or off, manually refresh the queue. This updates the document sorting for reviewers. For more information, see Turning Coverage Mode on and off.

How document assignment works

If a relational field is not set for the queue, five documents are checked out to each active reviewer at a time. As the reviewer saves their progress on those documents, more are checked out as needed.

For example, documents 1 through 5 are assigned to the first reviewer who starts review. If a second reviewer logs in immediately after, documents 6 through 10 are assigned to the second reviewer. As the first reviewer completes their work, documents 11 through 15 are assigned to them, and so on.

If a relational field is set for the queue, then the entire relational group for a document will also be checked out to that document's reviewer. This may mean that a reviewer has more than 5 documents checked out at a time. For more information, see Keeping document families together. Currently, the number of documents checked out is not customizable.

Keeping document families together

All Review Center queues have the option of setting a relational field. If this is set, the whole relational group of documents present in the queue will be checked out to the same reviewer. This keeps families, email threads, or other relational groupings together in one queue.

When a relational field is set, it takes priority over the sort method and document rank. For example, if you sort a saved search queue by size and set the relational field to Family Group, then the entire family of the largest document will be checked out to the first reviewer, even if it contains small documents. Likewise, if you set the relational field to Family Group for a prioritized review queue, the entire family of the highest ranked document will be checked out to the first reviewer, even if it contains low-ranked documents. Within that family, documents will be served up based on the sort specified in the relational view.

If you plan to code families in the related items pane as part of the reviewer workflow, we recommend that you do not include families in your queue. Otherwise, as you code documents in the related items pane, the coded family documents will still be served to reviewers.

Setting up the reviewer group

To give reviewers access to a queue, set up a reviewer group. You can either create a brand new group, or modify the permissions for an existing user group. You can assign multiple user groups to the same queue.

To set up a reviewer group:

1. Decide which user group or groups should contain the reviewers for the queue. For information on creating and editing groups, see Groups.
2. Add each group to the workspace.
3. Assign each reviewer group the following permissions:
   

   Object Security	Tab Visibility
   Document - View, Edit Review Center Queue - View	Review Queues

4. Add the reviewers to each group.

For more information about permissions, see Review Center security permissions.

Checking document permissions

Make sure the reviewer groups have permissions to the documents they need to review. Reviewers will only see documents they have access to. If any documents in the queue are in a secured documents folder, and a reviewer does not have permissions for it, those documents will not be checked out to the reviewer.

If a reviewer sees a message that there are no more documents to review in a queue, but there are uncoded documents left, check the document permissions.

For more information on document security, see Securing a folder for selected groups.

Creating required queue fields

Before creating a prioritized review queue, create the following fields:

• Review field—a single-choice field that serves as the coding field for your queue. This field should have at least one positive choice and one negative choice. Any other choices will be considered neutral.
• Rank Output—a decimal field that will hold the document ranks. Each prioritized review queue needs a separate Rank Output field on the Document object.
  If a field has two colons (::) in the name, this is called a reflected field. Reflected fields typically link two objects, and they cannot be used as the Rank Output field.

If you are creating a saved search queue, you do not need a Rank Output field, and the review field is optional.

For more information about creating new fields, see Fields.

Creating the saved search

No matter which queue type you choose, you will need a saved search to serve as the data source. This saved search contains all documents that will be available to reviewers.

When creating the saved search:

• The saved search must be public.
• If you want to make sure documents are not added or removed from the queue, make the saved search static.
• If you set a relational field on a template or queue, set the same field in the Related Items drop-down menu of the saved search Conditions tab. Only relational group members returned by the saved search will be included in the queue.

For full instructions on how to create a saved search, see Creating or editing a saved search.

Reducing document clutter

To streamline the review process, we recommend taking steps to reduce document clutter in your saved search. This can include any combination of the following:

• Removing out-of-scope documents
• Focusing on inclusive emails
• Running deduplication

Saved search recommendations for prioritized review queues

When setting up a saved search for a prioritized review queue, we recommend excluding the following file types:

• Compressed files, such as ZIP or RAR files
• System files
• Excel files with mostly numbers
• Image files
• CAD drawings
• Maps

We also recommend removing documents with poor text or no text. By default, the AI model uses the extracted text field to make its predictions. If the documents have text in a different field instead, you can select that field during queue setup.

Size limits for prioritized review queues

For prioritized review queues, we recommend a maximum of 10 million documents per queue. This assumes an average extracted text size of 30KB. If your documents are larger than 30KB on average, limit the number of documents so that the combined total is less than 300GB of extracted text.

Creating a queue template

Templates are unassigned queues that can be used as the basis for building other queues quickly. Queue templates can also be saved as part of your workspace template.

Most fields which are required for queues, such as the Review Field, are not required for a template. This enables you to create generalized templates ahead of time and leave those decisions to the queue creator.

To create a queue template:

1. Navigate to the Review Library tab.
2. Click the New Review Center Queue button.
3. Fill out the template settings. For a full list, see Queue and template settings list.
   • Set the Is Template field to On. This adds the new template to your template list and makes many fields optional.
   • If a field value needs to be changed for each queue, such as Reviewer Groups, leave that field blank.
4. Click Save.
   The template now appears in the Review Library list.

Editing premade templates

The Review Center application comes with several premade queue templates to choose from. These are designed for common tasks such as image review, searching for privileged documents, and comparing reviewer coding decisions to the AI model's decisions.

Currently, the premade templates are:

1. New Blank Queue—use this template to start a new queue from scratch. Change any settings, including the queue type, to fit your needs.
2. Prioritized Review—use this template for relevance review projects. Documents will be served in order based on how strongly the AI classifier predicts that each document is relevant.
3. Model/Reviewer QC—use this template to compare reviewer coding decisions against the AI model. Documents will be served in order based on the saved search sort.
4. Images—use this template to review images that do not have good extracted text. Documents will be served in order based on the saved search sort.
5. Spreadsheets—use this template to review spreadsheets such as Excel files. Documents will be served in order based on the saved search sort.
6. Potentially Privileged—use this template to review documents that reviewers marked as being potentially privileged. Documents will be served in order based on the saved search sort.
7. Needs Further Review—use this template to review documents that reviewers marked as needing further review. Documents will be served in order based on the saved search sort.
8. Docs with Ranks Below Zero—use this template to review documents with ranks below zero. These are documents that could not be classified. Documents will be served in order based on the saved search sort.

You can use these templates as-is. However, we recommend reviewing the settings and tailoring them to your needs.

To edit a queue template:

1. Navigate to the Review Library tab.
2. Click the Edit icon () next to the template you want to use.
3. Review and modify the template settings. For a full list, see Queue and template settings list.
4. Click Save.

Queue and template settings list

When you create or edit a Review Center queue or template, the following settings appear:

1. Name—the queue name reviewers will see.
2. Is Template—whether these queue settings will be used as a template for other queues.
   • If you are creating a template, toggle this On.
   • If you are creating a regular queue, toggle this Off.
   This field exists for all queues. If you toggle the Is Template setting to On for a regular queue, it disappears from the dashboard and becomes usable as a template for other queues. Toggling it off again returns the queue to the dashboard. The queue keeps all of its statistics and coding decisions, but the queue state resets to Not Started.
3. Template Description (templates only)—enter notes about the template such as its intended use, comments about field settings, etc.
4. Queue Type—choose either Saved Search or Prioritized Review.
5. Queue Label—create and choose organizational labels that will apply to this queue or to queues created from this template. Some label ideas include First Level Review, Second Level Review, or Quality Control. For more information, see Filtering the queue tab strip.
6. Reviewer Groups—select the permissions groups that include the reviewers for the queue. This field is not recommended for templates.
7. Data Source—select the saved search that contains the documents for your queue. For more information, see Creating the saved search.
8. Saved Model (Prioritized Review only)—select a saved model that contains relevance training from a previous queue. For more information, see Reusing saved models.
9. Rank Output Field (Prioritized Review only)—select the decimal field you created to hold the document rank scores.
10. Review Field—select the single choice field you created for review. For a Prioritized Review queue, this field must have two or more choices. For a Saved Search queue, this field and the choices are optional.
    1. Positive Choice—select the choice that represents the positive or responsive designation.
    2. Negative Choice—Select the choice that represents the negative or non-responsive designation.
       Any remaining choices are considered neutral.
11. Positive Cutoff—on a scale of 0 to 100, enter the document rank that will be the dividing line between positive and negative documents. All documents ranked at or above this number will be predicted positive, and all documents ranked below it will be predicted negative. By default, the cutoff is set at 50.
12. Relational Field—select a relational field for grouping documents in the queue. This makes reviewers receive related documents together, such as members of the same document family.
    If you set a relational field on a template or queue, set the same field in the Related Items drop-down of the saved search Conditions tab. Only relational group members returned by the saved search will be included in the queue. For more information, see Creating or editing a saved search.
13. Allow Coded in Review (Saved Search only)—controls whether documents coded outside of the queue will still be served up in the queue.
    • Toggle this On to allow outside-coded documents to be served up.
    • Toggle this Off to exclude outside-coded documents from being served up. These are found and removed during queue refreshes and every time a reviewer checks out a document.
    Prioritized review queues use outside-coded documents to train their predictions, but they only show them to reviewers if the Relational Field is set. For example, if the relational field is set to Family Group and some members of a document family are already coded, those will be served up to reviewers along with their family.
14. Include Random Items (Prioritized Review only)—select what percentage of random documents to serve to reviewers. For more information, see Including random documents in the queue.
15. Queue Display Options—select which statistics you want reviewers to see on the queue card in the Review Queues tab.
16. Coverage Mode (Prioritized Review only)—controls whether the queue serves up the highest-ranking documents first, or documents with middle ranks. For more information, see Using Coverage Mode.
    • Toggle this On to serve up mid-rank documents first. This is useful for training the model as quickly as possible. You can turn this mode off at any time.
    • Toggle this Off to serve up highest-rank documents first. This is the default setting for finding relevant documents quickly.
17. Text Precedence 1 (Prioritized Review only)—select the primary text field you want Review Center to analyze. By default, this is set to the Extracted Text field.
18. Text Precedence 2 (Prioritized Review only)—select the text field you want Review Center to analyze if the primary text field is empty. This field is optional.
    • If you choose a value for Text Precedence 2, another field called Text Precedence 3 appears, and so on. You can choose up to 5 long text fields to analyze.
    • Review Center analyzes text from the first field in the list that is not empty. It does not combine text or analyze multiple fields.
19. Queue Refresh—controls whether the queue automatically refreshes after coding activity in any queue. This refresh includes re-running the saved search and checking for outside-coded documents. For prioritized review queues, this also re-trains the classifier with the latest coding and re-ranks documents in order of predicted relevance. For more information, see Auto-refreshing the queue.
    • Toggle this On to refresh the queue automatically.
    • Toggle this Off to prevent automatic refreshes. You will still be able to manually trigger refreshes using the dashboard.
20. Reviewer Document View—select a view to control which fields reviewers see in the Documents panel of the Viewer. If you do not choose a view, this defaults to the lowest ordered view the reviewer has permission to access.
    • This panel shows reviewers a list of documents they previously reviewed in their queue. For more information, see Finding previously viewed documents.
    • If there are any conditions applied to the view, those conditions will also limit which documents appear in the panel.
21. Reviewer Layout—select the coding layout that you want reviewers to see by default when they enter the queue.
    • You can choose from the first 100 layouts in the workspace.
    • If you do not choose a layout, this defaults to the lowest ordered layout the reviewer has permission to access.
22. Anonymize Reviewer Names—controls whether reviewer names appear on the dashboard charts and tables. This setting does not anonymize coding fields added to views or saved searches.
    • Toggle this On to hide reviewer names on the dashboard and list them as Unknown 1, Unknown 2, and so on. These labels are assigned randomly every session. For example, a reviewer who is listed as Unknown 1 the first day might be listed as Unknown 4 another day.
    • Toggle this Off to show reviewer names on the dashboard.
23. Email Notification Recipients—enter email addresses to receive notifications about the queue status. These emails tell users when a manually-triggered queue preparation completes, a queue is empty, or a queue encounters an error while populating. To enter multiple email addresses, separate them with a semicolon. Do not include a space.

Creating a new queue from a template

To create a new queue using a queue template, use the Add Queue button on the Review Center dashboard.

To create a new queue from template using the dashboard:

1. Navigate to the Review Center tab.
2. Click the Add Queue button.
3. Select the template you want to use, then click Next.
4. Edit the queue settings. For a full list, see Queue and template settings list.
   1. Under Reviewer Groups, choose one or more reviewer groups.
   2. In the other fields, check the default values filled in by the template. Change any values that should be different for this queue.
5. Click Save.
   The new queue appears as a tab in the banner at the top of the dashboard.

All queue settings can also be edited after creating the queue.

After a queue has been created from a template, the two of them are no longer connected. You can edit the template without affecting the queue.

For information on starting, managing, and deleting queues, see Monitoring a Review Center queue.