Reviewer access

This page will cover the reviewer view and workflow in an Active Learning project.

This page contains the following information:

Project document view

Upon saving a new project, Relativity creates a new document list view that's tied to the Active Learning project. This is the only place a logged in user can enter a project queue. This view is automatically secured to the reviewers and returns documents previously reviewed by the currently logged in reviewers. When a reviewer first accesses the view, no documents appear because none have been coded. Documents appear as the reviewer codes documents from the Active Learning queue, the documents appear in the list.

The project document view enables or disables reviewer access based on whether or not the logged in reviewer is added to the queue in the Active Learning project.

Inactive reviewer

If a reviewer hasn't been added to the queue or if the queue is paused, they can't access the Active Learning queue from the project document view. The reviewer can still access the view but only for reviewing previously coded documents. An inactive reviewer can still code documents outside the queue if they have permissions to do so.

(Click to expand)

Project document view for inactive review

Active reviewer

If a reviewer is added to the queue and the queue is enabled, a blue banner appears in the project document view with a Start Review button. Clicking Start Review begins the review process for that logged in user.

If the queue is disabled or a reviewer is removed from the queue while reviewing documents, a warning message will appear.

(Click to expand)

Review workflow

The reviewer clicks Start Review to access the Active Learning queue viewer and start reviewing documents. This viewer looks the same no matter which type of queue is active. The Active Learning viewer contains the same functionality available in the standard viewer for non-Active Learning projects.

Note: Admins can still grant permissions to specific users or groups for certain icons in the Viewer.

Relativity checks the following before assigning a document to a reviewer:

  • The current reviewer has permission to the document
  • The document has not been coded already
  • The document is not assigned to any other active reviewer

Documents can only be assigned to one reviewer except in cases where the reviewer leaves the viewer without making a coding decision or is automatically logged out. When this happens, the queue can reassign the document. You can view the reviewer assignment from the Project Reviewers::User field.

Reviewers code documents on the review field specified for the project and then click Save and Next to get a new document. They can also code other fields not associated with the project. Reviewers can skip documents without making a coding decision unless the review field is a required field. They can also click Active Learning New Document icon to skip the current document and open a new document. In the upper right of the viewer, reviewers can see a running count of all documents they've coded in the project, including skipped documents. Skipped documents are also reflected in the Admin dashboard on Project Home.

Note: Reviewers can change the coding decision on documents they previously reviewed. These documents aren't considered manually-selected documents.

Active Learning reviewer protocol

Reviewers must code documents based on the so-called "four corners rule". This means that a document should be judged responsive or not responsive based solely on the extracted text of that document only. Documents that are relevant based on family members should not be coded relevant on the Active Learning review field. Although individual anomalies will not hurt the algorithm, too many in aggregate could cause a higher error rate. For more information on reviewer protocol and best practices, see Relativity Assisted Review Reviewer Protocol.

Active Learning process

The actual Active Learning process takes place using a combination of time and detection of new coding decisions. By default, the classification index rebuilds every twenty minutes since the last completed build. However, if Active Learning detects reviewer inactivity of at least five minutes, a build will take place before the twenty-minute threshold is reached.