Reviewer access

This page will cover the reviewer view and workflow in an Active Learning project.

This page contains the following information:

Project document view

Upon saving a new project, Relativity creates a new document list view that's tied to the Active Learning project. This is the only place a logged in user can enter a project queue. This view is automatically secured to the reviewers and returns documents previously reviewed by the currently logged in reviewers. When a reviewer first accesses the view, no documents appear because none have been coded. Documents appear in the list as the reviewer codes documents from the Active Learning queue.

The project document view enables or disables reviewer access based on whether or not the logged in reviewer is added to the queue in the Active Learning project.

Inactive reviewer

If a reviewer hasn't been added to the queue or if the queue is paused, they can't access the Active Learning queue from the project document view. The reviewer can still access the view but only for reviewing previously coded documents. An inactive reviewer can still code documents outside the queue if they have permissions to do so.

Project document view for inactive review

Active reviewer

If a reviewer is added to the queue and the queue is enabled, a blue banner appears in the project document view with a Start Review button. Clicking Start Review begins the review process for that logged in user.

If the queue is disabled or a reviewer is removed from the queue while reviewing documents, a warning message will appear.

Start Review banner on project document view

Review workflow

The reviewer clicks Start Review to access the Active Learning queue viewer and start reviewing documents. This viewer looks the same no matter which type of queue is active. The Active Learning viewer contains the same functionality available in the standard viewer for non-Active Learning projects.

Note: Admins can still grant permissions to specific users or groups for certain icons in the Viewer.

Relativity checks the following before assigning a document to a reviewer:

  • The current reviewer has permission to the document
  • The document has not been coded already
  • The document is not assigned to any other active reviewer

Documents can only be assigned to one reviewer except in cases where the reviewer leaves the viewer without making a coding decision or is automatically logged out. When this happens, the queue can reassign the document. You can view the reviewer assignment from the Project Reviewers::User field.

Reviewers code documents on the review field specified for the project and then click Save and Next to get a new document. They can also code other fields not associated with the project. Reviewers can skip documents without making a coding decision unless the review field is a required field. They can also click Active Learning New Document icon to skip the current document and open a new document. In the upper right of the viewer, reviewers can see a running count of all documents they've coded in the project, including skipped documents. Skipped documents are also reflected in the Admin dashboard on Project Home.

Note: Reviewers can change the coding decision on documents they previously reviewed. These documents aren't considered manually-selected documents. The next model build will include the most recent coding update.

Active Learning process

The actual Active Learning process takes place using a combination of time and detection of new coding decisions. By default, the classification index rebuilds every 20 minutes since the last completed build. However, if Active Learning detects reviewer inactivity of at least five minutes, a build will take place before the twenty-minute threshold is reached.

Active Learning viewer protocol

Best results are achieved when the reviewer codes based on the so-called "four corners rule". This means that a document should be judged responsive or not responsive based solely on the extracted text of that document only. Although individual anomalies will not hurt the algorithm, too many in aggregate could cause a higher error rate.

Active Learning queue relational functionality

The Active Learning queue does not serve up family members. However, relational functionality and email thread visualization is available for family-based review. If family members are coded and they are included in the Active Learning project, those documents contribute to the project and are reported as manually-selected documents.