Quality Control (QC) round

A quality control round is intended to provide reviewers with documents that Sample-Based Learning categorized so that it can calculate an overturn rate by comparing the categorized value with the value manually coded during this round.

This page contains the following sections:

Executing a QC round

To execute a quality control round:

  1. Click Start Round on the console.
  2. Select Quality control as the Round Type.
    Quality control round
  3. For the Saved search for sampling, select a saved search containing categorized documents (e.g., <Project Saved Search> - Categorized). See Viewing categorized and uncategorized documents for your RAR project for more information.
  1. Specify your desired Sampling Methodology settings. The sample set is the randomly-selected group of documents produced by to be used for manual review as a means of training the system.
  2. Note: The fields in the Sampling Methodology section are defaulted to the values on the project settings; however, if you select Training as the round type, you override those default values.

    • Statistical sampling- creates a sample set based on statistical sample calculations, which determines how many documents your reviewers need to code in order to get results that reflect the project universe as precisely as needed. Selecting this option makes the Margin of error field required.
      • Confidence level - the probability that the rate in the sample is a good measure of the rate in the project universe. This is used in the round to calculate the overturn range as well as the sample size, if you use statistical sampling.
      • Margin of error - the predicted difference between the observed rate in the sample and the true rate in the project universe. This is used in the round to calculate the overturn range as well as the sample size, if you use statistical sampling.
    • Percentage - creates a sample set based on a specific percentage of documents from the project universe. Selecting this option makes the Sampling percentage field required.
      • Sampling percentage - the percentage of the eligible sample population used to create the sample size.
    • Fixed sample size - creates a sample set based on a specific number of documents from the project universe. Selecting this option makes the second Fixed sample size field required.
      • Fixed sample size - the number of documents you want to include in your sample size.
  3. Click Calculate sample to display a calculation of the sample based on the documents eligible for sampling and the values specified in the Sampling Methodology section.
    Clicking Calculate sample displays the number of documents in the saved search selected for the round and the number of documents in the sample. If the values for the sample and/or saved search are unexpected, you can change any setting in the Start Round layout and re-calculate before clicking Go. You can't calculate the sample if you don't have access to the saved search selected for the round. This button is disabled if you've selected Stratified sampling as the sampling type.
  4. Sample calculator display

  5. Specify how to batch documents out for review.
    • Automatically create batches - determines whether or not a batch set and batches are automatically created for this round's sample set. By default, this field is set to whatever value was specified in the project settings. Once the sample set has been created, you can view and edit the corresponding batch set in the Batch Sets tab.
    • Maximum batch size - the maximum number of documents that the automatically-created batches will contain. This is required if the Automatically create batches field above is set to Yes. This value must be greater than zero or an error appears when you attempt to save the project. The batch set and batches created from this project are editable after you create the project. By default, this field is set to whatever value was specified in the project settings.
  6. Click Go.
  7. Proceed to Reviewing documents for a QC round.

Note: When the round is created, the field specified as the Use as an example field is set to Yes by default for documents included in the round. If you delete a round, RAR reverts the Use as an example field value to Not Set (null).

Reviewing documents for a QC round

When reviewing documents for a QC round, you only review categorized documents. You are testing the accuracy of the categorized results of your Sample-Based Learning project. Reviewers are coding documents that have already been categorized by the system. They will not be aware of this fact. They will code as normal, but the system will evaluate their coding values against documents that the system categorized. See Sample-Based Learning document review for more information on protocol for assigning documents out and reviewing documents during a round.

Note: If you're done using a project, it's better for workspace performance if you finish the round rather than leaving it in a status of either Review in Progress or Review complete.

Reviewing Sample-Based Learning reports during a QC round

After the QC round is started and all the documents in the sample are coded, admins assign the seed documents out after review after reading the reports to be corrected. Reviewers make the corrections to any seed documents that are causing issues (see Evaluating overturns and making corrections).

The following reports should be reviewed after QC round sample documents have been coded but before finishing the round:

  • Round Summary report – useful after categorization because it shows the changes in categorization percentage from round to round. Also provides categorization volatility. See Round Summary report.
  • Control Set Statistics report – tracks progress of precision and recall and F1. Also gives the Summary of Rounds. See Control Set Statistics report.
  • Overturn summary report – tracks overturn percentages round to round. There are no overturns prior to a QC round. See Overturn Summary report.
  • Viewing overturned documents - The Overturned Documents view permits an Assisted Review admin to view documents that require re-evaluation quickly and efficiently. You can focus on a single round and filter by the highest ranking overturns. You may also use the pivot feature to see the most influential seed documents (documents that are responsible for a large number of overturns). Once you identify documents as needing further analysis, you can click on a link in order to review the document immediately. See Viewing overturned documents.
  • Rank Distribution report – shows level of conceptual similarity between human coded documents and the overall categorized documents. See Rank Distribution report.
  • Project Summary report – tracks overall project health. You can see a snapshot of overturn and categorization results as well as control set statistics in one place. See Project Summary report.

Note: If issues are also being categorized by Assisted Review, you can also review the Issue Reports.

Evaluating overturns and making corrections

During a computer-assisted review, a case team moves through several rounds of coding to train the system on the document collection and validate the computer’ results. This isn’t a formal round, but a between-rounds workflow used to make any necessary adjustments to the project to prepare for the next round. It consists of identifying, analyzing, and correcting (re-coding) documents which have a significant and adverse effect on project results. You are finding the seed documents that caused the overturns and then making any coding corrections to those seed documents that need to be made.

Potential coding errors are reported in the Overturned Documents link in a Relativity project’s console. The Overturned Documents view permits an Assisted Review admin to view documents that require re-evaluation quickly and efficiently. You can focus on a single round and filter by the highest ranking overturns. You may also use the pivot feature to see the most influential seed documents (documents that are responsible for a large number of overturns). Once you identify documents as needing further analysis, you can click on a link in order to review the document immediately. See Viewing overturned documents.

Note: We recommend that you make these adjustments prior to finishing a round and categorizing documents. The system can then make use of the corrections performed, and then apply them to the next true round.

Consider the following when your reviewers are evaluating overturns and making corrections:

Correcting coding inconsistencies between true or conceptual duplicates:

  • Each seed-overturn pair has a rank (or score) which indicates the degree of conceptual similarity they share. The maximum possible score is 100, which means the two documents are conceptual duplicates. Conceptual duplicates are documents which may or may not have identical text, but do contain the same conceptual content according to the analytics index. While it is possible that conceptual duplicates may also be exact textual duplicates (i.e., documents with the same MD5 hash value), this should not be assumed from a score of 100.
  • We recommend that you use the Overturn Documents report to locate these documents by filtering on the round and sorting by descending rank. A good best practice is to re-evaluate each seed-overturn pair having a rank of 95 and higher to see which document was coded correctly, as well as whether each is a suitable example.

Identifying and correcting the most influential seed documents:

  • When viewing overturn reports at the end of a round, the same few documents can be responsible for many overturns. If those seed documents were incorrectly coded, they can greatly inflate the overturn rate for the entire project. Finding and correcting these situations is an essential component of QC round protocol.
  • The quickest way to find the most influential documents is by using Pivot on the Overturned Documents report. Simply choose Seed document in the Group by drop-down list and leave the <Total Only> drop-down list as is.

Using the Overturn Analysis related items pane:

  • Once a document has been targeted for re-evaluation during a QC round, you can navigate directly to it using the hyperlinks in the Overturned Documents report. Once you reach the core reviewer interface, open the Overturn Analysis related items pane by clicking the RAR icon in the bottom right corner.
  • Clicking the file icon next to the document's control number opens the target document in a separate viewer window. A reviewer can compare the two documents side by side to assist in the decision-making process.

Note: The Overturned Documents view is helpful for review management, but you may want to prevent users from seeing the rest of the project when they only need access to overturns.

You can also provide reviewers access to overturns via the field tree, which includes an overturn status field for your project. To pursue this option, create a view that can be used in conjunction with the field tree.

Finishing a QC round

Once all of the documents in the sample set have been coded, you should finish the round. You also have the option of finishing a round before all of the sample set documents have been coded.

To finish a QC round:

  1. Click Finish Round on the console.
    Finish round button
  2. Specify whether you want to categorize documents when you finish the round. You have two options depending on your project:

    • Categorize for designation - categorize all documents in the project based on their designation coding.
    • Categorize for issues - categorize all documents in the project based on their issue coding. This is only available if you have added a key issue field to the project and a reviewer has issue-coded at least one document in the sample set.

  3. Specify whether you want to save categorization results from the previous round when you finish the current round. You may have two options depending on your project:
    • Save designation results - save the results of designation coding from the previous categorization. This is useful because when categorization runs, the previous results are cleared in order to apply the new category values. You can't save designation results if you did not categorize designations in a previous round.
    • Save issue results - save the results of issue coding from the previous categorization. This is only available if you have added a key issue field to the project. You can only save issue results if you categorized issues in a previous round.

    Note: You shouldn't save results at the end of every round. Saving results, especially for larger cases, can add several hours to the time it takes to finish the round.

  4. Enter the naming for your categorization results.
    • Categorization results set name - the name of the categorization results set. By default, this is the name of the previous round. This is only available for editing if you are saving designation and/or issue results.
    • Categorization results set description - a description of the categorization results. This is only available for editing if you are saving designation and/or issues results.
  5. Finish round layout

  1. Click Go. If you choose to both categorize and save results, the saving of results is performed first, then categorization.