aiR for Review Prompt Criteria validation

Prompt Criteria validation gathers metrics to check whether the Prompt Criteria are effective and defensible before using them on a larger data set. Using aiR for Review and Review Center in tandem, you can set up a smaller document sample, oversee reviewers, and compare aiR's relevance predictions to actual coding results.

This functionality is currently only enabled for the Relevance and Relevance & Key analysis types.

See related topics:

Prerequisite

The Review Center application must be installed to run the validation workflow with aiR for Review.

How validation fits into aiR for Review

aiR for Review leverages the prompt criteria across a three-phased workflow:

Diagram representing workflow from Develop to Validate to Apply

  1. Develop—user write and iterate on the Prompt Criteria (review instructions) and test on a small document set until aiR’s recommendations align sufficiently with expected relevance and issue classifications.
  2. Validate—user leverages the integration between aiR for Review and Review Center to compare results and validate the Prompt Criteria.
  3. Apply—user applies the verified Prompt Criteria on much larger sets of documents.

The Prompt Criteria validation process covers phase 2, Validate.

High-level Prompt Criteria validation workflow

The diagram details the steps (in orange) for Prompt Criteria validation, which can occur after the Prompt Criteria develop phase.

High level diagram of the prompt validation workflow

The Validate and Apply phases involve several steps that cross between aiR for Review and Review Center:

Process Flow Application Used

1. Identify target review set & choose validation settings.

Set up the validation sample by choosing the sample size, desired margin of error for the validation statistics, and other settings.

2. Run aiR for Review on the sample to receive predictions.

When a validation sample is created, a Review Center queue is automatically created. Run the sample documents through aiR for Review to obtain the relevance predictions that will be used for comparison with the manual human reviews in Review Center.

 aiR for Review

3. Manually review and code the documents in the validation sample.

Human reviewers code the documents in the sample for comparison to the aiR predictions. For the sake of validation, the human coding decisions are considered "correct." The Review Center dashboard tracks reviewers’ progress and compares their choices to aiR.

4. Evaluate the statistical results of the validation.

After human reviewers finish reviewing the validation sample, final validation statistics display comparing their results with aiR for Review’s predictions. These results are then evaluated.

5. Accept or reject the validation results.

After reviewing the results, decide whether to accept or reject them. If the results are accepted, the validated Prompt Criteria can be used for all documents in the target data source, and the process moves to aiR for Review to apply the criteria to the larger data set. If they are rejected, the team goes back to continuing to develop the Prompt Criteria.

 Review Center
6. Optionally apply the accepted validated Prompt Criteria against the entire target population.

The validated Prompt Criteria can be applied to the entire target population by creating an Apply project set. See Using project sets for more information.

 aiR for Review