Revising the prompt criteria

After running an aiR for Review job for the first time on the sample set, the initial results on the dashboard can be used as feedback for improving the prompt criteria. The cycle of examining the results, revising the prompt criteria, then running a new job on the sample documents is known as iterating on the prompt criteria. Refer to Best practices for more information. Also see Job capacity, size limitations, and speed for details on document and prompt limits.

In particular, ask the following questions about each document:

  • Did aiR for Review and the human reviewer agree on the relevance of the document?
  • Read the aiR for Review rationale and considerations. Do they make sense?
  • Do the citations make sense?

For all of these, if you see something incorrect, make notes on where aiR seems to be confused and rephrase the prompts. Here are the most common sources of confusion:

  • Insufficient context—For example, an internal acronym, key person, or code word may not have been defined. To fix this, add it to the proper section of the Case Summary tab.
  • Ambiguous instructions or unclear language—To fix this, edit the instructions on the Relevance, Key Documents, or Issues tabs.

In general, consider how you would help a human reviewer making the same mistakes. For example, if aiR for Review is having trouble identifying a specific issue, try explaining the criteria for that issue with simpler language.

After you have revised the prompt criteria to address any weak points, run the analysis again. Continue refining the prompt criteria until results accurately predicts the human coding decisions for all test documents in the sample.

aiR for Review only looks at the extracted text of each document. If a human reviewer marked a document as relevant because of an attachment or other criteria beyond the extracted text, aiR for Review will not be able to match that relevance decision.

For additional resources, refer to these articles on the Community site:

Increasing the job size

When aiR for Review accurately matches human coding decisions on the initial sample documents, increase the sample size. Typically, we recommend starting with an initial sample of about 50 documents, then increasing it to include another 50. However, you may find a different number works better for your project.

To increase the job size:

  1. Add the fresh documents to the saved search that acts as the project's data source. For more information about saved searches, see Creating or editing a saved search.
  2. Have a skilled human reviewer review the fresh documents. We recommend doing this before running aiR for Review, so that the reviewer is not biased by aiR's predictions.
  3. On the aiR for Review Projects tab, select the project.
  4. At the top of the project dashboard, click the refresh symbol next to the data source's name.
  5. In the Project Metrics section, click Not Analyzed. This selects the new documents.
  6. After the document count has updated, click Analyze [X] documents.
    The analysis job runs on the new documents, while the previously run documents keep their old results.

After you have run the job on the larger sample, continue revising the prompt criteria until it returns satisfactory results. Continue to increase the job size incrementally until you feel satisfied with the prompt criteria. After that, use the refined prompt criteria on the larger set of documents. You can do this either from the dashboard, or as a mass operation.