Best practices

Refer to the tips and recommendations below for prompt criteria and sample document sets to effectively use aiR for Review.

Tips for writing prompt criteria

The prompt criteria entered often aligns with a traditional review protocol or case brief in that they describe the matter, entities involved, and what is relevant to the legal issues at hand.

When writing prompt criteria, use natural language to describe why particular types of documents should be considered relevant. Write them as though you were describing them to a human reviewer.

• Write clearly—use active voice, use natural speaking phrases and terms, be explicit.
• Be concise—write as if "less is more," summarize lengthy text or only include key passages from a long review protocol. The prompt criteria have an overall length limit of 15,000 characters.
• Simply describe the case—do not give commands, such as “you will review XX."
• Use positive phrasing—phrase instructions in a positive way when possible. Avoid negatives ("not" statements) and double negatives.
• Use natural writing format styles—use whatever writing format makes the most sense to a human reader. For example, bullet points might be useful for the People and Aliases section, but paragraphs might make sense in another section. Clear formatting, layout, and spelling of information aids aiR's logical processing.
• Is it important?—ask yourself will the criteria affect the results, it is essential.
• Avoid legal jargon or explanations—for example, don't use legal jargon, such as "including but not limited to" and "any and all," and don't include explanations of the law.
• Use ALL CAPS—helps identify essential information for the model to focus on, for example use "MUST" instead of "should."
• Identify internal jargon and phrases—the learning language model (LLM) has essentially “read the whole Internet.” It understands widely used slang and abbreviations, but it does not necessarily know jargon or phrases that are internal to an organization.
• Identify aliases, nicknames, and uncommon acronyms—for example, a nickname for William may be Bill, or BT may be an abbreviation for the company name Big Thorium.
• Identify unfamiliar emails—normal company email addresses do not need identified, but unfamiliar ones should, for example Dave Smith may use Dave.Smith@AcmeCompany.com and skippy78@gmail.com.
• Iterate, iterate, iterate—test the prompt criteria and review the results, adjust it to obtain more accurate predictions and results.

Refer to the helper examples in the prompt criteria text boxes of the dialogs for additional guidance entering criteria in each field.

For additional guidance on prompt writing, see the following resources on the Community site:

• aiR for Review Prompt Writing Best Practices—downloadable .pdf of writing guidelines
• aiR for Review example project—detailed example of adapting a review protocol into prompt criteria
• AI Advantage: Aiming for Prompt Perfection?—on-demand webinar discussing prompt criteria creation

Recommended prompt criteria iteration workflow

We recommend the following workflow for developing prompt criteria:

1. For your first analysis, run the prompt criteria on a saved search of 50-100 test documents that are a mix of relevant, not relevant, and challenging documents.
2. Compare the results to human coding. In particular, look for documents that the application coded differently than the humans did and investigate possible reasons. This could include unclear instructions, needing to define an acronym or code word, or other blind spots in the prompt criteria.
3. Tweak the prompt criteria to adjust for blind spots.
4. Repeat steps 1 through 3 until the application predicts coding decisions accurately for the test documents.
5. Test the prompt criteria on a sample of 50 more documents and compare results. Continue tweaking and adding documents until you are satisfied with the results for a diverse range of documents.
6. Finally, run the prompt criteria on a larger set of documents.

aiR for Review only sees the extracted text of a document. It does not see any non-text elements like advanced formatting, embedded images, or videos. We do not recommend using aiR for Review on documents such as images, videos, or spreadsheets with heavy formulas. Instead, use it on documents whose extracted text accurately represents their content and meaning.

Tips for creating sample documents set

Before setting up the aiR for Review project, create a saved search that contains a small representative set of documents for testing the aiR for Review prompt criteria during the initial development phase. This sample set is intended to be similar to those used when training a review team. See Creating or editing a saved search for details about saved searches.

Here are some guidelines:

• Deduplication—Reduce the size of the population. For example, remove duplicate information, documents with no text, too much text, low extracted text, and password protected files. Possible tools to use: Email threading to focus on only inclusive emails, Textual near duplicate (TND), Hash duplicates of attachments and loose records, run extracted text size script, and remove documents with no text or too much text.
• Diversity —Ensure diversity in the sample so that it covers all categories of interest to provide representation across what you’re likely to see. Possible tools to use: Clustering to pull documents from a variety of clusters, Stratified sampling script, Targeted searching, and Random sampling of rich populations.
• Richness—aim for a rich sample containing a good representation of relevance and issues. Recommendation is 50-75% relevant documents and 25-50% not relevant documents. Possible tools to use: Data filters, Search terms report (STR), Key custodians to focus on, and already identified relevant documents.
• Size—include 50-100 documents to start that are a mix of relevant, not relevant, and challenging documents, then expand the number gradually as needed with the prompt iterations.
• Key features—make sure the documents highlight all the key features of your relevance criteria.
• Human reviews—have human reviewers code the documents in advance.

For more information about choosing documents for the sample, see Selecting a Prompt Criteria Iteration Sample for aiR for Review on the Community site.