Analytics indexes

LSI is a wholly mathematical approach to indexing documents. Instead of using any outside word lists, such as a dictionary or thesaurus, LSI leverages sophisticated mathematics to discover term correlations and conceptuality within documents. LSI is language-agnostic, meaning that you can index any language and it learns that language. LSI enables Analytics to learn the language and, ultimately, the conceptuality of each document by first processing a set of data called a training set. The training set of documents may be the same as the set of documents that you want to index or categorize. Alternatively, it may be a subset of these documents, or it could be a completely different set of documents. This training set is used to build a concept space in the Analytics index.

Using LSI, Analytics inspects all the meaningful terms within a document and uses this holistic inspection to give the document a position within a spatial index. The benefits of this approach include the following:

• Analytics learns term correlations (interrelationships) and conceptuality based on the documents being indexed. Therefore, it always is up-to-date in its linguistic understanding.
• Analytics indexes are always in memory when being worked with, so response time is very fast.
• Analytics is inherently language agnostic. It can index most languages and accommodate searches in those same languages without additional training. We recommend creating separate indexes for large populations of different language documents.

Concept space

When you create an Analytics index, Relativity uses the training set of documents to build a mathematical model called a concept space. The documents you are indexing or categorizing can be mapped into this concept space. While this mathematical concept space is many-dimensional, you can think of it in terms of a three-dimensional space. The training set of documents enables the system to size the concept space and create the algorithm to map searchable documents into the concept space. In the concept space, documents that are closer together are more conceptually similar than documents that are further from each other.

Concept rank

Throughout Analytics, item similarity is measured using a rank value. Depending on the feature, the rank may be referred to as a coherence score, rank, or threshold. In each scenario, the number represents the same thing.

Because the Analytics engine builds a spatial index, every document has a spatial relationship to every other document. Additionally, every term has a spatial relationship to every other term.

The concept rank is an indication of distance between two items. In the Categorization feature, it indicates the distance between the example document and the resulting document. In Keyword Expansion, it indicates the distance between the two words. The rank does not indicate a percentage of relevance.

For example, when running a concept search, the document that is closest to the query is returned with the highest conceptual score. A higher score means the document is more conceptually related to the query. Remember that this number is not a confidence score or a percentage of shared terms, it is a measurement of distance.

SVM is used solely by Active Learning. With SVM, you don't need to provide the Analytics index with any training text. The system learns from your reviewers and constantly updates the model. SVM predicts the relevance of uncoded documents based on their distance to the hyperplane. This differs from Latent Semantic Indexing (LSI) which is also composed of a multi-dimensional space, but uses a nearest-neighbor approach to predict documents.

Hyperplane

When you create a Classification-type index, Relativity takes reviewer's coding decisions and pulls them into a high-dimensional model. The hyperplane helps differentiate between relevant and not relevant documents.

After the hyperplane is established, all documents without a coding decision are pulled into the model and mapped on either side of the hyperplane based on the model's current understanding of the difference between relevant and not relevant.

Rank

Rank measures the strength or confidence the model has in a document being relevant or not relevant. Rank is measured on a scale from 100 to 0. As documents move farther away from the hyperplane on either side, their score is pushed closer to 100 or 0.

1. Click the Indexing & Analytics tab and select Analytic Indexes.
2. Click New Analytics Index. The Analytics Index Information form appears.
3. Complete the fields on the Analytics Index Information form. See Index creation fields. Fields in orange are required.
4. Click Save.

   Note: If no documents appear in the saved search, or if the search contains fields that would cause the index to error, a warning message appears upon clicking Save.

When you save the new index, the Analytics Index console becomes available. See Analytics index console operations.

Index creation fields

The Analytics Index Information form contains the following fields:

Index Information fields

• Name - the name of the index. This value appears in the search drop-down menu after you activate the index.
• Order - the number that represents the position of the index in the search drop-down menu. The lowest-numbered index is at the top. Items that share the same value are sorted in alphanumeric order.
• Index type - select Conceptual.
• Searchable set - the document set searched when you use the Analytics index. This set should include all of the documents on which you want to perform any Analytics function. Only documents included in the searchable set are returned when you run clustering, categorization, or any other Analytics feature. See Searchable set considerations for more information on creating an optimized searchable set search.

  Note: Indexes no longer have the default saved searches available to set as the Searchable set (Relativity 9.5.196.102 and above). For older indexes that utilize the default saved searches, a new saved search must be set when editing the index after upgrading to Relativity 9.5.196.102 or above. If you do not choose a new search, you will only be able to build and activate or deactivate the index.

• Email notification recipients - send email notifications when your index successfully completes, fails, or when the index is disabled because it is not being used. Enter the email address(es) of the recipient(s). Separate entries with a semicolon. A message is sent for both automated and manual index building.

Advanced Settings fields

• Relativity Analytics server - select the Analytics server that should be associated with the Analytics index. This list includes only Analytics servers added to the resource pool for the workspace.See Servers for more information.
• Training set - the document set from which the Analytics engine learns word relationships to create the concept index. By default, this is the same search as the searchable set. See Training set considerations for more information including considerations when using Data Grid.

Note: Indexes no longer have the default saved searches available to set as the Training set (Relativity 9.5.196.102 and above). For older indexes that utilize the default saved searches, a new saved search must be set when editing the index after upgrading to Relativity 9.5.196.102 or above. If you do not choose a new search, you will only be able to build and activate or deactivate the index.

• Remove documents that errored during population - setting this option to Yes removes documents that received errors during population so that you don't have to manually remove them from the population table in the database. Setting this option to No means that you must click the Remove Documents in Error button after population if you want to remove those errored documents. Documents that are removed when you select this are listed in the Excluded from searchable and/or training field(s) in the field tree browser. Documents that are removed by this automated process are never again included in a build for as long as the index exists.
• Continue index steps to completion - select Yes for this option to allow index population to proceed through the entire indexing process from population of the index to building the index, to activating the index without having to manually perform each of those steps.
• Concept Stop Words - determines the words you want the conceptual index to suppress. You can add or remove stop words from the list. Separate each word with a hard return. If you need to modify this list after index build, you will need to re-build the index. Please note that a Full Population is not required, as the concept stop words are applied during the Build phase of index creation.
  Concept stop words are very common terms that are filtered from the Analytics index in order to improve quality. By default, the Analytics index contains a Concept Stop Words field of only English stop words. These are customizable by the user in Relativity. The English stop words should be replaced by the appropriate stop words for the language being indexed. Analytics classification indexes are copied over when a workspace is used as a template; it is recommended that you create multiple indexes in the workspace template for each commonly indexed language.
  We do not publish stop word lists. The following are three places where you can find stop word lists:
  • Translating the English stop words list to the desired language manually
  • Ranks NL (http://www.ranks.nl/stopwords) - 40 languages are available.
  • Microsoft SQL Server Stop Lists - 33 languages are available. Perform a query as follows:

  SELECT LCID, Name FROM sys.syslanguages
  
  SELECT * FROM sys.fulltext_system_stopwords WHERE language_id = ####
  

• Optimize training set - when checked, this specifies whether the Analytics engine automatically includes only conceptually valuable documents in the training set while excluding conceptually irrelevant documents. Enabling this eliminates the need to manually remove bad training documents from the training set.
  • Yes - includes only quality training documents in the training set. For more information, see Optimize training set.
  • No - puts all documents from the data source selected for the Training set field into the training set without excluding conceptually-irrelevant documents.

• Dimensions - determines the dimensions of the concept space into which documents are mapped when the index is built; more dimensions increase the conceptual values applied to documents and refine the relationships between documents. The default setting is 100 dimensions.

  Note: A larger number of dimensions can lead to more nuances due to more subtle correlations that the system detects between documents. However, higher dimensionality requires more resources from the Analytics server, especially RAM memory. Higher dimensionality has a diminishing return on results once you exceed 300 or more dimensions.

• Remove English email signatures and footers - removes signatures and footers from emails containing English characters only. By default, this is set to Yes for new indexes and No to existing ones. Enabling this tells the Analytics engine to auto-detect and suppress email confidentiality footers and signatures from the text it ingests, so that the index is free of any text that is extraneous to the authored content. Setting this to Yes could slow down index population speed; however we strongly recommend against turning this off. When this is set to No, the Enable email header filter options are enabled.
  Setting this to Yes does the following during the index population:
  • Enables the email header
• Enable email header filter - when set to Yes, this filter removes common header fields (such as To, From, and Date) and reply-indicator lines, but it does not remove content from the Subject line. Use this filter to ensure that the headers in the concept space don't overshadow the authored content. This prevents the Analytics engine from discovering unwanted term correlations and including commonly occurring, low-content terms, such as To, From, and others. We recommend setting this filter on all indexes. To set this option to No, you must set Remove English email signatures and footers to No.

Note: Relativity recognizes words as being part of the email header only when they're at the beginning of a new line and followed by a colon. The email filter also filters out subsequent lines, but only if the new line begins with whitespace. For example, if the email contains a BCC section that's multiple lines, each subsequent line would have to begin with a space, otherwise it's not filtered out.

Note: Classification indexes are used only by Active Learning projects.

1. Click the Indexing & Analytics tab and select Analytic Indexes.
2. Click New Analytics Index. The Analytics Index Information form appears.
3. Complete the fields on the Analytics Index Information form. See Index creation fields. Fields in orange are required.
4. Click Save.

   Note: If no documents appear in the saved search, or if the search contains fields that would cause the index to error, a warning message appears upon clicking Save.
   

When you save the new index, the Analytics Index console becomes available. See Analytics index console operations.

Index creation fields

The Analytics Index Information form contains the following fields:

Index Information fields

• Name - the name of the index. This value appears in the search drop-down menu after you activate the index.
• Index type - select Classification.
• Searchable set - the document set searched when you use the Analytics index. This set should include all of the documents on which you want to perform any Analytics function. Only documents included in the searchable set are returned when you run clustering, categorization, or any other Analytics feature. See Searchable set considerations for more information on creating an optimized searchable set search.
  Notes:
  • For best results, we recommend no more than 9 million documents in the search.
  • Indexes no longer have the default saved searches available to set as the Searchable set (Relativity 9.5.196.102 and above). For older indexes that utilize the default saved searches, a new saved search must be set when editing the index after upgrading to Relativity 9.5.196.102 or above.

• Email notification recipients - send email notifications when your index successfully completes, fails, or when the index is disabled because it is not being used. Enter the email address(es) of the recipient(s). Separate entries with a semicolon. A message is sent for both automated and manual index building.

Advanced Settings fields

• Relativity Analytics server - select the Analytics server that should be associated with the Analytics index. This list includes only Analytics servers added to the resource pool for the workspace.See Servers for more information.

Training set considerations

Note: Training set considerations only apply to conceptual indexes.

A training set is a set of documents that the system uses to learn the language, the correlation between terms, and the conceptual value of documents. This set formulates the mapping scheme of all documents into the concept space. Because the system uses this set to learn, include only the highest quality documents when creating the training set. The system needs authored content with conceptually relevant text to learn the language and concepts.

Use the following settings when creating a training set saved search:

• Only the authored content of the document should be indexed. Typically, the Extracted Text field is the only field returned as a column in a training set saved search. If there is authored content in another long text field (i.e., OCR Text, Translation, etc.), then this field should be included as well. You should never train on any sort of metadata fields like Custodian or Email Author or system fields like Control Number. Including these fields skews your results when trying to find conceptually related documents.

  Note: Do not index single choice, multiple choice, or multiple object fields.
  

• Documents that don't contain authored content or conceptual data are useless for training the Analytics index. The following types of documents should be excluded from the training set:
  • Compressed files (like ZIP or RAR files)
  • System files
  • Excel files with mostly numbers
  • Image files
  • CAD drawings
  • Maps
  • Calendar items
  • Documents with OCR errors, such as OCR not processed
  • Documents with poor quality OCR, such as OCR on handwritten documents.
  • Documents with little or no text (less than 0.2 KB)
• During population, words beginning with a number are not included in the index (ex. 4ward). Words ending in a number (mp3) or with numbers embedded (passw0rd) are indexed as written.

Optimize training set

When you select the Optimize training set feature on an Analytics index, you improve the quality of that index by excluding documents that could result in inaccurate term correlations due to their low conceptual value, such as:

• Very short documents
• Very long documents
• Lists containing a significant amount of numbers
• Spreadsheet-like documents
• System log files
• Text resulting from processing errors

To perform this automatic removal of bad documents from the training set, the Analytics engine evaluates documents based on:

• Word count
• Uniqueness
• Number count
• Punctuation marks
• Words with many characters (50+)

If the training set optimization excludes a document, the following results are possible:

• If the document is designated to be populated as both a training and searchable document, Relativity populates it as a searchable document only. The document could be returned in a concept search, assuming it meets the minimum rank.
• If the document is designated to be populated only as a training document, Relativity doesn't populate it into the index at all.
• If the document is designated to be populated as a searchable document only, the Analytics engine doesn't examine it.

Searchable set considerations

For conceptual indexes, the searchable set is the collection of documents to be clustered, categorized, or returned in a concept query. The searchable set of documents is typically larger than the training set. There are fewer documents culled from the searchable set.

For classification indexes, the searchable set is the collection of documents to be ranked by the Active Learning model. The searchable set must contain example documents to train the model.

Use the following settings when creating a searchable set saved search:

• Index only the authored content of the document. We recommend returning as few fields as possible. Typically, the Extracted Text field is the only field returned as a column in a searchable set saved search. If there is authored content in another long text field (i.e., OCR Text, Translation, etc.), include this field as well. Returning email header fields in your search may cause recipient names and email address components to appear in clusters.

  Note: Do not index single choice, multiple choice, or multiple object fields.

• Documents that don't contain any conceptual data cannot be returned by any conceptual analytics operations. Consider excluding the following types of documents from the searchable set:
  • Compressed files (like ZIP or RAR files)
  • System files
  • Excel files with mostly numbers
  • Image files
  • CAD drawings
  • Maps
• During population, words beginning with a number are not included in the index (ex. 4ward). Words ending in a number (mp3) or with numbers embedded (passw0rd) are indexed as written.

Identifying training and searchable set documents

When you populate the index, the Conceptual Index multi-choice field on the Document object lists whether a document is included in the training set, searchable set, or both. This field is populated every time the index is populated with a full or incremental population. You can use this field as a condition in a saved search to return only training or searchable set documents.

You can also find searchable and training set documents in the field tree, as well as those which were excluded from training when you enabled the Optimize training set field on the index.