

Unlike traditional searching methods like dtSearch, Analytics is an entirely mathematical approach to indexing documents. It doesn’t use any outside word lists, such as dictionaries or thesauri, and it isn’t limited to a specific set of languages. Unlike textual indexing, word order is not a factor.
The basis of conceptual analytics and Active Learning is an Analytics index. There are two types of indexes:
Note: The searchable set and training set used for Analytics index creation are now referred to as the data source and training data source. See Creating an Analytics index.
You can run the following Analytics operations on documents indexed by a conceptual index:
You're a system admin and you need to create an Analytics index to help you organize and assess the semantic content of a large data set. One of your firm's clients, a large construction company, recently became involved in litigation regarding the use of materials that they weren’t informed were potentially environmentally damaging when they purchased them from a major supplier. Subsequently, they handed over to you a large group of documents related to hazardous substances that they suspect are found in their building materials.
Before creating the index, you run repeated content identification to find disclaimers and confidentiality footers in the dataset and link them to the Analytics profile.
You create a new index with the name of "Hazardous Materials" and for the Analytics profile, you select the one you already created for this index, which you called Hazardous Materials Profile. You leave the Order field at its default value.
Now, for the training data source, or the document set from which the Analytics engine learns word relationships to create the concept index, you select a saved search that you already created, which contains conceptually rich documents and excludes any computer-generated files. For the main data source, you use a saved search which excludes non-conceptual data. You leave the Optimize training set and Automatically remove signatures and footers fields at their default values and save the index.
Then you go through the steps of populating and building the index. Once the index is built and active, you can cluster, categorize, and start diving deeper into the data your client provided you.
LSI is a wholly mathematical approach to indexing documents. Instead of using any outside word lists, such as a dictionary or thesaurus, LSI leverages sophisticated mathematics to discover term correlations and conceptuality within documents. LSI is language-agnostic, meaning that you can index any language and it learns that language. LSI enables Relativity Analytics to learn the language and, ultimately, the conceptuality of each document by first processing a set of data called a training data source. The training data source may be the same as the set of documents that you want to index or categorize. Alternatively, it may be a subset of these documents, or it could be a completely different set of documents. This training data source is used to build a concept space in the Analytics index.
Using LSI, Analytics inspects all the meaningful terms within a document and uses this holistic inspection to give the document a position within a spatial index. The benefits of this approach include the following:
When you create an Analytics index, Relativity uses the training data source to build a mathematical model called a concept space. The documents you are indexing or categorizing can be mapped into this concept space. While this mathematical concept space is many-dimensional, you can think of it in terms of a three-dimensional space. The training data source enables the system to size the concept space and create the algorithm to map searchable documents into the concept space. In the concept space, documents that are closer together are more conceptually similar than documents that are further from each other.
Throughout Analytics, item similarity is measured using a rank value. Depending on the feature, the rank may be referred to as a coherence score, rank, or threshold. In each scenario, the number represents the same thing.
Because the Analytics engine builds a spatial index, every document has a spatial relationship to every other document. Additionally, every term has a spatial relationship to every other term.
The concept rank is an indication of distance between two items. In the Categorization feature, it indicates the distance between the example document and the resulting document. In Keyword Expansion, it indicates the distance between the two words. The rank does not indicate a percentage of relevance.
For example, when running a concept search, the document that is closest to the query is returned with the highest conceptual score. A higher score means the document is more conceptually related to the query. Remember that this number is not a confidence score or a percentage of shared terms, it is a measurement of distance.
SVM is used solely by Active Learning. With SVM, you don't need to provide the Analytics index with any training text. The system learns from your reviewers and constantly updates the model. SVM predicts the relevance of uncoded documents based on their distance to the hyperplane. This differs from Latent Semantic Indexing (LSI) which is also composed of a multi-dimensional space, but uses a nearest-neighbor approach to predict documents.
When you create a Classification-type index, Relativity takes reviewer's coding decisions and pulls them into a high-dimensional model. The hyperplane helps differentiate between relevant and not relevant documents.
After the hyperplane is established, all documents without a coding decision are pulled into the model and mapped on either side of the hyperplane based on the model's current understanding of the difference between relevant and not relevant.
Rank measures the strength or confidence the model has in a document being relevant or not relevant. Rank is measured on a scale from 100 to 0. As documents move farther away from the hyperplane on either side, their score is pushed closer to 100 or 0.
Analytics uses only the documents you provide to make a search index. Because no outside word lists are used, you must create saved searches to dictate which documents are used to build the index. However, if you want to limit search results to certain document groups or have more than one language in the document set, multiple indexes might give you better results.
Note: Permissions for the Search Index object must be kept in sync with permissions on the Analytics Index object.
Note: If no documents appear in the saved search, or if the search contains fields other than the extracted text field, a warning message appears upon clicking Save.
When you save the new index, the Analytics Index console becomes available. See Analytics index console operations.
The Analytics Index Information form contains the following fields:
Note: A larger number of dimensions can lead to more nuances due to more subtle correlations that the system detects between documents. However, h
SELECT LCID, Name FROM sys.syslanguages
SELECT * FROM sys.fulltext_system_stopwords WHERE language_id = ####
Note: Classification indexes are used only by Active Learning projects.
Note: If no documents appear in the saved search, or if the search contains fields that would cause the index to error, a warning message appears upon clicking Save.
When you save the new index, the Analytics Index console becomes available. See Analytics index console operations.
The Analytics Index Information form contains the following fields:
If you want to apply item-level or workspace-level security to an Analytics index, you must secure both the Analytics Index object and the Search Index object for that particular index.
Restricting a group from viewing an Analytics Index does not restrict them from searching on the index unless access to the corresponding Search Index is also restricted.
Note: If you’re applying item-level security from the Search Indexes tab, you may need to create a new view and add the security field to the view.
Note: Training data source considerations only apply to conceptual indexes.
A training data source is a set of documents that the system uses to learn the language, the correlation between terms, and the conceptual value of documents. This data source formulates the mapping scheme of all documents into the concept space. Because the system uses this data source to learn, include only the highest quality documents when creating the training data source. The system needs authored content with conceptually relevant text to learn the language and concepts.
Use the following settings when creating a saved search to use as a training data source:
Note: Do not index single choice, multiple choice, multiple object, or any number fields, such as Extracted Text Size.
When you select the Optimize training set feature on an Analytics index, you improve the quality of that index by excluding documents that could result in inaccurate term correlations due to their low conceptual value, such as:
To perform this automatic removal of bad documents from the training data source, the Analytics engine evaluates documents based on:
If the optimization excludes a document, the following results are possible:
For conceptual indexes, the data source is the collection of documents to be clustered, categorized, or returned in a concept query. The data source is typically larger than the training data source. There are fewer documents culled from the data source.
For classification indexes, the data source is the collection of documents to be ranked by the Active Learning model. The data source must contain example documents to train the model.
Use the following settings when creating a saved search to use as a data source:
Note: Do not index single choice, multiple choice, multiple object, or any number fields, such as Extracted Text Size.
Note: Analytics indexes automatically suppress documents larger than 30 MB before sending them to the Analytics engine, so removing these yourself is not required. However, because the suppression process takes time, removing them in advance can make an index build more quickly. Removing other large, non-conceptual files such as number-heavy Excel files gives the same benefit.
When you populate the index, the Conceptual Index multi-choice field on the Document object lists whether a document is included in the data source, training data source, or both. This field is populated every time the index is populated with a full or incremental population. You can use this field as a condition in a saved search to return only training or data source documents.
You can also find data source and training data source documents in the field tree, as well as those which were excluded from training when you enabled the Optimize training set field on the index.
Index sizes are limited by default, and some large documents are excluded from indexing as follows.
By default, indexes are limited by the following parameters:
Indexes can include up to 12 million documents each.
Conceptual indexes can reach up to 60 GB in size.
These limits can be changed by the administrator. For help with adjusting index size limits, contact Relativity Support.
Analytics indexes automatically suppress documents larger than 30 MB before sending them to the Analytics engine. Suppressed large documents will appear in the Document Exceptions. You can also view suppressed documents from the Document list by using the Excluded from Training and Excluded from Searchable Set choices on the Analytics Index Document field.
Once you save the Analytics index, the Analytics index console appears. From the Analytics index console, you can perform the following operations:
Note: Population statistics and index statistics are only available for conceptual indexes.
To populate the Analytics index on the full set of documents, click Run on the Analytics Index console, then choose Full from the modal that appears. This adds all documents from the data source and training data source to the ready-to-index list. Document “preprocessing” also occurs to clean up text. This includes the following:
Once population is complete, the index builds.
Note: If you have access to SQL, you can change the priority of any Analytics index-related job (index build, population, etc.) by changing the value on the Priority column in the ContentAnalystIndexJob database table for that index. This column is null by default, null being the lowest priority. The higher you make the number in the Priority column, the higher priority that job becomes. When you change the priority of a job while another job is in progress, Analytics doesn't stop the in-progress job. Instead, the job will finish before starting on the new highest priority.
While the index is populating, the following console option becomes available:
Once population is complete, you have the option to populate incrementally to account for new or removed documents from the data source and training data source on the ready-to-index list. To perform an incremental build, click Run on the console, then choose Incremental from the modal that appears. See Incremental population considerations for conceptual indexes for more information.
Once population is complete, the index will build automatically. During this phase, training data source documents and Latent Semantic Indexing (LSI) are used to build the concept space based on the relationships between words and documents. Data source documents are mapped into the concept space, and stop/noise words (very common words) are filtered from the index to improve quality.
Please note that the index is unavailable for searching during this phase.
You can monitor the progress of any Analytics index process with the progress panel at the top of the layout.
(Click to expand)
Population and index building occurs in the following stages, which will appear within the progress panel:
Building
The following fields appear in the Document Breakdown section:
Note: If an Analytics index goes unused for 15 days, it is automatically disabled to conserve server resources. It then has a status of Inactive and is not available for use until it is activated again. This setting is determined by the
Building a conceptual index automatically activates it. This makes the index available for users by adding the index to the search drop-down menu on the Documents tab and to the right-click menu in the viewer. All active indexes are searchable.
Note: Activating an index loads the index's data into RAM on the Analytics server. Enabling a large number of indexes at the same time can consume much of the memory on the Analytics server, so you should typically only leave indexes active that are actively querying or classifying documents.
Once a conceptual index is activated, you have the option of deactivating it.
You may need to deactivate an index for the following reasons:
To deactivate an index, click Deactivate Index on the console. A yellow banner will appear at the top of the console.
To reactivate the index, click Reactivate Index on the banner.
Note: If you deactivate an index, you can't run concept searches against the index and keyword expansion becomes unavailable on the index.
If index-level exceptions occur while populating or building a classification index, you have the option of retrying them from the console. To do this, click Retry Exceptions.
If exceptions occur while populating or building a conceptual index, the system will retry them automatically.
Retrying exceptions attempts to populate the index again.
Note: You can only populate one index at a time. If you submit more than one index for population, they'll be processed in order of submission by default.
When errored documents are removed from population in a conceptual index, they appear on the index console in the Document Exceptions panel. This panel only appears when exceptions exist.
The panel includes the following fields:
To see a list of population statistics, click Show Population Statistics.
Note: Population statistics are only available for conceptual indexes.
This option is available immediately after you save the index, but all rows in this window display a value of 0 until population is started.
This displays a list of population statistics that includes the following fields:
To see an in-depth set of index details, click Show Index Statistics. This information can be helpful when investigating issues with your index.
Note: Index statistics are only available for conceptual indexes.
Clicking this displays a view with the following fields:
There may be times when you need to update your index. Depending on the update you’re making, you can save time by running an incremental population or only running a build. The following table outlines various workflows for different index updates.
Workflow | Index update |
---|---|
Adding new documents that:
|
|
Adding new documents that:
|
|
Removing documents from the data source or training data source |
|
Updating stop/noise words |
|
Updating extracted text (ex. Updating poor quality OCR text) |
|
Updating filters (email header, repeated content) |
|
Incremental populations don't necessarily force Analytics to go through every stage of an index build.
When managing or updating indexes with new documents, consider the following guidelines:
You can run an incremental population to add or remove documents from your data source and training data source. This results in an index taking substantially less time to build, and therefore less downtime.
To perform an incremental population, click Run on the console, then choose Incremental from the modal that appears. This checks for changes in both the data source and training data source and updates the index to match.
If extracted text has changed, you have updated the stop/noise words, or you have applied different filters, you must run a full population.
Repeated content filters can be linked to an Analytics index either automatically, using the top filters chosen by the system, or by manually selecting individual filters. These linked filters will only apply to the currently open Analytics conceptual index; they will not be applied to structured analytics sets. These can only be linked to conceptual indexes, not classification indexes.
The maximum number of linked repeated content filters per index is 1,000. This includes both manually and automatically linked filters.
By default, when a conceptual index runs, it will automatically link the top 200 repeated content filters to the index. These are chosen by multiplying the number of occurrences times word count, then selecting the top 200 in descending order.
The following settings apply when automatically linking repeated content filters:
The default number of linked filters is 200, and this number is controlled by the Repeated content filters to link field on the index creation screen. You can change this to any number from 0 to 1000 (inclusive) when you create the index. For more information, see Index creation fields.
If a repeated content filter has the Ready to index field set to No, it will be excluded from linking to the index, even if it would otherwise be counted as a top filter.
If a repeated content filter has the Ready to index field set to Yes, it will be included as an automatically linked filter, even if it is not a top filter according to the calculation. It will count towards the number of filters set in the Repeated content filters to link field. For example, if the Repeated content filters to link field is set to 200, and there are 5 low-ranked filters which have Ready to index set to Yes, the index will link those 5 plus the top 195 filters for a total of 200.
The linking process takes place every time the conceptual index runs. This means that if some repeated content filters are deleted, or if the Ready to index field values change, these changes will be reflected after the next index re-run.
If a conceptual index has both manually and automatically linked filters attached, the manually linked ones will not be changed by the index re-runs and will remain linked. Manually linked filters do not count towards the number in the Repeated content filters to link field, but they do count towards the 1,000 filter maximum.
Use the Repeated Content Filters section on an Analytics index layout to manually link repeated content filters when the Analytics index is not open in Edit mode.
To manually link one or more existing repeated content filters to an Analytics index, perform the following steps:
For more information on repeated content and regular expression filters, see Repeated content filters.
On this page
Why was this not helpful?
Check one that applies.
Thank you for your feedback.
Want to tell us more?
Great!