dtSearch

dtSearch icon

Relativity's dtSearch engine provides advanced search functionality such as proximity, stemming, and fuzzy searches across any field type. It also supports the use of Boolean operators and custom noise word lists and the basic searching features available in keyword searches. After building your dtSearch index, the Dictionary search option becomes available.

Note: Relativity partitions a single index into smaller indexes, called sub-indexes, which multiple workers build simultaneously. This increases performance by spreading out the work over a configurable number of agents. When you perform a search, Relativity runs your query on the smaller indexes in parallel. The application then federates and returns your results. For more details, see the Ask The Expert Training content: Searching: Best Practices for dtSearch Builds.

See these related pages:

Using dtSearch indexes

There are roughly three million files relevant to a case you are working on, including emails, email attachments, invoices, and technical manuals related to construction practices and material handling. It is early in the case, and you need to gain an understanding of the data set. You also need to retrieve certain text related to five substances that you know are prevalent in this data, as an employee from the construction company mentioned these specifically in an email to you. To do this, you need to be able to perform proximity, stemming, and fuzzy searches on your data set. So you create a new dtSearch index.

You call the index Hazardous Materials dtSearch so that you can identify it in the Search drop-down menu on the Documents list. You might also create an Analytics index for this case with a similar name, so make sure to differentiate them clearly. For the Searchable set field, you select a saved search that you have already created called Hazardous Materials searchable set, which has documents to which you have already applied keywords related to the substances mentioned in your client's email.

Because many of the invoices and emails in your data set contain references to various purchases of building materials made by various departments in the construction company you are helping to represent, you set the Auto-recognize date, email, and credit card numbers field to Yes.

You leave all other fields at their default settings and save the index. You then build and activate the index so that you can select it in the Search drop-down menu.

When you select the index and search your document set on it, you run proximity searches to see how close terms relating to hazardous substances occur to the names of the building materials that may or may not contain them. The searches you run include the following:

  • lead W/10 paint
  • lead W/10 plumbing pipes
  • lead W/10 connectors
  • lead W/10 solder
  • asbestos W/10 insulation and
  • asbestos W/10 pipe coverings
  • asphalt W/10 sealant
  • asphalt W/10 adhesives
  • radioactive W/10 fluorescent lamps
  • radioactive W/10 smoke detectors

As you keep running these proximity searches, you get down to a small group of intriguing emails between a prospective buyer, your client, and a prospective seller. This may prove that the seller had knowledge of the fact that those building materials were potentially dangerous when they were negotiating a price with your client. This discovery turns out to be crucial to the case.

dtSearch index

You can build custom dtSearch indexes for a subset of documents or for certain document fields in a workspace. You must have the appropriate permissions to complete this task. See Workspace security.

Before you begin, you need to create a saved search that includes the fields that you want to include in the index. You can then name the index based on the document search set used to create it.

Note: Within a field, dtSearch truncates any string longer than 32 characters that does not contain a space character. It indexes only the first 32 characters of the string. For more information, see Searching for words longer than 32 characters.

Accent-insensitive indexes

By default, Relativity builds an accent-insensitive index. In an accent-sensitive index, some characters translate to the base character, which causes those characters and any terms containing those characters to be treated the same in a search terms report.

Note: dtSearch uses .ABC files, but only for characters in the range from 33 to 127. Relativity handles all other characters according to the definitions in the Unicode character tables.

  • Example: Relativity converts accented characters like á or ñ to the unaccented versions, a or n.
  • Example: If you search for the term fröhlich, searching that term as fröhlich or frohlich would both return the hit. However, highlighting in the Viewer may not display both variations.

Creating a dtSearch index

To create a new dtSearch index:

  1. Use the search feature to navigate to the Search Indexes tab.
  2. Click New Search Index.
    The dtSearch Index Information form appears with required fields having an orange asterisks.
  3. Complete the fields on the dtSearch index form. See more details, see Fields.
  4. Click Save to display the index details page. The index details page now displays three additional read-only fields and the dtSearch index console. See Fields and dtSearch console.
  5. Click Build Index: Full. A dialog box asks you to verify that you want to run a full build. You can also select Activate this index upon completion. Indexes must be active to search them.
  6. Click OK to build your index.
    Note: Network problems can slow down your dtSearch builds. If a dtSearch manager or worker agent encounters a network-related error during the build process, it executes up to three retry attempts at 30-second intervals.
  7. If you did not select Activate this index upon completion in the dialog box, click Activate Index on the console. The index will not activate if there are errors. Activating an index makes it available in the Search menu.
  8. (Optional) Click Refresh Page at any point in the build to see the index's current build status. If errors occur during the build, the Retry Errors button enables on the console under the Errors and Status heading. Click this button to try to resolve any errors.

Once the index builds, the console enables more options. See dtSearch console.

Fields

The dtSearch index page includes the following fields:

dtSearch Index Information

Add information to the following dtSearch index fields:

  • Name—the dtSearch index name. This name appears within the search with menu in the Documents tab.
  • Order—the integer value, positive or negative, representing the position of the index in the search indexes list. Indexes sort from lowest, top, to highest, bottom, order number. Those with the same order number sort alphanumerically.
  • Searchable set—the saved search of documents for indexing. Relativity indexes the documents returned by the search as well as the returned documents' fields. It may use a dtSearch or an Analytics index. Make sure the index is active.
    Note: When creating a saved search for a dtSearch index, the best practice is to use long text fields. You will see a warning message, when creating a dtSearch index, if you select a saved search that does not contain at least one long text field.
  • Email notification recipients—enter the email addresses of those who should receive notifications of whether the dtSearch index build succeeded or failed.
  • Notes—enter any considerations, or thoughts, about the search index.
Searchable Set - Current Indexable Fields

This section is a list of the fields and field types in the saved search used for indexing.

Note: The fields listed may not match those indexed if someone made updates after indexing the searchable set.

If a long-text field not included in the searchable set, it's possible to not have search any search results returned. For better search results, check the saved search to ensure a field with the field type as Long Text. For example, extracted text is included.

Advanced Settings

Add information to the following advanced setting fields:

  • Auto recognize date, email, and credit card numbers—a yes/no field. See Auto-recognition for details.
  • Create accent sensitive—a yes/no field. When set to Yes, dtSearch indexes are sensitive to accents and other language-specific characters.
  • Skip malicious files—a yes/no field. Select Yes to skip potentially malicious quarantined files and continue processing the index. The final index breakdown includes the number of skipped documents.
  • Use Default Sub-index Size—select Yes to use the default sub-index size of 16 GB.

    Sub-index size is automatically optimized for system performance, unless you select No. Then you must enter sub-index size.

  • Custom Sub-Index Size (GB)—determines the size of each sub-index created when you generate a dtSearch index. The field must contain a numerical value between 1-16 GBs.

  • Index share—the location of the file share for storing the search index. Select the bold (default) location. You will see a warning about system performance if you select a location other than the default.

Noise Words

Edit the list of words that Relativity ignores during indexing.

Alphabet

Edit the index’s alphabet file. See Making a character searchable.

Note: If you search for long, uninterrupted strings that have no spaces or word breaks, such as when you have made a character searchable, dtSearch truncates the string after 32 characters and inserts a wildcard. For more information, see Searching for words longer than 32 characters.

dtSearch console

The dtSearch index console includes the following options:

  • Build Index: Full—creates a full build of the index. During the build, the button toggles to Cancel Build. If you add an additional field to your index or change the auto-recognize or accent sensitive settings, you must perform a full build.
    Note: Canceling the build stops the indexing thread, leaving the index in an unstable state. Relativity deletes these indexes from the population table and gives them an inactive status. You cannot search on an index with an inactive status until you run a full build. Canceling also deletes the index files in the index share.
  • Build Index: Incremental—updates an index after adding or removing documents. During an incremental build the existing index remains available for searching, but changes to the index are not reflected in search results until the incremental build completes. Canceling an incremental build returns the index to its previous state.
    Note: The incremental build process copies each sub-index that requires modification, updates the copy, then replaces existing sub-indexes with the updated copies. When run, the Case manager agent removes duplicate sub-indexes. The system automatically compresses a sub-index during an incremental build only if the sub-index fragmentation level is equal to or greater than the sub-index fragmentation threshold value.
  • Compress Index—compresses the dtSearch index returning all sub-indexes with a fragmentation level greater than zero to a fragmentation level of zero. You can search on the original, uncompressed, dtSearch index while compression is in progress. Once compression completes, the system automatically replaces the old sub-indexes with the defragmented sub-indexes. When run, the Case manager agent removes duplicate sub-indexes.
    • Note: The Compress Index button only runs compression on sub-indexes that have a fragmentation level greater than zero. Canceling compression returns the index to its original fragmented state before compression began.
  • Deactivate Index—deactivates the index and removes it from the search with menu in the Documents tab, but not from the database.
  • Swap Index—swaps your index with a replacement index to use its resources while your index builds or is inactive or disabled for any reason. This enables you to keep searching while your primary index experiences downtime. You can only select indexes in the Replacement Index with an Active status. This index you swap to does not automatically run an incremental update.
    Selecting the index from the drop-down list and clicking OK completes the index swap. You cannot reverse the swap results in the current dialog box. You must close this swap and run it again to swap back or swap another time. This functionality is useful in limited cases, for example, if you are performing a full rebuild on a very large index. Since dtSearch incremental builds are online, you can search documents once indexed.
    Note: The Swap Index function updates anything in the Views table, which affects batches, saved searches, nested searches, and more.
  • Retry Errors—enables only if errors occur, you can use this button to resolve errors.
  • Show Document Errors—enables only if document errors occur. This button creates an exportable list of document-level errors.
  • Show Detailed Status—shows you statistical data for the index, including:
    • Doc Count—the total number of documents in the index.
    • Index Size—the size of the index in bytes.
    • Created Date—the date you created the index.
    • Updated Date—the date you updated the index.
    • Last Build Duration—how long the last build took to complete in hours, minutes, and seconds.
  • Refresh Page—shows the index's current build status.

dtSearch index page

After you create and build a dtSearch index, the dtSearch page has several sections where you can view details about your index.

Index Status

You can view the state of your dtSearch index from the Index Status section. The name of the Index Status section populates with the name of your dtSearch index. When you are building an index, this section changes to a progress bar where you can track your index's progress in real-time. When the index is no longer in progress, this section changes to a static field that displays the below fields.

  • Status—the status of the index. For example, Active - Indexed or Inactive - Indexed.
  • Document Breakdown —the number of indexed documents.

dtSearch Index Information

The dtSearch Index Information section provides general details about the settings applied to your dtSearch index. This section has the following information:

  • Name—the name of your index.
  • Order—the integer value, positive or negative, representing the position of the index in the search indexes list. Indexes sort from lowest, top, to highest, bottom, order number. Those with the same order number sort alphanumerically.
  • Searchable set—the set of documents to be indexed. You can choose from any saved search in the workspace.
  • Email notification recipients—the emails that receive an email notification when your index population fails or completes.
  • Notes—the notes added by the user that created the index.

Searchable Set - Current Indexable Fields

The fields listed may not match those indexed if someone made updates after indexing the searchable set.

Advanced Settings

The Advanced Settings section provides sub-index details about your dtSearch index. This section has the following information:

  • Auto-recognize date, email, and credit card numbers—a yes/no field.
  • Create accent sensitive—a yes/no field. When set to Yes, dtSearch indexes are sensitive to accents and other language-specific characters.
  • Skip malicious files—a yes/no field. Select Yes to skip potentially malicious quarantined files and continue processing the index. The final index breakdown includes the number of skipped documents.
  • Use Default Sub-Index Size—a yes/no field that determines whether to use the default 16 GB size or a custom size. Custom Sub-Index Size field.
  • Index share—the location of the file share for storing the search index. Select the bold (default) location. You will see a warning about system performance if you select a location other than the default.

Noise Words

The list of words that Relativity ignored during indexing.

Alphabet

The index’s alphabet file. See Making a character searchable.

Note: If you search for long, uninterrupted strings that have no spaces or word breaks, such as when you have made a character searchable, dtSearch truncates the string after 32 characters and inserts a wildcard. For more information, see Searching for words longer than 32 characters.

Temporary Index Details

The Temporary Index Details section only appears during an incremental build. This table displays sub-indexes that copy from your original index and are in the process of modification during the incremental build. Once the sub-indexes in this table update, they replace the original sub-indexes from which they were copied. This section has the following information:

  • Population Table—the name of the table that a sub-index is populating.
  • Build Status—the state that the sub-index is currently in.
  • Worker Agent—the name of the agent that is handling the sub-index.
  • Worker Agent Status—the current state of the worker agent.
  • Index File Share—the location where you store your sub-index.
  • Document count—the number of documents assigned to the sub-index.
  • Error(s)—any errors encountered by the sub-index.
  • Fragmentation Level—the fragmentation level of the sub-index. Any index equal to or greater than the sub-index fragmentation threshold appears in red.

Current Index Details

The Current Index Details section displays the sub-indexes that make up your dtSearch index. This section has the following information:

  • Population Table—the name of the table that a sub-index is populating.
  • Build Status—the current state of the sub-index.
  • Worker Agent—the name of the agent that is handling the sub-index.
  • Worker Agent Status—the current state of the worker agent.
  • Index File Share—the location where you store your sub-index.
  • Document count—the number of documents assigned to the sub-index.
  • Error(s)—any errors encountered by the sub-index.
  • Fragmentation Level—the fragmentation level of the sub-index. Any index equal to or greater than the sub-index fragmentation threshold appears in red.
  • Rebuild Selected Sub-Indexes—manually rebuilds selected sub-indexes. Do not use this option unless directed by the Support team.
    Rebuild Selected Sub-Indexes

View Audit

Using the View Audit button, you can see when a user modified dtSearch index settings. The View Audit layout has the following fields:

  • Action
  • Field
  • Old Value
  • New Value
  • User Name
  • Timestamp
  • Details

Temporary storage

If you specify a temporary storage location, dtSearch builds the index in this directory and then copies the index over to the final index share when the build completes. Using a temporary storage location could speed up the build time and reduce network contention