Short message search index (Advanced Access)

Preview the new short message search indexing experience in Relativity. This document provides information on searching message-level and event-level metadata in short messages. While similar to dtSearch, there are differences. For example, you can search specific fields by query in addition to keywords and phrases.

Currently, the only way to search or filter on events is to filter on the Relativity Short Message Format (RSMF) document, or use a dtSearch index. Both methods have limitations.

	Description	Example	Search precision
Elasticsearch	Searches on message-level and event-level metadata.	If you enter a keyword, without writing a field-specific query, Relativity searches all message-level fields, including the message body. Searching for "John" returns results that match that term everywhere Relativity finds it. Such as in the sender_display and message_body fields.	High
dtSearch	Searches cannot search specific aspects.	If you searched "John Smith," the results would show that the phrase "John Smith" appears in the document, but not if he specifically sent a message, reacted, left the chat, and so forth.	Medium
Filter	Searches on document-level fields only.	If you want to find documents that have a message sent by John Smith, the closest you can get is by checking the Participants field. The Participants field tells you if John Smith was in the DM or channel at any point in time.	Low

Elasticsearch index

The Elasticsearch index stores and searches on message-level and event-level metadata for greater precision. You can search for messages sent by a particular participant, during a specific time frame, or with specific reactions.

With the Elasticsearch index, you can enter an actual query instead of simple text to search for specific details. For example, suppose you want to search the sender_display field. In this case, you can search specifically on the sender_display field using Elastic Query Syntax (EQS) language.

You can still search with regular text. However, Relativity executes the search like dtSearch. Overall, you can create more complex searches with the Elasticsearch index.

For more information on the EQS and which fields are searchable, see Resources .

Terms

It's important to understand short message terminology and how it differs from other document terms. Below are common terms and their meaning when working with short message searches.

Term	Meaning
Document, RSMF document	This term refers to all documents in the RSMF document type. An RSMF document may contain one or more messages. It has document-level metadata that provides aggregate data for items within a document. Example metadata includes participants, which is anyone involved in a document. Right now, searching on the document level metadata is the only way to filter RSMFs.
Message, Event RSMF message, Event	Messages and events are individual actions within an RSMF document. These can include a sent message, a reaction, joining or leaving a chat, and so forth.
Message-level metadata, Event-level metadata	Message-level and event-level metadata is metadata for a specific message. For example, sent messages have sender_display metadata attached to them with the sender's screen name. Timestamp metadata tells you the exact date and time somebody sent the message.

Install the application

To get started searching on short message metadata, you must first install you’ll need to take the one time step of installing the Search AI app to your workspace.

Before you install the app, you must:

1. Send over the names of the workspaces you want to use.
   Send these names to your specific Product Manager or Customer Success Manager.
2. Get confirmation that processing has deployed to those workspaces.
   The same Product Manager will contact you.
3. Re-processed any RSMF data.
   For more information on processing data, see Processing.

Once you complete those steps, you can install the application in your workspace.

1. Navigate to the workspace where you want to install the application.
2. Navigate to the Relativity Applications tab.
3. Click New Relativity Application to display an application form.
4. Click the Select from Application Library radio button in the Application Type section.
5. Click the ellipses icon in the Choose from Application Library field.
   
6. Select Search AI on the Select Library Application dialog. This dialog only displays applications added to the Application Library. If Search AI is not included in the list, see Installing applications.
7. Click Ok to display the application in the Choose from Application Library field. The application form also displays the following fields:
   • Schema Version—displays the version of the application that you are installing.
   • User-friendly URL—displays a user-friendly version of the application's URL. This field may be blank.
   • Application Artifacts—displays object types and other application components by clicking the plus icon.
   • Map Fields—there are no fields available in Search AI for mapping.
8. Click Import to install the Search AI app into the workspace.
9. Review the import status of the application. Verify that the installation was successful or resolve errors. See Viewing import status and Troubleshooting application installation errors.

The Search AI app is now in your workspace. You can begin building an index.

Short message search index workflow

The short message search index workflow is similar to other search indexes. You must first build an index before being able to search it.

With Elasticsearch indexes, you can search using the search bar. You can enter a query you write yourself or use condition cards to select fields and filters to refine your search scope. While the Elasticsearch runs at the message and event levels, you will see the results at the document level.

Note: Documents returned may contain one or more hits at the message or event level.

Building the full index

To build the index, perform the following steps:

1. Navigate to the Search Indexes tab.
2. Select the Elasticsearch entry. Relativity automatically generates this index and makes it available.
3. Click the Build Full Index button located in the Build and Manage Index console.
   

When the build is complete, the Status field shows Completed. The build process includes all RSMF documents in the workspace, so you do not have to select a saved search or specific indexed fields.

Below are the possible statuses when building a short message index:

• In progress—states that the index is currently building.
• Completed—the build finished successfully.
• Failed—50% or more is considered failed, or canceled. When a build fails, you can find more information in the Retry Errors report. For more information, see Retry errors.
• Completed, Eventing active with errors—when there is an error on a message level coding decision. You must resolve the error. For more information, see Retry errors.

After the build is complete, you can also run incremental builds. For more information, see Running incremental index builds.

Running incremental index builds

After running a successful full build, you can run incremental builds. Incremental builds only run on new RSMF documents processed into the workspace and will add them to the index. As a result, incremental builds run faster than a full build.

Consider the following items when running an incremental build:

• Incremental builds are enabled only when the index status is Active. For more information, see Index Status.

• Incremental builds do not index any changes made to existing document data since the last full build.

• Cancel functionality is available during a build. Clicking Cancel will delete the existing index data. This means the next time you want to run a build, you must run a full build. There is a confirmation modal before canceling.

To start an incremental build:

1. Run a previous successful full index build.
2. Click Build Incremental Index located in the Build and Manage Index console.
3. In the Incremental build confirmation window, select the checkbox if you want to activate the index once it's complete. Otherwise, leave it blank.

4. Click Yes to begin the incremental build.

The job is submitted for indexing and will begin once an Index Manager Agent becomes available.

After successful completion, the newly indexed files are integrated with the existing search index. If the job is unsuccessful, you can check and retry errors by clicking the Retry Errors button located in the Errors and Status console. For more information, see Retry errors.

Automating incremental index builds using Automated Workflows

You can automatically trigger an incremental index build when new documents are added into a workspace using automated workflows. For more information, see Automated Workflows.

To set up the Build Elasticsearch Index Automated Workflow:

1. Navigate to the Automated Workflows tab.
2. Click the New Automated Workflow button.
3. Select New Documents Added as the trigger.
4. Click Add Action and select Build Elasticsearch Index.
5. Select the Elasticsearch index from the drop-down menu.
6. Add a Name and Email for notification on the left panel.

After creating the automated workflow, the building of Elasticsearch indexes is now automated.

Retry errors

The dtSearch Index console includes an Errors and Status section with the following:

• Retry Errors—click the button to automatically fix the errors and retry building the index.
• Show Document Errors—click the link to see individual files (includes both Control Number and Document Artifact ID) and their errors to see if you can resolve them. If needed, use the Document Artifact ID link to display the document in the Viewer and the Export link to export the errored files. After exporting them, you can take steps to fix corrupted file and reprocess them.
• View Documentation—click the link to navigate to the RelativityOne documentation site.

Retry errors workflow

When retrying errors, the first step to take is to click the Retry Errors button to retry building the index with the errored documents. It is possible there were system issues. If there were, some documents will be fixed without reprocessing the documents.

If there are still errored documents, the next step would be to click the Show document errors link. If the list is manageable, you can go through each document to fix any document issues.

The final step would be to export the document information and either create a saved search to exclude the documents or to reprocess the documents.

Eventing

These are the three fields that, when data is updated, will have these automatic updates occur.

The three custom coding fields:

• custom_responsive—this field reflects any message level coding decisions made on the Responsive field using the coding layout.
• custom_privileged—this field reflects any message level coding decisions made on the Privileged field using the coding layout.
• custom_note—this field reflects any message level coding notes made in the Notes field using the coding layout.

When these fields change, it sends an event. Those events can be searched in near real time without having to build an index. For more information on coding fields, see Resources .

If an automatic update fails to index, the index will change to a new status, Completed, Eventing active with errors. When eventing is active with errors, the respective documents will display in the Errored Documents modal when you click the Show Document Errors link. Click the Retry Errors button to start fixing the issue. For more information, see Retry errors.

Search workflow

After you build the index, you can now search it from the Documents tab in two ways:

• Index search bar
• Condition cards

Index search bar

To search using the search bar:

1. Navigate to the Documents tab.
2. Click the drop-down arrow next to the search bar, and select Elasticsearch.
3. Enter a search query or enter a keyword to search across all fields. To view sample queries, see Query examples.
4. Click the search icon or press Enter to start the search.

Condition cards

To search using condition cards:

1. Navigate to the Documents tab.
2. Open a saved search or the search conditions panel.
3. Click New Condition.
4. Select (Index Search) as the condition type.
5. From the Index drop-down menu, select Elasticsearch.
6. Enter a search query or enter a keyword to search across all fields. To view sample queries, see Query examples.
7. Click Apply.

Viewing results

After the query runs, the document list displays all the documents with at least one message or event matching the query.

You can also see the search results in the Short Message Viewer. Search results are highlighted in the Viewer. A yellow highlight appears around the metadata icon to indicate there is a message with a highlight.

Navigate to each hit by using the Persistent Highlight navigation buttons () on the top toolbar of the Viewer.

Short message search guide

There are several options for searching the short message search index. The first is to enter keywords that automatically search across all metadata fields (including message-level and event-level text), or search specific field names and keywords. This method is the same as dtSearch.

Another option is to search by query. The basic format for a query is: FIELD_NAME: Keyword, where FIELD_NAME is the name of a specific event-level metadata field, for example, timestamp. The tables below lists four of the more common fields names you might use. For a complete list of searchable fields, see All searchable fields.

The colon operator is similar to IS LIKE in that the query looks for any instances where the keyword appears in that field for an event. Keywords should correspond to the field type. For example, by searching the sender_display field, you might enter "John Smith." For searching the timestamp field, you would enter a date-formatted keyword, such as 2024-02-09.

You can also combine searches across different fields using OR and AND operators similar to traditional query languages. To view a list of search syntax, see Elastic Query Syntax.

Common fields

This section displays the four most common fields used for short message searching, along with tips for maximize your search results.

Field name	Real name	Type	Description	Example
Sender Display	sender_display	Text	Display name of the sender. Display names are not standardized so they can be different in different platforms for the same people.	Lilliana Huff - P1

Field name	Real name	Type	Description	Example
Timestamp	timestamp	Date	Returns the date and time the user sent the message.	2021-01-05T09:15:45

Field name	Real name	Type	Description	Example
Message Body	message_body	Text	The actual text of the message.	I hear Sally has his ear. Tell Sally we'll take her to a nice dinner as a token of our appreciation in exchange for some business with Dr. Louis.

Field name	Real name	Type	Description	Example
Event Type	event_type	Keyword	Multi-choice field. Options include message, disclaimer, join, or leave.	message

Query examples

Use the following examples as a starting point for creating your query strings.

Example one

sender_display:(Abbie OR "Kyson Stanley") AND conversation_type:channel AND timestamp:[2012-10-13 TO 2023-12-12]

In English: Find messages sent by either Abbie or Kyson Stanley from October 13th 2012 and December 12th 2023, within a channel, not direct message.

Example two

sender_display: "Patience Mayo" AND message_body: "if you can cough" AND timestamp:[2012-10-13T09:15:00 TO 2023-10-13T15:15:00]

In English: Find all messages with 'Patience Mayo' in the display name, where the exact phrase "if you can cough" appears in the message body, and sent between 9:15 am on October 13th 2012 and 3:15pm on October 13th 2023.

Example three

is_edit:true OR deleted:true OR event_type:(join OR leave)

In English: Find all messages that were either edited or deleted, or a join or leave event.

Example four

message_body: (fraud OR "don't mention this" OR "take it offline") AND timestamp:<2023-10-01

In English: Find all messages containing the terms "fraud", "don't mention this", or "take it offline", sent before October 1, 2023.

Resources

Use the sections below to view a list of all searchable short message fields and a list of basic EQS operators and functions.