Short message search index (Advanced Access)

Preview the new short message search indexing experience in Relativity. This document provides information on searching message-level and event-level metadata in short messages. While similar to dtSearch, there are differences. For example, you can search specific fields by query in addition to keywords and phrases.

Currently, the only way to search or filter on events is to filter on the Relativity Short Message Format (RSMF) document, or use a dtSearch index. Both methods have drawbacks.

	Description	Example	Search precision
ElasticSearch	Searches on message-level and event-level metadata.	If you enter a keyword, without writing a field-specific query, Relativity searches all message-level fields, including the message body. Searching for "John" returns results that match that term everywhere Relativity finds it. Such as in the sender_display and message_body fields.	High
dtSearch	Searches cannot search specific aspects.	If you searched "John Smith," the results would show that the phrase "John Smith" appears in the document, but not if he specifically sent a message, reacted, left the chat, and so forth.	Medium
Filter	Searches on document-level fields only.	If you want to find documents that have a message sent by John Smith, the closest you can get is by checking the Participants field. The Participants field tells you if John Smith was in the DM or channel at any point in time.	Low

ElasticSearch index

The ElasticSearch index stores and searches on message-level and event-level metadata for greater precision. You can search for messages sent by a particular participant, during a specific time frame, or with specific reactions.

With the ElasticSearch index, you can enter an actual query instead of simple text to search for specific details. For example, suppose you want to search the sender_display field. In this case, you can search specifically on the sender_display field using Elastic Query Syntax (EQS) language.

You can still search with regular text. However, Relativity executes the search like dtSearch. Overall, you can create more complex searches with the ElasticSearch index.

For more information on the EQS and which fields are searchable, see Resources .

Terms

Understanding short message terminology, and how it differs from other document terms, is important. The following tables includes common terms when working with short message searches.

Term	Meaning
Document, RSMF document	This term refers to all documents in the RSMF document type. An RSMF document may contain one or more messages. It has document-level metadata that provides aggregate data for items within a document. Example metadata includes participants, which is anyone involved in a document. Right now, searching on the document level metadata is the only way to filter RSMFs.
Message, Event RSMF message, Event	Messages and events are individual actions within an RSMF document. These can include a sent message, a reaction, joining or leaving a chat, and so forth.
Message-level metadata, Event-level metadata	Message-level and event-level metadata is metadata for a specific message. For example, sent messages have sender_display metadata attached to them with the sender's screen name. Timestamp metadata tells you the exact date and time somebody sent the message.

Install the application

To get started searching on short message metadata, you’ll need to take the one time step of installing the Search AI app to your workspace.

Before you install the app, you must:

1. Send over the names of the workspaces you want to use.
   Send these names to your specific Product Manager or Customer Success Manager.

2. Get confirmation that processing has deployed to those workspaces.
   The same Product Manager will contact you.

3. Re-processed any RSMF data.
   For more information on processing data, see Processing.

Once you complete those steps, you can install the application in your workspace.

1. Navigate to the workspace where you want to install the application.
2. Navigate to the Relativity Applications tab.
3. Click New Relativity Application to display an application form.
4. Click the Select from Application Library radio button in the Application Type section.
5. Click the ellipses icon in the Choose from Application Library field.
   .
6. Select Search AI on the Select Library Application dialog.
   This dialog only displays applications added to the Application Library. If Search AI is not included in the list, see Installing applications.
7. Click Ok to display the application in the Choose from Application Library field. The application form also displays the following fields:
   • Schema Version—displays the version of the application that you are installing.
   • User-friendly URL—displays a user-friendly version of the application's URL. This field may be blank.
   • Application Artifacts—displays object types and other application components.
   • Map Fields—there are no fields available in Search AI for mapping.
8. Click Import to install the Search AI app into the workspace.
9. Review the import status of the application. Verify that the install was successful or resolve errors. See Viewing import status and Troubleshooting application installation errors.

The Search AI app is now in your workspace. You can now build an index.

Short message search index workflow

The short message search index workflow is similar to other search indexes. You must build an index first and then search that index.

With ElasticSearch indexes, you can search using the search bar. You can enter a query you write yourself or use condition cards to select fields and filters to refine your search scope. While the ElasticSearch runs at the message and event levels, you will see the results at the document level.

Note: Documents returned may contain one or more hits at the message or event level.

Building the index

To build the index, perform the following steps:

1. Navigate to the Search Indexes tab.
2. Select the ElasticSearch entry. Relativity automatically generates this index and makes it available.
3. Click the Build Index button located in the Build and Manage Index console.
   

When the build is complete, the Status field shows Completed. The build process includes all RSMF documents in the workspace, so you do not have to select a saved search or specific indexed fields.

After the build is complete, you can also run incremental builds. For more information, see Incremental builds.

Incremental builds

After running a successful full build, you can run incremental builds. Incremental builds only run on new RSMF documents processed into to the workspace and will add them to the index. This means incremental builds run faster than a full build.

Consider the following items when running an incremental build:

• Incremental builds are enabled only when the index status is Active. For more information, see Index Status.

• Incremental builds do not index message level coding decisions added since the last full build.

• Cancel functionality is available during a build. Clicking Cancel will delete the existing index data. This means the next time you want to run a build, you must run a full build. There is a confirmation modal before canceling.

To start an incremental build,

1. Run a previous successful full index build.

2. Click Build Incremental Index.

3. In the Incremental build confirmation window, click Yes.
   Toggle whether you would like to active this index upon completion, or not.

After clicking Yes, Relativity initiates the index job and a message stating Job submitted; waiting for Index Manager Agent to become available appears.

After successful completion, the newly indexed files are integrated with the existing search index. If the job completes unsuccessfully, you can check and retry errors by clicking the Retry Errors button.

Automated workflow builds

You can now trigger an incremental index build when new documents are added into a workspace using automated workflows. For more information, see Automated Workflows.

To set up the Build Elastsearch Index Automated Workflow:

1. Navigate to the Automated Workflows tab.
2. Click the New Automated Workflow button.
3. Select New Documents Added as the trigger.
4. Click Add Action and select Build Elasticsearch Index.
5. Select the Elasticsearch index from the drop-down menu.
6. Add a Name and Email for notification on the left panel.

After creating the automated workflow, the building of Elasticsearch indexes is now automated.

Retry errors

The dtSearch Index console includes a Errors and Status section with the following:

• Retry Errors—click the Retry Errors button to automatically fix the errors and retry building the index.
• Show document errors—click the Show document errors link to see individual files and their errors to see if you can resolve them. You can also export the errored files. After exporting them, you can take steps to fix corrupted file and reprocess them.
• Show documentation—click the Show documentation link to navigate to the RelativityOne documentation site.

Retry errors workflow

When retrying errors, the first step to take is to click the Retry Errors button. Clicking this button gets Relativity to retry building the index with the errored documents. It is possible there were system issues. If there were, some documents will be fixed without reprocessing the documents.

If there are still errored documents, the next step would be to click the Show document errors link. If the list is manageable, you can go through each document to fix any document issues.

The final step would be to export the Document IDs and either create a saved search to exclude the documents or to reprocess the documents.

Search workflow

After you build the index, you can now search it. Navigate to the Documents tab.

You can search using the following two ways:

• Index search bar

• Condition cards

Index search bar

To search using the search bar:

1. Click the drop-down menu just left of the search bar.
2. Select ElasticSearch.
3. Enter a search query or enter a keyword to search across all fields. To view sample queries, see Query examples.
4. Click the search icon, or press <ENTER>, to start the search.

Condition cards

To search using condition cards:

1. Open a saved search or the search conditions panel.
2. Click New Condition.
3. Select (Index Search) as the condition type.
4. From the Index drop-down menu, select ElasticSearch.
5. Enter a search query or enter a keyword to search across all fields. To view sample queries, see Query examples..
6. Click Apply.

Viewing results

After the query runs, the document list displays all the documents with at least one message or event matching the query.

You can also see the search results in the Short Message Viewer. Search results are highlighted in the Viewer. A yellow highlight appears around the metadata icon to indicate there is a message with a highlight.

Navigate to the Automated Workflows tab.

Click the New Automated Workflow button.

Select New Documents Added as the trigger.

Click Add Action and select Build Elasticsearch Index.

Select the Elasticsearch index from the drop-down menu.

Add a Name and Email for notification on the left panel.

Select New Documents Added as the trigger.

Click Add Action and select Build Elasticsearch Index.

Select the Elasticsearch index from the drop-down menu.

Add a Name and Email for notification on the left panel.

There are several options for searching the short message search index. The first is to enter keywords that automatically search across all metadata fields (including message-level and event-level text), or search specific field names and keywords. This method is the same as dtSearch.

Another option is to search by query. The basic format for a query is: FIELD_NAME: Keyword, where FIELD_NAME is the name of a specific event-level metadata field, for example, timestamp. The table below lists four of the more common fields names you might use. For a complete list of searchable fields, see All searchable fields.

The colon operator is similar to IS LIKE in that the query looks for any instances where the keyword appears in that field for an event. Keywords should correspond to the field type. For example, by searching the sender_display field, you might enter "John Smith." Searching the timestamp field, you would enter a date-formatted keyword, such as 2024-02-09.

You can also combine searches across different fields using OR and AND operators similar to traditional query languages. To view a list of search syntax, see Elastic Query Syntax.

Common fields

This section displays the four most common fields used for short message searching, along with tips for maximize your search results.

Field name	Real name	Type	Description	Example
Sender Display	sender_display	Text	Display name of the sender. Display names are not standardized so they can be different in different platforms for the same people.	Lilliana Huff - P1

Field name	Real name	Type	Description	Example
Timestamp	timestamp	Date	Returns the date and time the user sent the message.	2021-01-05T09:15:45

Field name	Real name	Type	Description	Example
Message Body	message_body	Text	The actual text of the message.	I hear Sally has his ear. Tell Sally we'll take her to a nice dinner as a token of our appreciation in exchange for some business with Dr. Louis.

Field name	Real name	Type	Description	Example
Event Type	event_type	Keyword	Multi-choice field. Options include message, disclaimer, join, or leave.	message

Query examples

Use the following examples as a starting point for creating your query strings.

Example one

sender_display:(Abbie OR "Kyson Stanley") AND conversation_type:channel AND timestamp:[2012-10-13 TO 2023-12-12]

In English: Find messages sent by either Abbie or Kyson Stanley from October 13th 2012 and December 12th 2023, within a channel, not direct message.

Example two

sender_display: "Patience Mayo" AND message_body: "if you can cough" AND timestamp:[2012-10-13T09:15:00 TO 2023-10-13T15:15:00]

In English: Find all messages with 'Patience Mayo' in the display name, where the exact phrase "if you can cough" appears in the message body, and sent between 9:15 am on October 13th 2012 and 3:15pm on October 13th 2023.

Example three

is_edit:true OR deleted:true OR event_type:(join OR leave)

In English: Find all messages that were either edited or deleted, or a join or leave event.

Example four

message_body: (fraud OR "don't mention this" OR "take it offline") AND timestamp:<2023-10-01

In English: Find all messages containing the terms "fraud", "don't mention this", or "take it offline", sent before October 1, 2023.

Resources

Use the sections below to view a list of all searchable short message fields and a list of basic EQS operators and functions.