Short message search index (Advanced Access)

Preview the new short message search indexing experience in Relativity. This document provides information on searching message-level and event-level metadata in short messages. While similar to dtSearch, there are differences. For example, you can search specific fields by query in addition to keywords and phrases.

Currently, the only way to search or filter on events is to filter on the Relativity Short Message Format (RSMF) document, or use a dtSearch index. Both methods have limitations.

	Description	Example	Search precision
Elasticsearch	Searches on message-level and event-level metadata.	If you enter a keyword, without writing a field-specific query, Relativity searches all message-level fields, including the message body. Searching for "John" returns results that match that term everywhere Relativity finds it. Such as in the sender_display and message_body fields.	High
dtSearch	Searches cannot search specific aspects.	If you searched "John Smith," the results would show that the phrase "John Smith" appears in the document, but not if he specifically sent a message, reacted, left the chat, and so forth.	Medium
Filter	Searches on document-level fields only.	If you want to find documents that have a message sent by John Smith, the closest you can get is by checking the Participants field. The Participants field tells you if John Smith was in the DM or channel at any point in time.	Low

Elasticsearch index

The Elasticsearch index enables you to create more complex searches on Relativity Short Message Format (RSMF) document than using dtSearch or filtering. It stores and searches on message-level and event-level metadata for enhanced accuracy. By using the metadata, you can search for RSMF messages sent by specific participants, within defined time frames, with specific reactions, as well as edited or deleted messages.

With the Elasticsearch index, you can enter an actual query instead of simple text to search for specific details. For example, suppose you want to search the sender_display field. In this case, you can search specifically on the sender_display field using Elastic Query Syntax (EQS) language.

Once key messages are located, you can apply message-level coding to make it easier to identify what's important in the conversation. This is especially useful during investigations with large amounts of short message data.

Note: Elasticsearch only supports RSMF messages processed in Relativity.

You can still search using regular text, but Relativity uses dtSearch for execution.

For more information on EQS and which fields are searchable, see Resources .

Terms

It's important to understand short message terminology and how it differs from other document terms. Below are common terms and their meaning when working with short message searches.

Term	Meaning
Document, RSMF document	This term refers to all documents in the RSMF document type. An RSMF document may contain one or more messages. It has document-level metadata that provides aggregate data for items within a document. Example metadata includes participants, which is anyone involved in a document. Right now, searching on the document level metadata is the only way to filter RSMFs.
Message, Event RSMF, Message Event	Messages and events are individual actions within an RSMF document. These can include a sent message, a reaction, joining or leaving a chat, and so forth.
Message-level metadata, Event-level metadata	Message-level and event-level metadata is metadata for a specific message. For example, sent messages have sender_display metadata attached to them with the sender's screen name. Timestamp metadata tells you the exact date and time somebody sent the message.

Installing the application

To get started searching on short message metadata, you must first install the Search AI app to your workspace.

Before you install the app, you must:

1. Send over the names of the workspaces you want to use.
   Send these names to your specific Product Manager or Customer Success Manager.
2. Get confirmation that processing has deployed to those workspaces.
   The same Product Manager will contact you.
3. Re-processed any RSMF data.
   For more information on processing data, see Processing.

Once you complete those steps, you can install the application in your workspace.

1. Navigate to the workspace where you want to install the application.
2. Navigate to the Relativity Applications tab.
3. Click New Relativity Application to display an application form.
4. Click the Select from Application Library radio button in the Application Type section.
5. Click the ellipses icon next to the Choose from Application Library field.
   
6. Select Search AI in the Select Library Application list. This list only displays applications added to the Application Library. If Search AI is not included in the list, see Installing applications.
7. Click Ok to display the application in the Choose from Application Library field. The application form also displays the following fields:
   • Schema Version—displays the version of the application that you are installing.
   • User-friendly URL—displays a user-friendly version of the application's URL. This field may be blank.
   • Application Artifacts—displays object types and other application components by clicking the plus icon.
   • Map Fields—there are no fields available in Search AI for mapping.
8. Click Import to install the Search AI app into the workspace.
9. Review the import status of the application. Verify that the installation was successful or resolve errors. See Viewing import status and Troubleshooting application installation errors.

The Search AI app is now in your workspace. You can begin building an index.

Short message search index workflow

The short message search index workflow is similar to other search indexes. You must first build an index before being able to search it.

With Elasticsearch indexes, you can search using the search bar. You can enter a query you write yourself or use condition cards to select fields and filters to refine your search scope. While the Elasticsearch runs at the message and event levels, you will see the results at the document level.

Note: Documents returned may contain one or more hits at the message or event level.

Building the full index

To build the index, perform the following steps:

1. Navigate to the Search Indexes tab.
2. Select the Elasticsearch entry. Relativity automatically generates this index and makes it available.
3. Click the Build Full Index button located in the Build and Manage Index console.
   

The Status field shows Completed when the build finishes successfully. The build process includes all RSMF documents in the workspace, so you do not have to select a saved search or specific indexed fields.

Below are the possible statuses when building a short message index:

• In progress—states that the index is currently building.
• Completed—the build finished successfully.
• Failed—50% or more is considered failed, or canceled. If a build fails, you can find more information in the Retry Errors report. See Retry errors for more information.
• Completed, Eventing active with errors—when there is an error on a message-level coding decision. You must resolve the error. See Retry errors for more information.

After the build is complete, you can also run incremental builds. For more information, see Running incremental index builds.

Running incremental index builds

After running a successful full build, you can run incremental builds. Incremental builds only run on new RSMF documents processed into the workspace and will add them to the index. As a result, incremental builds run faster than a full build.

Consider the following items when running an incremental build:

• Incremental builds are enabled only when the index status is Active. For more information, see Index Status.
• Incremental builds do not index any changes made to existing document data since the last full build.
• Cancel functionality is available during a build. Clicking Cancel will delete the existing index data. This means the next time you want to run a build, you must run a full build. There is a confirmation modal before canceling.

To start an incremental build:

1. Run a previous successful full index build.
2. Click Build Incremental Index located in the Build and Manage Index console.
3. In the Incremental build confirmation window, select the checkbox if you want to activate the index once it's complete. Otherwise, leave it blank.
   
4. Click Yes to begin the incremental build.

The Index Manager Agent will start indexing the job as soon as it becomes available.

After successful completion, the newly indexed files are integrated with the existing search index. If the job is unsuccessful, you can check and retry errors by clicking the Retry Errors button located in the Errors and Status console. See Retry errors.

Automating incremental index builds using Automated Workflows

You can automatically trigger an incremental index build when new documents are added into a workspace using automated workflows. For more information, see Automated Workflows.

To set up the Build Elasticsearch Index Automated Workflow:

1. Navigate to the Automated Workflows tab.
2. Click the New Automated Workflow button.
3. Select New Documents Added as the trigger.
4. Click Add Action and select Build Elasticsearch Index.
5. Select the Elasticsearch index from the drop-down menu.
6. Add a Name and Email for notification on the left panel.

After creating the automated workflow, the building of Elasticsearch indexes is now automated.

Retry errors

The dtSearch Index console includes an Errors and Status section with the following:

• Retry Errors—click the button to retry any documents that failed to index during the previous build.
• Show Document Errors—click the link to see individual files (includes both Control Number and Document Artifact ID) and their errors on the Errored Documents modal. Click the Document Artifact ID link to display the document in the Viewer. Click Export to link to export the errored files to take steps to fix corrupted file and reprocess them. See Retry errors workflow below for more information.
• View Documentation—click the link to navigate to the RelativityOne documentation site.

Retry errors workflow

When retrying errors, the first step to take is to click the Retry Errors button to retry building the index with the errored documents. It is possible there were system issues. If there were, some documents will be fixed without reprocessing the documents.

If there are still errored documents, the next step would be to click the Show document errors link. If the list is manageable, you can go through each document to fix any document issues.

• Indexing Error—a problem occurred while indexing documents. Click Retry Errors to re-index the failed documents.
• Coding Error—a problem occurred when updating a message-level coding field. Click Retry Errors to re-index the failed documents.
• Metadata Storage Errors—a problem occurred when trying to index the metadata. Click Retry Errors to re-index the failed documents.
• Empty File—the file contains no data. The document file did not process correctly or an error occurred during data ingestion through Integration Points. Reprocess the document in Processing or Integration Points.

The final step would be to export the document information and either create a saved search to exclude the documents or to reprocess the documents.

Eventing

When the following three custom coding fields are updated using message-level coding in the Viewer, it sends an event to the application and their data is automatically refreshed in the index.

• custom_responsive—this field reflects any message-level coding decisions made on the Responsive field using the coding layout.
• custom_privileged—this field reflects any message-level coding decisions made on the Privileged field using the coding layout.
• custom_note—this field reflects any message-level coding notes made in the Notes field using the coding layout.

Those events can be searched within a few minutes without having to build an index. For more information on coding fields, see Resources .

If an automatic update fails to index, the index will change to the status, Completed, Eventing active with errors. When eventing is active with errors, the respective documents will display in the Errored Documents modal when you click the Show Document Errors link. Click the Retry Errors button to start fixing the issue. Additionally, a banner will appear on the Documents list page if one or more message-level coding decisions were not indexed. Navigate to the index details page to retry the documents. For more information, see Retry errors.

Search workflow

After you build the index, you can now search it from the Documents tab in two ways:

• Index search bar
• Condition cards

Index search bar

To search using the search bar:

1. Navigate to the Documents tab.
2. Click the drop-down arrow next to the search bar, and select Elasticsearch.
3. Enter a search query or enter a keyword to search across all fields. To view sample queries, see Query examples.
4. Click the search icon or press Enter to start the search.
   

Condition cards

To search using condition cards:

1. Navigate to the Documents tab.
2. Open a saved search or the search conditions panel.
3. Click New Condition.
4. Select (Index Search) as the condition type.
5. From the Index drop-down menu, select Elasticsearch.
6. Enter a search query or enter a keyword to search across all fields. To view sample queries, see Query examples.
7. Click Apply.
   

Viewing results

After the query runs, the document list displays all the documents with at least one message or event matching the query.

You can also see the search results in the Short Message Viewer. Search results are highlighted in the Viewer. A yellow highlight appears around the metadata icon to indicate there is a message with a highlight.

Navigate to each hit by using the Persistent Highlight navigation buttons () on the top toolbar of the Viewer.

Short message search guide

There are several options for searching the short message search index. The first is to enter keywords that automatically search across all metadata fields (including message-level and event-level text), or search specific field names and keywords. This method is the same as dtSearch.

Another option is to search by query. The basic format for a query is: FIELD_NAME: Keyword, where FIELD_NAME is the name of a specific event-level metadata field, for example, timestamp. The tables below lists four of the more common fields names you might use. For a complete list of searchable fields, see All searchable fields.

The colon operator is similar to IS LIKE in that the query looks for any instances where the keyword appears in that field for an event. Keywords should correspond to the field type. For example, by searching the sender_display field, you might enter "John Smith." For searching the timestamp field, you would enter a date-formatted keyword, such as 2024-02-09.

You can also combine searches across different fields using OR and AND operators similar to traditional query languages. To view a list of search syntax, see Elastic Query Syntax.

Common fields

This section displays the four most common fields used for short message searching, along with tips for maximize your search results.

Field name	Real name	Type	Description	Example
Sender Display	sender_display	Text	Display name of the sender. Display names are not standardized so they can be different in different platforms for the same people.	Lilliana Huff - P1
Search tips: If searching for multiple senders, make sure to use parentheses and add AND, OR operators between the different objects. For example, sender_display:(Dilan OR "John Smith" OR John*) Sender display is not a standardized field. It is the user's display name in a given platform. Note: The display may be different across different platforms. Putting quotation marks around the display name gives different results than searching without them. For example, searching sender_display:"John Smith" searches for the exact phrase, while sender_display:(John Smith) searches for John and Smith separately and returns both sets of results.

Field name	Real name	Type	Description	Example
Timestamp	timestamp	Date	Returns the date and time the user sent the message.	2021-01-05T09:15:45
Search tips: The format for date and time is yyyy-mm-ddThh:mm:ss. When searching only a date, use yyyy-mm-dd. When searching dates and times, use four digits for the year, and two digits for the month and day. For example,2024-02-09. Using shortened date formats, such as 24-2-9 will not work. This same applies to the time format. You can use '00' for time as long as you complete the entire sequence: 00:00:00. Use brackets [ ] to include dates and times on either side of the query. Use curly brackets { } to exclude dates and times on either side of the query. You can search for single dates or date ranges. If you want to find messages before a certain date, use timestamp:< yyyy:mm:dd. For messages after a certain date, use timestamp:> yyyy:mm:dd.

Field name	Real name	Type	Description	Example
Message Body	message_body	Text	The actual text of the message.	I hear Sally has his ear. Tell Sally we'll take her to a nice dinner as a token of our appreciation in exchange for some business with Dr. Louis.
Search tips: If you are only searching for text or want to search across all fields, you can enter the keyword directly in the search bar. If you are searching for multiple keywords, make sure to use parentheses between the different objects. For example, message_body: (Cat OR "litter box" OR dog*).

Field name	Real name	Type	Description	Example
Event Type	event_type	Keyword	Multi-choice field. Options include message, disclaimer, join, or leave.	message
Search tips: If you are searching multiple event types, make sure to use parentheses between the different objects. For example, event_type: (join or leave).

Query examples

Use the following examples as a starting point for creating your query strings.

Example one

sender_display:(Abbie OR "Kyson Stanley") AND conversation_type:channel AND timestamp:[2012-10-13 TO 2023-12-12]

In English: Find messages sent by either Abbie or Kyson Stanley from October 13th 2012 and December 12th 2023, within a channel, not direct message.

Example two

sender_display: "Patience Mayo" AND message_body: "if you can cough" AND timestamp:[2012-10-13T09:15:00 TO 2023-10-13T15:15:00]

In English: Find all messages with 'Patience Mayo' in the display name, where the exact phrase "if you can cough" appears in the message body, and sent between 9:15 am on October 13th 2012 and 3:15pm on October 13th 2023.

Example three

is_edit:true OR deleted:true OR event_type:(join OR leave)

In English: Find all messages that were either edited or deleted, or a join or leave event.

Example four

message_body: (fraud OR "don't mention this" OR "take it offline") AND timestamp:<2023-10-01

In English: Find all messages containing the terms "fraud", "don't mention this", or "take it offline", sent before October 1, 2023.

Resources

Use the sections below to view a list of all searchable short message fields and a list of basic EQS operators and functions.