Short message search index (Advanced Access)

We welcome you to preview the new short message search indexing experience in Relativity. This document provides information on searching message-level and event-level metadata in short messages. While similar to dtSearch, there are differences. For example, you can search specific fields by query in addition to keywords and phrases.

Currently, the only way to search or filter on events is to filter on the Relativity Short Message Format (RSMF) document, or use a dtSearch index. Both methods have drawbacks. For example, when filtering on RSMF documents, you can only filter on document-level fields. If you want to find documents that have a message sent by John Smith, the closest you can get is by checking the Participants field. The Participants field tells you if John Smith was in the DM or channel at any point in time. It's an imprecise search. If you use dtSearch, you can index all of the text in an RSMF document but cannot search on specific aspects. For example, if you searched "John Smith," the results would show that the phrase "John Smith" appears in the document, but not if he specifically sent a message, reacted, left the chat, and so forth.

ElasticSearch index

Because of these limitations, Relativity built a new short message search index. The new index stores and searches on message-level and event-level metadata for greater precision. You can search for messages sent by a particular participant, during a specific time frame, or with specific reactions. You can still search with regular text, however, where Relativity executes the search like dtSearch. For example, if you enter a keyword, without writing a field-specific query, Relativity searches all message-level fields, including the message body. Searching for "John" returns results that match that term everywhere Relativity finds it. Such as in the sender_display and message_body fields.

You can create more complex searches with the new search index. It is more powerful than either keyword or dtSearch. The most notable change is you can now enter an actual query instead of simple text to search for specific details. For example, suppose you want to search the sender_display field. In this case, you can search specifically on the sender_display field using Elastic Query Syntax (EQS) language. The good news is that most of it is intuitive. This document explains the finer points for using EQS and which fields are searchable.

Terms

Understanding short message terminology and how it differs from other document terms is important. Below are some common terms when working with short message searches.

Term	Meaning
Document, RSMF document	This term refers to all documents in the RSMF document type. An RSMF document may contain one or more messages. It has document-level metadata that provides aggregate data for items within a document. Example metadata includes participants, which is anyone involved in a document. Right now, searching on the document level metadata is the only way to filter RSMFs.
Message, Event RSMF message, Event	Messages and events are individual actions within an RSMF document. These can include a sent message, a reaction, joining or leaving a chat, and so forth.
Message-level metadata, Event-level metadata	Message-level and event-level metadata is metadata for a specific message. For example, sent messages have sender_display metadata attached to them with the sender's screen name. Timestamp metadata tells you the exact date and time somebody sent the message.

Short message search index workflow

The short message search index workflow is similar to other search indexes in that you build an index first and then search that index. Searching the index is where you will find differences from dtSearch. With ElasticSearch indexes, you can search using the search bar to enter a query you write yourself or use condition cards to select fields and filters to refine your search scope. While the ElasticSearch runs at the message and event levels, you will see the results at the document level. Documents returned may contain one or more hits at the message or event level.

Installing the Short Message Search application

To get started searching on short message metadata, you’ll need to take the one time step of installing the Search AI app to your workspace.

Before you install the app, you must:

1. Send over the names of the workspaces you want to use.

2. Get confirmation that processing has deployed to those workspaces.

3. Re-processed any RSMF data.

Once you complete those steps, you can install the application in your workspace.

1. Navigate to the workspace where you want to install the application.
2. Navigate to the Relativity Applications tab.
3. Click New Relativity Application to display an application form.
4. Click the Select from Application Library radio button in the Application Type section.
5. Click in the Choose from Application Library field.
6. Select Search AI on the Select Library Application dialog.
   This dialog only displays applications added to the Application Library. If Search AI is not included in the list, see Installing applications.
7. Click Ok to display the application in the Choose from Application Library field. The application form also displays the following fields:
   • Schema Version—displays the version of the application that you are installing.
   • User-friendly URL—displays a user-friendly version of the application's URL. This field may be blank.
   • Application Artifacts—displays object types and other application components.
   • Map Fields—there are no fields available in Search AI for mapping.
8. Click Import to install the Search AI app into the workspace.
9. Review the import status of the application. Verify that the install was successful or resolve errors. See Viewing import status and Troubleshooting application installation errors.

The Search AI app is now in your workspace, and you can now build an index.

Building the index

To build the index, perform the following steps:

1. Navigate to the Search Indexes tab.
2. Select the ElasticSearch entry. Relativity automatically generates this index and makes it available.
3. Click the Build Index button located in the Build and Manage Index console.
   

When the build is complete, the Status field shows Completed. The build process includes all RSMF documents in the workspace, so you do not have to select a saved search or specific indexed fields.

After the build is complete, you can also run incremental builds. For more information, see Incremental builds.

Incremental builds

After running a successful full build, you can run incremental builds. Incremental builds only run on new RSMF documents processed into to the workspace and will add them to the index. This means incremental builds run faster than a full build.

Note: We are working to connect it to Automated Workflows in the coming weeks.

Consider the following items when running an incremental build:

• Incremental builds are enabled only when the index status is Active. For more information, see Index Status.

• Incremental builds do not index message level coding decisions added since the last full build.

• Cancel functionality is available during a build. Clicking Cancel will delete the existing index data. This means the next time you want to run a build, you must run a full build. There is a confirmation modal before canceling.

To start an incremental build,

1. Run a previous successful full index build.

2. Click Build Incremental Index.

3. In the Incremental build confirmation window, click Yes.
   Toggle whether you would like to active this index upon completion, or not.

After clicking Yes, Relativity initiates the index job and a message stating Job submitted; waiting for Index Manager Agent to become available appears.

After successful completion, the newly indexed files are integrated with the existing search index. If the job completes unsuccessfully, you can check and retry errors by clicking the Retry Errors button.

Searching

After you build the index, you can now search it. Navigate to the Documents tab. You can search from the search bar or through condition cards.

Searching with the index search bar

1. Click the drop-down menu just left of the search bar and select ElasticSearch.
2. Enter a search query or enter a keyword to search across all fields. To view sample queries, see Query examples.
3. Click the search icon, or press <ENTER>, to start the search.

Searching through condition cards

1. Open a saved search or the search conditions panel.
2. Click New Condition.
3. Select (Index Search) as the condition type.
4. From the Index drop-down menu, select ElasticSearch.
5. Enter a search query or enter a keyword to search across all fields. To view sample queries, see Query examples..
6. Click Apply.

Viewing results

After the query runs, the document list displays all the documents with at least one message or event matching the query.

Quick search guide

There are several options for searching the short message search index. The first is to enter keywords that automatically search across all metadata fields (including message-level and event-level text), or search specific field names and keywords. This method is the same as dtSearch.

Another option is to search by query. The basic format for a query is: FIELD_NAME: Keyword, where FIELD_NAME is the name of a specific event-level metadata field, for example, timestamp. The table below lists four of the more common fields names you might use. For a complete list of searchable fields, see All searchable fields.

The colon operator is similar to IS LIKE in that the query looks for any instances where the keyword appears in that field for an event. Keywords should correspond to the field type. For example, by searching the sender_display field, you might enter "John Smith." Searching the timestamp field, you would enter a date-formatted keyword, such as 2024-02-09.

You can also combine searches across different fields using OR and AND operators similar to traditional query languages. To view a list of search syntax, see Elastic Query Syntax.

Common fields

This section displays the four most common fields used for short message searching, along with tips for maximize your search results.

Field name	Real name	Type	Description	Example
Sender Display	sender_display	Text	Display name of the sender. Display names are not standardized so they can be different in different platforms for the same people.	Lilliana Huff - P1

Field name	Real name	Type	Description	Example
Timestamp	timestamp	Date	Returns the date and time the user sent the message.	2021-01-05T09:15:45

Field name	Real name	Type	Description	Example
Message Body	message_body	Text	The actual text of the message.	I hear Sally has his ear. Tell Sally we'll take her to a nice dinner as a token of our appreciation in exchange for some business with Dr. Louis.

Field name	Real name	Type	Description	Example
Event Type	event_type	Keyword	Multi-choice field. Options include message, disclaimer, join, or leave.	message

Query examples

Use the following examples as a starting point for creating your query strings.

Example one

sender_display:(Abbie OR "Kyson Stanley") AND conversation_type:channel AND timestamp:[2012-10-13 TO 2023-12-12]

In English: Find messages sent by either Abbie or Kyson Stanley from October 13th 2012 and December 12th 2023, within a channel, not direct message.

Example two

sender_display: "Patience Mayo" AND message_body: "if you can cough" AND timestamp:[2012-10-13T09:15:00 TO 2023-10-13T15:15:00]

In English: Find all messages with 'Patience Mayo' in the display name, where the exact phrase "if you can cough" appears in the message body, and sent between 9:15 am on October 13th 2012 and 3:15pm on October 13th 2023.

Example three

is_edit:true OR deleted:true OR event_type:(join OR leave)

In English: Find all messages that were either edited or deleted, or a join or leave event.

Example four

message_body: (fraud OR "don't mention this" OR "take it offline") AND timestamp:<2023-10-01

In English: Find all messages containing the terms "fraud", "don't mention this", or "take it offline", sent before October 1, 2023.

Resources

Use the sections below to view a list of all searchable short message fields and a list of basic EQS operators and functions.