Search Grid for dtSearch

What's changing

In versions 7.2-7.5, Relativity built dtSearch indexes by merging several small indexes into one large index for a workspace. When a user performed a dtSearch in Relativity, it ran against that single index. Beginning in Relativity 8, dtSearch indexes consist of several-to-many sub-indexes. Relativity searches against these indexes simultaneously. This change results in significant performance increases in both searching and building.

On the build side, one of the biggest time consumptions was the merge phase, when each sub-index would be merged into the final, large index. With Search Grid, that merge phase is eliminated completely.

Performance enhancements

Relativity Lab Testing found that building speeds nearly double with the addition of the Search Grid. When running basic and complex searches, they complete significantly faster. Depending on how you scale your hardware, these performance changes could vary. For a full overview of Search Grid performance testing and metrics, please refer to the Performance Metrics section.

Infrastructure changes with Search Grid

Prior to Relativity 8, dtSearch index building was performed on an agent server, while searching took place on a web server. Index building takes place on the same hardware in Relativity 8, but agent servers are now used to host search requests.

Due to architectural changes made to Relativity 8 dtSearch incremental build, your infrastructure must be able to handle duplicate copies of all sub-indexes that require modification or compression when running an incremental build. For incremental builds that are only adding new documents, the system copies and updates only one sub-index. However, if your incremental build is removing documents from previously existing sub-indexes, then the system copies and updates each sub-index from which documents were removed. Duplicate sub-indexes are removed when the Case manager agent runs. Verify that incremental builds are sustainable within your firm's infrastructure. See dtSearch for more information regarding incremental builds, compression and sub-index fragmentation threshold settings.

Schema changes

This section explains the differences between the schema prior to Relativity 8 and the new schema for Search Grid in Relativity 8.

Schema prior to Relativity 8

Prior to Relativity 8, Relativity built dtSearch indexes using two tables, eddsdbo.dtSearchIndex and eddsdbo.dt_POP_##. The former table was used to store a list of all indexes and their settings. The latter was a list of all documents in an index, as well as their current status, which would be updated during a build.

Schema in Relativity 8

Beginning in Relativity 8, the database schema remains largely the same, with the addition of one new table type. For each sub-index that is built, there is now an additional POP table, named after the master index. The name is appended with a number for the sub-index, as shown below:

Search Grid FAQ

All currently-built dtSearch indexes function as expected when you upgrade, but consider the following three considerations for the indexes.

  1. When you upgrade, you must add at least one Search agent to an Agent server in the resource pool in order to execute searches on legacy or new indexes. See Resource pools.
  2. All existing ZSR (cache) tables are dropped to account for the changes in the database schema. This doesn’t change any results, but does mean that any dtSearch queries that were run since the last time an index was built don't have their results cached for faster retrieval when re-running.
  3. All legacy indexes that were built pre-Relativity 8 now require a Full Build for their next update. Incremental Build isn't available.

Assuming all infrastructure is in place, utilizing the Search Grid on any new or existing indexes requires a Full Build. All existing indexes are still functional, but remain as a single, large index until it is rebuilt.

Not necessarily. The Search Grid requires a new Search agent, in addition to the indexing agents that were used previously. If you already have a web server dedicated to the dtSearch WebAPI, you can simply convert that into a new agent server used for searching. If this is not an option, you can use an existing agent server and add a new Search agent.

The Search agents automatically use all available threads on a server, so only one agent should be on each agent server. For example, if you have a four core server, place one search agent on it and it uses all four cores for executing searches.

Additionally, in previous versions, we would sometimes recommend multiple dtSearch Index Manager agents, as those performed the merge during an index build. With Search Grid, the merge phase has been removed entirely, so there should only be a single dtSearch Index Manager in a resource pool.

Most of the values in the Instance setting table have remained the same upon upgrading, but one important change is the addition of thedtSearchDefaultSubIndexSize. This controls the number of documents that are contained in a single sub-index and is set to 250,000 by default. The dtSearch index Manager uses this value to determine how many total sub-indexes to build, based on the number of documents in your index.

Another addition to the Instance setting table is the dtSearchDefaultSubIndexFragmentationThreshold. This value is used as the default value for the Sub-index fragmentation threshold field on the dtSearch Index Information screen. The value must be an integer greater than or equal to 1. When the fragmentation level for a sub-index equals or exceeds the threshold level, the system automatically compresses the sub-index the next time an incremental build is performed.

As stated above, thedtSearchDefaultSubIndexSize is the biggest factor in determining how many sub-indexes are created. In Relativity 7.5, documents would be evenly distributed among all available dtSearch Index Worker agents. For example, if there were 100,000 documents in an index and four dtSearch Index Workers, each agent would work on 25,000 documents.

With Search Grid, the agents build sub-indexes based on the instance setting. So in the same scenario above, with a dtSearchDefaultSubIndex size of 250,000, a single dtSearch Index Worker agent builds one index of 100,000 documents. This leaves the other three agents available for additional jobs.

For more analysis of Search Grid performance metrics, refer to dtSearch in the Performance baselines and recommendations topic.