Search Grid for dtSearch
What's changing
On the build side, one of the biggest time consumptions was the merge phase, when each sub-index would be merged into the final, large index. With Search Grid, that merge phase is eliminated completely.
Performance enhancements
Relativity Lab Testing found that building speeds nearly double with the addition of the Search Grid. When running basic and complex searches, they complete significantly faster. Depending on how you scale your hardware, these performance changes could vary. For a full overview of Search Grid performance testing and metrics, please refer to the Performance Metrics section.
Infrastructure changes with Search Grid
Prior to Relativity 8, dtSearch index building was performed on an agent server, while searching took place on a web server. Index building takes place on the same hardware in Relativity 8, but agent servers are now used to host search requests.
Due to architectural changes made to Relativity 8 dtSearch incremental build, your infrastructure must be able to handle duplicate copies of all sub-indexes that require modification or compression when running an incremental build. For incremental builds that are only adding new documents, the system copies and updates only one sub-index. However, if your incremental build is removing documents from previously existing sub-indexes, then the system copies and updates each sub-index from which documents were removed. Duplicate sub-indexes are removed when the Case manager agent runs. Verify that incremental builds are sustainable within your firm's infrastructure.
Schema changes
This section explains the differences between the schema prior to Relativity 8 and the new schema for Search Grid in Relativity 8.
Schema prior to Relativity 8
Prior to Relativity 8, Relativity built dtSearch indexes using two tables, eddsdbo.dtSearchIndex and eddsdbo.dt_POP_##. The former table was used to store a list of all indexes and their settings. The latter was a list of all documents in an index, as well as their current status, which would be updated during a build.
Schema in Relativity 8
Beginning in Relativity 8, the database schema remains largely the same, with the addition of one new table type. For each sub-index that is built, there is now an additional POP table, named after the master index. The name is appended with a number for the sub-index, as shown below:
Search Grid FAQ
Will Search Grid affect my current indexes?
All currently-built dtSearch indexes function as expected when you upgrade, but consider the following three considerations for the indexes.
- When you upgrade, you must add at least one Search agent to an Agent server in the resource pool in order to execute searches on legacy or new indexes.
- All existing ZSR (cache) tables are dropped to account for the changes in the database schema. This doesn’t change any results, but does mean that any dtSearch queries that were run since the last time an index was built don't have their results cached for faster retrieval when re-running.
- All legacy indexes that were built pre-Relativity 8 now require a Full Build for their next update. Incremental Build isn't available.
How can I take advantage of the Search Grid?
Assuming all infrastructure is in place, utilizing the Search Grid on any new or existing indexes requires a Full Build. All existing indexes are still functional, but remain as a single, large index until it is rebuilt.
Will I need new hardware to use the Search Grid?
Not necessarily. The Search Grid requires a new Search agent, in addition to the indexing agents that were used previously. If you already have a web server dedicated to the dtSearch WebAPI, you can simply convert that into a new agent server used for searching. If this is not an option, you can use an existing agent server and add a new Search agent.
How many Manager and Search agents should there be per server?
The Search agents automatically use all available threads on a server, so only one agent should be on each agent server. For example, if you have a four core server, place one search agent on it and it uses all four cores for executing searches.
Additionally, in previous versions, we would sometimes recommend multiple dtSearch Index Manager agents, as those performed the merge during an index build. With Search Grid, the merge phase has been removed entirely, so there should only be a single dtSearch Index Manager in a resource pool.
What are the most important eddsdbo.InstanceSetting table values?
Most of the values in the Instance setting table have remained the same upon upgrading, but one important change is the addition of the
Another addition to the Instance setting table is the
Is there a way to configure the amount of sub-indexes built?
As stated above, the
With Search Grid, the agents build sub-indexes based on the instance setting. So in the same scenario above, with a dtSearchDefaultSubIndex size of 250,000, a single dtSearch Index Worker agent builds one index of 100,000 documents. This leaves the other three agents available for additional jobs.
What metrics have you seen so far when searching? What about when building?
For more analysis of Search Grid performance metrics, refer to dtSearch in the Performance baselines and recommendations topic.
