Combining email threading and textual near duplicate identification

When creating a structured analytics set (SAS), you can select multiple operations to run together. Two of the operations, email threading and textual near duplicate identification (TND), have similar functions when applied to emails. Depending on your situation, you may want to combine them or run them separately.

See these related pages:

How email threading and TND interact

When email threading and textual near duplication identification (TND) operations run as part of the same structured analytics set, they each process only some of the documents. Email threading only processes email documents and their attachments, and TND only processes standalone documents and email attachments. This is because email threading already tracks duplicate emails, and the threaded results are more comprehensive than TND would be on the same emails.

If you run the operations independently, on separate sets, TND will run across all documents, including emails. Email threading will still only work on emails and their attachments, regardless of what other operations are included.

Pros and cons of running email threading and TND together

For most situations, running email threading and TND together saves time and works well. Email threading not only checks emails against each other for duplicate content, but it also tracks email threads, identifies inclusive emails, and can be used for email thread visualization. This is often more useful for reviewing emails than TND. Running them together also saves document ingestion and comparison time, which can significantly speed up the process.

However, there may be situations where you also want the TND fields populated for emails, not just the email threading fields. For example:

If you need to identify emails as near duplicates of non-email documents.
If your team uses the TND fields to batch out documents to reviewers.

In those cases, we recommend running TND and email threading on two separate structured analytics sets.

Comparing separate versus combined runs

If you run email threading and TND separately:

TND runs across all documents, including emails.
The overall analysis time is generally slower.
All documents receive a value in the Textual Near Duplicate Group, Textual Near Duplicate Similarity, and Textual Near Duplicate Principal fields.

If you run email threading and TND in a single structured analytics set:

TND only runs on standalone documents and email attachments. It does not run on emails.
The overall analysis time is generally faster.
The TND fields behave as follows:
- Textual Near Duplicate Group—remains empty.
- Textual Near Duplicate Similarity—set to zero.
- Textual Near Duplicate Principal—set to False.

You can also combine both approaches. For example, you might choose to set up your primary review workflow using a SAS that combines email threading and TND, but create a separate SAS later that runs only TND. In that case, set up the second SAS to write the Textual Near Duplicate Group to a different relational field than the first one uses. This prevents overwriting previous TND data on any non-emails in the set.