Deduplication considerations

The following scenarios depict what happens when you publish a processing set using each of the deduplication methods available on the processing profile.

Note the following special considerations regarding deduplication:

• Deduplication is applied only on Level 1 non-container parent files. If a child file (Level 2+) has the same processing duplicate hash as a parent file or another child file, then they will not be deduplicated, and they will be published to Relativity, regardless of whether the hash field has the same value. This is done to preserve family integrity. You can find out the level value for a file by mapping the Level metadata from the Field Catalog.
• In rare cases, it’s possible for child documents to have different hashes when the same files are processed into different workspaces. For example, if the children are pre-Office 2007 files, then the hashes could differ based on the way the children are stored inside the parent file. In such cases, the child documents aren’t identical (unlike in a ZIP container) inside the parent document. Instead, they’re embedded as part of the document’s structure and have to be created as separate files, and the act of separating the files generates a new file entirely. While the content of the new file and the separated file are the same, the hashes don’t match exactly because the OLE-structured file contains variations in the structure. This is not an issue during deduplication, since deduplication is applied only to the level-1 parent files, across which hashes are consistent throughout the workspace.
• At the time of publish, if two data sources have the same order, or if you don't specify an order, deduplication order is determined by Artifact ID.
• The deduplication method selected for a given Processing Set applies only to the data being published in that set. When a set is published with deduplication enabled, Relativity deduplicates it against all previously processed data in the workspace — regardless of what deduplication settings those prior sets used. See the following examples:
  • If the current set is configured with None, all documents will be published, regardless if past sets had deduplication enabled.
  • If the current set is configured with Custodial or Global, the current set will deduplicate against all applicable data (matching the Custodian or all previously Published sets), even data that was originally published with None.

Global deduplication

When you select Global as the deduplication method on the profile, documents that are duplicates of documents that were already published to the workspace in a previous processing set aren't published again.

Custodial deduplication

When you select Custodial as the deduplication method on the profile, documents that are duplicates of documents owned by the custodian specified on the data source aren't published to the workspace.

No deduplication

When you select None as the deduplication method on the profile, all documents and their duplicates are published to the workspace.

Global deduplication with attachments

When you select Global as the deduplication method on the profile and you publish a processing set that includes documents with attachments, and those attachments are duplicates of each other, all documents and their attachments are published to the workspace.

Global deduplication with document-level errors

When you select Global as the deduplication method on the profile, and you publish a processing set that contains a password-protected document inside a zip file, you receive an error. When you unlock that document and republish the processing set, the document is published to the workspace. If you follow the same steps with a subsequent processing set, the unlocked document is de-duplicated and not published to the workspace.

Technical notes for deduplication

The system uses the algorithms described below to calculate hashes when performing deduplication on both loose files (standalone files not attached to emails) and emails for processing jobs that include either a global or custodial deduplication.

The system calculates hashes in a standard way, specifically by calculating all the bits and bytes that make the content of the file, creating a hash, and comparing that hash to other files in order to identify duplicates.

The following hashes are involved in deduplication:

• MD5/SHA1/SHA256 hashes—provide a checksum of the physical native file.
• Deduplication hashes—the four email component hashes (body, header, recipient, and attachment) processing generates to de-duplicate emails.
• Processing duplicate hash—hash values generated during processing duplication are as follows: for loose files, a SHA256 hash is generated from the physical file's SHA256 hash. For emails, a hash is generated from the email's metadata properties. See Calculating deduplication hashes for emails for more details.

Fields used for comparing files in deduplication

The following fields are used when comparing emails for deduplication:

• With global deduplication, the Processing Duplicate Hash field is compared to the Processing Duplicate Hash fields for processed files, much like loose files.
• With custodial deduplication, the Processing Duplicate Hash field is compared to the Processing Duplicate Hash fields of specific custodians.
• The MD5, SHA256, SHA1, and Email Conversation Index are not considered in deduplication of email files.

Calculating MD5/SHA1/SHA256 hashes

To calculate a file hash for native files, the system:

1. Opens the file.
2. Reads 8k blocks from the file.
3. Passes each block into an MD5/SHA1/SHA256 collator, which uses the corresponding standard algorithm to accumulate the values until the final block of the file is read. Envelope metadata (such as filename, create date, last modified date) is excluded from the hash value.
4. Derives the final checksum and delivers it.

Relativity cannot calculate the MD5 hash value if you have FIPS (Federal Information Processing Standards cryptography) enabled for the instance.

Calculating deduplication hashes for emails

Relativity calculates the MessageBody, Header, and Recipient hash values from the text metadata, which are case-sensitive. A hash generated from Bob.Smith@Email.com will be different from a hash generated from bob.smith@email.com. Attachment hash values are generated from the attached native and are not case sensitive.

MessageBodyHash

To calculate an email’s MessageBodyHash, the system:

1. Captures the PR_BODY tag from the MSG (if it’s present) and converts it into a Unicode string.
2. Gets the native body from the PR_RTF_COMPRESSED tag (if the PR_BODY tag isn’t present) and either converts the HTML or the RTF to a Unicode string.
3. Removes all carriage returns, line feeds, spaces, and tabs from the body of the email to account for formatting variations. An example of this is when Outlook changes the formatting of an email and displays a message stating, “Extra Line breaks in this message were removed.”
   The removal of all the components mentioned above is necessary because if the system didn't do so, one email containing a carriage return and a line feed and another email only containing a line feed would not be deduplicated against each other since the first would have two spaces and the second would have only one space.
4. Constructs a SHA256 hash from the Unicode string derived in step 2 or 3 above.

HeaderHash

To calculate an email’s HeaderHash, the system:

1. Constructs a Unicode string containing Subject<crlf>SenderName<crlf>SenderEMail<crlf>ClientSubmitTime.
2. Derives the SHA256 hash from the header string. The ClientSubmitTime is formatted with the following: m/d/yyyy hh:mm:ss AM/PM. The following is an example of a constructed string:
   • RE: Your last email
   • Robert Simpson
   • robert@relativity.com
   • 10/4/2010 05:42:01 PM

RecipientHash

The system calculates an email’s RecipientHash through the following steps:

1. Constructs a Unicode string by looping through each recipient in the email and inserting each recipient into the string. Note that BCC is included in the Recipients element of the hash.
2. Derives the SHA256 hash from the recipient string RecipientName<crlf>RecipientEMail<crlf>. The following is an example of a constructed recipient string of two recipients:
   • Russell Scarcella
   • rscarcella@relativity.com
   • Kristen Vercellino
   • kvercellino@relativity.com

AttachmentHash

To calculate an email’s AttachmentHash, the system:

1. Derives a SHA256 hash for each attachment.
   • If the attachment is a not an email, the normal standard SHA256 file hash is computed for the attachment.
   • If the attachment is an e-mail, we use the e-mail hashing algorithm described to Calculating deduplication hashes for emails to generate all four de-dupe hashes. Then, these hashes are combined, as described in Calculating the Relativity deduplication hash, to generate a single SHA256 attachment hash.
2. Encodes the hash in a Unicode string as a string of hexadecimal numbers without <crlf> separators.
3. Constructs a SHA256 hash from the bytes of the composed string in Unicode format. The following is an example of constructed string of two attachments:
   • 80D03318867DB05E40E20CE10B7C8F511B1D0B9F336
     ⤷EF2C787CC3D51B9E26BC9974C9D2C0EEC0F515C770B8
     ⤷282C87C1E8F957FAF34654504520A7ADC2E0E23EA

ICS/VCF files are deduplicated not as emails but as loose files based on the SHA256 hash. Since the system now considers these loose files, Relativity is no longer capturing the email-specific metadata that it used to get as a result of ICS/VCF files going through the system's email handler.

Calculating the Relativity deduplication hash

To derive the Relativity deduplication hash, the system:

1. Constructs a string that includes the SHA256 hashes of all four email components described above, as seen in the following example. For more information, see Calculating deduplication hashes for emails.
   • 6283cfb34e4831c97e363a9247f1f01beaaed01d
     ⤷b3a65a47be310c27e3729a3ee05dce5acaec3696
     ⤷c681cd7eb646a221a8fc376478b655c81214dca7
     ⤷419aabee6283cfb34e4831c97e363a9247f1f01b
     ⤷eaaed01db3a65a47be310c27e3729a3ee3843222
     ⤷f1805623930029bad6f32a7604e2a7acc10db9126e3
     ⤷4d7be289cf86e
2. We convert the above string to a UTF-8 byte array.
3. We then take that byte array and generate a SHA256 hash of it.

Deduplication for short message formats

Relativity short message format (RSMF) files produced from Slack, Microsoft Teams, Cellebrite/UFDR, and Google Chat data are deduplicated in two separate layers. The layers run at different stages, use different comparison logic, and operate on different units of data:

• Layer 1 - conversion-time deduplication runs while RSMF files are being written. It operates on individual events (messages) inside a conversation, and its behavior is different for each source format.
• Layer 2 - publish-time deduplication runs during the Publish phase. It operates on whole RSMF files, and its behavior is the same regardless of source format.

Understanding which layer marked an item is the key to interpreting deduplication results. A message that survives Layer 1 can still end up inside a file that Layer 2 marks as a duplicate, and a file that Layer 2 marks as unique may still contain messages that Layer 1 collapsed.

Layer 1 — conversion-time deduplication

During conversion, each source format handles repeated or consecutive events differently before RSMF files are written. Only Cellebrite/UFDR performs true duplicate filtering. Slack merges consecutive events that share a timestamp. Teams and Google Chat apply no duplicate filtering at all, but both fold message edits and deletions into the original event.

Format	Duplicate filtering	Edit and deletion handling	Duplicate counts written
Slack	Consecutive events with the same time stamp are merged	Folded into the prior event	None
Teams	None	Consecutive events with the same message ID are folded into the prior event	None
Cellebrite/UFDR	Yes — matching events are skipped	Not applicable	Detected duplicates field and X-RSMF-DuplicatedEventCount header
Google Chat	None	Message-version entries are folded into the original event	None

Slack

The converter compares each event's timestamp against the previous written event in the same conversation. If a consecutive event carries the same timestamp, it is merged into the prior event — including any edits or deletions — rather than written as a new RSMF event.

The comparison looks only at the immediately preceding event, and the baseline resets at the start of each conversation. No duplicate count is retained, and no duplicate-count field or header is written.

Slack's underlying event ID is a composite of user and timestamp. It is not the value used for this comparison.

Teams

No duplicate filtering is applied. No events are discarded, and unsupported or unrecognized event types are still emitted as placeholder events.

However, consecutive messages that share the same message ID are folded into the previous event as edit history, a deleted flag, or both, which reconstructs the edit and deletion history of the message. Because those versions are folded in, they do not appear as separate RSMF events. This is message-version handling rather than duplicate filtering.

Cellebrite/UFDR

The converter compares four values against the single previous written event:

• timestamp
• sender ID
• message body
• attachments, compared as an order-independent set

If all four match, the event is skipped instead of written.

Cellebrite/UFDR is the only format that reports duplicate counts, and it produces two distinct values:

• Detected duplicates—a per-event custom field recording the length of the current consecutive run, written on the primary event.
• X-RSMF-DuplicatedEventCount—a per-conversation total written to the RSMF header. This value is surfaced downstream as the field Rsmf/DuplicatedMessageCount.

Google Chat

No conversion-time deduplication is applied. Every original message is written to RSMF. The converter retains no duplicate count, emits no Detected duplicates field, and writes no X-RSMF-DuplicatedEventCount header.

Message edits and deletions arrive as separate message-version entries and are folded into the original event, keyed on version type and matching message ID. An edit adds edit history and updates the message body; a deletion sets a deleted flag. As with Teams, this is message-version handling rather than duplicate filtering.

Layer 2 — publish-time file deduplication

During the Publish phase, each file is assigned a deduplication identity built from a fixed, ordered set of SHA-256 hash slots: file, header, recipient, body, and attachment. Slots that are empty or disabled are zero-filled. The slots are concatenated in order — the identity is a conditional concatenation, not an arithmetic sum of the hashes.

For an RSMF file, which is treated as a chat or message item, the file-bytes slot is always empty. The effective identity is therefore content-only:

RSMF deduplication identity = header_hash + recipient_hash + body_hash + attachment_hash

The file-bytes slot still exists in the underlying structure, but it is populated only for physical files that are neither email nor chat items.

Each component hash is calculated as follows:

Component	Calculated from
Header	Subject, From name and address, and date normalized to UTC
Recipient	To names and email addresses
Body	Normalized body text, with whitespace stripped
Attachment	The byte hashes of the embedded attachments

Deduplication statuses

The RSMF file, not the individual messages inside it, is the unit of deduplication. One hash record is created per file.

Within the set of jobs in scope, files are assigned one of the following statuses:

Status	Meaning
Primary	One file per group of files sharing an identity
Duplicate	The remaining files in that group
Unique	A file whose identity matches no other file in scope
None	Container files

Where multiple files share an identity, a single primary is selected based on ranked path.

Deduplication works the same way no matter where the data came from. Messages from Slack, Teams, Cellebrite/UFDR, and Google Chat are all treated identically. The only distinction it makes is between message items and non-message items.

Key limitations

Deduplication does not operate on individual messages at publish time

Because Layer 2 compares whole files, two RSMF files that overlap in content but are not byte-for-byte equivalent in the hashed components will both be retained. Duplicate messages spread across non-identical files are not collapsed at publish time.

Time-based slicing determines whether overlapping exports deduplicate

RSMF files are produced by a configurable time-window slicer. The default Slice by value is 24 hours, and Google Chat uses the same slicer. A conversation is also force-sliced when it reaches either 10,000 events or 2 GB of attachments.

Slices are anchored to a fixed calendar grid (midnight, hour multiples, and Monday for weekly slices) rather than to the start date of each export. Daylight saving time is not taken into account. See RSMF slicing configuration reference for details on slice intervals.

Because slices anchor to a grid, two exports that use the same Slice by setting and both fully cover a given interior window produce the same partition of events for that window. Identical content yields an identical hash, so those interior slices do deduplicate against each other.

Non-matches are confined to:

• boundary slices at the edges of each export's date range, where coverage of the window is partial
• date ranges covered by only one of the two exports
• cases where the two exports use different Slice by settings

In practice, cross-export deduplication degrades at the boundaries and when Slice by settings differ. It does not fail wholesale.