Promoting data between workspaces through Integration Points

Note: The title of this topic used to be "Sync with Integration Points."

You can use Integration Points to promote (move) data from one Relativity workspace to another Relativity workspace. This can be thought of as a one-way sync process from a source workspace to a destination workspace and not a back and forth syncing process between them.

Once you have tagged documents for inclusion in or exclusion from the data you want to promote to review, you can access Integration Points to start the job or jobs that will send those documents to the review workspace.

This topic provides details on Integration Points as it is used to promote data between workspaces. For information, see Integration Points

This page contains the following information:

Special considerations for large data promotion workflows

For large data promotion workflows, we recommend configuring integration point jobs into batches. The following suggestions can be used when creating job batches:

When using Tag Documents with Job Name option, the job document count should be no more than 500,000.
When Tag Documents with Job Name is not selected, no limit on documents count applies.
The recommended number of fields to be mapped should not exceed 100. Additionally, it is best to map as few long text fields as possible.

Promoting (exporting) to a Relativity workspace

In addition to exporting to a CSV and load file, you have the option of promoting (exporting) data from a source workspace to a destination workspace in Relativity. You can do this by selecting Relativity as the destination for the data that you intend to transfer and then selecting a specific workspace where you want that data to go. This is the most efficient way to move documents from one workspace to another. Since it's a one-step process, it does not require you to export and re-import your data.

Notes:

You can transfer document objects and non-document object metadata to a destination workspace. The non-document object must already exist in the destination workspace for non-document object metadata to be promoted.
You can move original images and natives or metadata. You are not required to have Integration Points installed on the destination workspace.
The data promotion process supports transfer of non-document objects linked to the Relativity Applications installed in the destination workspace. For example, if you want to push entities object to another workspace, make sure that the proper application is installed there, for example Legal Hold. You need to have the Relativity Application permission in both the Workspace Security and Object Security sections set to View.
The export process sorts documents and images by name, which is typically the Control Number. This process takes into account entire names, which means that for names with numbers, it is advised to preserve the number of digits in the names by filling with leading zeros. For example:
- Test2_1, Test2_2, Test2_10 will be sorted as: Test2_1, Test2_10, Test_2
- Test2_01, Test2_10, Test2_02 will be sorted as : Test2_01, Test2_02, Test2_10
If promoted fields have associative objects, these objects need to be transferred first to ensure successful promotion of fields having references to them.
The data promotion process will not transfer custom objects created by users in a workspace.
The data promotion process cannot transfer complex objects that have relationships to other objects, such as Search Terms Report (STR), structured analytic sets, and Integration Points profiles, because related objects' ArtifactIDs may be different in the source and destination workspaces.
When the Copy Images radio button is selected, the field mapping section is disabled because only the control number is required and available in this scenario. If you want to transfer other field metadata, you must create a new integration point without choosing to copy images.
Integration Points uses the following delimiters to configure the Import API for the destination workspace. Other delimiters typically configured with the Import API, for example via the RDC, are not utilized:
- Multi-value delimiter: ASCII 029
- Nested-value delimiter: ASCII 030

Setup

To create an integration point specifically for exporting a workspace, follow the steps below.

Navigate to the Integration Points tab.
Click New Integration Point.
Complete the following fields in the Setup category of the Create Integration Point layout:

Name—the name of your integration for reference purposes.
Type—select Export to designate this as an export job. Selecting this sets the Source field to be Relativity.
Source—this is automatically set to Relativity since you are exporting data out of a Relativity workspace.
Destination—select Relativity. This is because you do not want to export directly to another Relativity workspace, but instead want to leverage the RDC export functionality, which is available on a subsequent layout only if you select Load File here.
Transferred Object—select the available Document or non-document object meta data you want to promote to the destination workspace. The non-document object must already exist in the destination workspace. The data promotion process will not transfer custom objects created by users in a workspace. Refer to the Notes above for additional information.
Profile—use this to complete the remaining Integration Points settings based on the settings of a saved profile. This includes all of the fields in the Connect to Source layout, as well as field mappings. If no profiles exist in the workspace, you do not have the option of selecting them. To apply a profile that you have already created, select it from the drop-down list. For more information on profiles, see Integration Points profiles.
Email Notification Recipients—enter the email addresses of those who should receive notifications of whether the integration point export succeeded or failed. Use semi colons to separate email addresses.
Log Errors—select Yes or No to denote whether Relativity tracks item level errors.
- If you select Yes, each job also logs any item level errors.
- If you select No, Relativity does not log these item level errors.
- Regardless of your selection, job-level errors are always recorded in Relativity.
Enable Scheduler—gives you the option of scheduling additional exports. Selecting Yes makes the following fields available:
- Frequency—the interval at which Relativity runs this integration point.
  - Daily—select this option to promote data once every day.
  - Weekly—select this option to promote data on a weekly basis. You can specify how often in the Reoccur field, in which you will provide a number value in the Every # week(s) choice. You can then specify on which day of the week the data promotion will take place by checking any of the days of the week listed.
  - Monthly—select the day of the month that you want this integration point to promote data once every month.
  - Reoccur—enter the number of month(s) in which this integration point repeatedly promote data.
  - Send On:
    - Day __ the month—select the day of the month that you want this integration point to promote data.
    - The __ of the Month—select this option for this integration point to promote data on the chosen day of every month. For example, "The Second Friday of the month

Start Date—the date that you want Integration Points to start promoting the data.
End Date—(Optional) the date that you want Integration Points to stop promoting the data. Leaving the End Date blank causes the Integration Point to run indefinitely at the scheduled interval.
Scheduled Time—the time at which this integration point promote data. This time is local to your PC, not to the server.

Click Next to advance to the Source Information layout.

Connect to Source

Continue to create your import integration point by connecting Relativity to the data source by following the steps below.

In the Connect to Source layout, complete the following fields:

Source—select Document as the transferred object during setup, then select Saved Search or Production from the drop-down list. If you select any other object as the transferred object during setup, this is automatically set to View.
Saved Search/Production/View—depending on the selected Transferred Object, one of the following fields displays:
- Saved Search—elect an existing saved search to use from the drop-down list or click the ellipsis button for an alternative way to select a saved search.
- Production Set—select an existing production set to use from the drop-down list.
- View—select an existing View to use from the drop-down list.
Destination Workspace—select an existing workspace to export your saved search or production to.
Note: Relativity will validate whether the selected object is available in the destination workspace and transfer only objects linked to Relativity Applications installed in the destination workspace. If the selected object is not available in the destination workspace, an error message will display.
Location—select a folder or production set as the destination location.
- Folder—select the drop-down menu to expand the folder structure, locate the folder from which you want to export the data, and select it. This field is only available if you have selected Folder.
- Production Set—select a production set to export the data to. Click the plus sign to create a new production set.
  Note: For successful operation, promote Production Set with Append/Overlay mode.
Create Saved Search—select Yes to create a saved search in the destination workspace. This saved search's name will take the name given to the integration point. This choice is enabled only if Yes is selected for the Tag Documents with Job Name field.
Tag Documents with Job Name—select Yes if you want to tag all documents in the scope of the job with a unique ID, which allows you to easily identify these documents in the target workspace. Otherwise, select No. The default value is Yes. Refer to Working with promoted documents for steps on how to view the promoted documents when needed.

Click Next to advance to the Destination Information layout.

Map Fields

Map the attributes or fields so that Integration Points imports the targeted data into specific Relativity fields by following the steps below.

In the Fields Mapping wizard, you have the following options for mapping fields:
- Use the Shift+click and Ctrl+click method to select multiple fields at a time, similar to field mapping in the Relativity Desktop Client.
- Use the text search boxes above the Source and Destination lists to find a particular field.
- Use the single and double arrows or double-click a field to move selected fields between columns.
- Use the horizontal scroll bar in each column as needed to fully view long field names.
- Each column displays the total number of fields available in that column. This allows you to quickly compare the number of mapped fields to ensure they match. If they do not match, an error will display when you attempt to save the Integration Point.
- The field names in the Destination column include the type of each field listed.
- Click the Map All Fields button between the Source and Destination columns to automatically map all fields with matching names, except single/multiple object fields. If you have Destination fields that are mapped to Fields in the Field Catalog, Relativity tries to find name matches between these Catalog Fields, as well.
  - If the WebAPIPath instance setting in the kCur.IntegrationPoints section is not configured correctly after upgrade or installation, the Source field list will not be able to display any attributes and will be empty. As a result, you will not be able to map fields.
  - You do not need to map all attributes or fields. Only the Unique Identifier and Object Identifier are required. The Unique Identifier should contain a value that no other item in the workspace contains. For example, use the GUID or distinguishedName attribute. The Object Identifier is the specific field on the object that holds the displayed identifier, which might not be unique. For example, the Full Name field is the Object Identifier of the Entity RDO but it might not be unique. These two identifier values can be the same.
- Click the Map Saved Search button between the Source and Destination columns to automatically map fields which are configured in the Saved Search that is selected as a data source on the previous screen. If you have Destination fields that are mapped to Fields in the Field Catalog, Relativity tries to find name matches between these Catalog Fields, as well.
Complete the following Import Settings on the Map Fields layout:
- Overwrite—determines how the system overwrites records once you promote data to the review workspace.
  - Append Only—promote only new records into the review workspace.
  - Overlay Only—update existing records only in the review workspace. Any documents with the same workspace identifier are overlaid. This field acts as a link indicating to Relativity where to import the data. When you select this option, you must complete the Multi-Select Field Overlay Behavior field described below.
  - Append/Overlay—adds new records to the review workspace and overlays data on existing records. When you select this option, you must complete the Multi-Select Field Overlay Behavior field described below. You are not able to create folders or re-folder documents when you select Append/Overlay.
- Copy Images—choose to copy, or not copy, images to the destination workspace. Selecting Yes will copy images to your destination workspace. When yes is selected, an Image Precedence field needs to be completed. Selecting No will not copy images to your destination workspace. When no is selected, a Copy Native Files needs to be completed.
  - Image Precedence—select Original Images or Produced Images to copy to the destination workspace.
  - Copy Native Files—select Physical Files, Links Only, or No to determine how to copy native files to the destination workspaces.
    - Physical files—select to copy any native files from the source workspace to the destination workspace.
    - Links Only—select to add only links to the destination workspace that direct back to the source workspace documents.
      Note: When choosing this option to promote documents using only links from repository to review workspace, keep in mind that if you delete the native files in the repository workspace or if you delete or archive the repository workspace itself, the links to the promoted documents will be broken in the review workspace. In this situation, if you wanted to maintain the files in the review workspace, you first need to promote the physical files to that workspace before deleting them from the repository workspace or before deleting or archiving the repository workspace. For details on promoting subsets of data, see Repository workspace special considerations.
      Note: Be aware that when a non-administrator user copies native files by selecting the Links Only option, the System Created by column on the Job History tab will list the system administrator’s name instead of the non-administrator user's name.
    - No—select to not create additional copies of your native files. No native files will be pushed to the destination workspace. Only metadata, such as extracted text or coding decisions, will be pushed. For example, if you have a document with native files and coding decisions, and you select the No option, only metadata (the coding decisions) will be promoted to the destination workspace. Refer to additional notes below regarding this choice.
      - Doing this maintains a single copy of that data file no matter how many times you use the document in your workspace.
      - You have the option of selecting No for an initial run of an integration point and then Yes for a subsequent run. Doing this saves you time on the initial job while then retaining copies of the native files on the final job.
      - The benefit of selecting No for this field is that you save on storage and speed, as it tells the system to look for a file path in the source workspace.
  - Use Folder Path Information—use a metadata field to build the folder structure for the documents that you promote to the review workspace.
    - Select Read From Field or Read From Folder Tree to use a metadata field to build the folder structure for the documents that you promote to the review workspace. Selecting either makes the Folder Path Information field required below.
    - Select No if you do not want to build a folder structure based on metadata. In this case, Relativity loads all documents directly into the folder indicated by the promote destination, and you create no new folders in the destination workspace.
    - You have the option of creating folders or re-foldering documents when you select Append/Overlay for the Overwrite field.
    - You have the option of re-foldering documents for the Overwrite field through the Move Existing Documents field below.
  - Folder Path Information—specify a metadata field to build the folder structure for the documents that you promote to the review workspace.
  - Move Existing Documents—re-folder documents that were previously imported into the destination workspace, but were only placed in the root case folder and not to any subfolders. This field is useful for situations in which you want to add new data to the destination workspace while overlaying existing data.
    - Select Yes to move existing documents into the folders provided in the Folder Path Information field. For example, you previously imported custodian Sally Smith's files into the destination workspace's root folder; now, in addition to placing the documents from the saved search specified in this integration point in their subfolder, you also want to move those previously migrated documents into Sally Smith's subfolder, you would select Yes.
    - Select No if you don’t want to re-folder existing documents.
  - Copy Files to Repository - determines whether or not Relativity will copy files from the location in the image load file field to a selected document repository.
    - Yes—select this to copy files to a document repository that you select in the File Repository field below.
    - No—select this if the native files already reside in their final location, which is accessible by Relativity.

Click Save to save this integration point with these export settings.

During the mapping process, Relativity validates whether the selected objects are available in the destination workspace in the final step of the wizard. When fields cannot be mapped from the Source to the Destination, a pop-up modal displays them. You can either choose to keep the fields in the Source column or skip them and remove them.

Another solution that Integration Points data promotion provides is automatically remapping fields when the destination workspace is changed. When you change the destination workspace in either the integration point or integration point profile, fields can vary between the source and destination workspace. When the destination workspace is changed, the new fields in the workspace are automatically mapped.

The artifacts are mapped based on their names only if the ArtifactIDs are different in the new destination workspace. They are mapped so they the integration points job will not fail. This means that if the source workspace fields cannot match to the new destination workspace fields, those unmatched fields are presented at the end of the mappings list for easy identification. Relativity also informs the you that the original source fields from the profile were mapped with corresponding fields from the destination workspace. The message will be similar to this, "We restored the fields mapping as destination workspace has changed."

Mapping saved searches

The Map Saved Searches button automatically maps the fields from the saved search. It also maps the object identifier fields, even when such field is not selected in the saved search. If any unsupported fields are manually mapped, clicking the Save button displays a pop-up modal stating that the job might fail.

Once Relativity saves the integration point, you will be able to run it and export the data based on the settings you specified. See Running the export job for details.

Working with promoted documents

To view the documents you promoted to the review workspace with Tag Documents with Job Name option enabled:

Navigate to the destination workspace.
Select the Saved Search browser and select the search you created to bring back documents that were promoted from the source workspace.
Note the following fields on the saved search view:
- Relativity Source Case—the name of the workspace in which you tagged documents for inclusion and exclusion and from which you promoted your tagged documents to the review workspace.
- Relativity Source Job—the name of the Integration Point that you used to promote tagged documents to the review workspace.

You can now review these documents and apply coding decisions for responsiveness and/or issues designation.

Reusing coding decisions

You can re-use the coding decisions you made on reviewed documents and promote them back into the source workspace through another Integration Point. For example, you could run another promote job to conduct a privilege overlay on documents in the source workspace.

To do this, follow the steps below.

Select the saved search you created to promote documents back to the source workspace.
Navigate to the Integration Points tab.
Create a new integration point that specifies the following values, which differ from those you entered for the promote job you ran previously:
- Destination Workspace—select the original source workspace, specifically the workspace from which you previously promoted documents to the review workspace.
- Saved Search—select the saved search you created to promote documents back to the source workspace.
- Field Mappings—map only Control Number (Object Identifier) and Privilege Designation.
- Overwrite—select Overlay Only.
Click Run.

Calculating integration point job statistics on-demand

You can calculate the number of documents, as well as images and natives if applicable, to be transferred in the scope of a particular Integration Point.

Navigate to the General tab of Integration Points > Integration Points Details for a particular integration point job.

Note: Only before the first calculation for this integration point, you will see the "Press 'Calculate statistics' button" message in the Total of Documents field.

Click the Calculate statistics button under Transfer Options in the right-hand panel.

A message box displays indicating that this action will launch the calculation of Saved Search or Production content, depending on the chosen Integration Point, and that this operation can be time consuming, depending on the size of the content. The statistics calculation will run in the background so you can still use other Transfer Options or leave this page and return to it for the results later.

Calculation results appear on the details page and depend on Integration Point content. The total number of documents with a date/time mark of the last calculation will display. Depending on the promoted content, the total number and size of images or natives may also display. The sample below displays natives.

Performance baselines

The following table presents performance baselines for using the Relativity provider in Integration Points to promote documents and metadata between workspaces.

To better identify Integration Points capabilities of exporting data between workspaces, the testing sets of Integration Points jobs were divided into three bucket categories: small, medium and large.

The jobs which yielded the following metrics were performed on a basic test environment of Relativity. Due to differences in data, infrastructure, and configuration, these should not be used as a benchmark of what you expect to see in a production environment. The results may not scale linearly.

Small	Medium	Large
Size 10-50 MB Records max 200 Fields max 200 Append Onl Duration 1-2 minutes	Size 51-1500 MB (natives) Extracted text - null Records max 5K Fields max 300 Append only Duration 2.30 min	Size 25GB (natives) Records max 500K Extracted text - null Fields max 100 Append only Duration 115min
	Size 51-1500 MB (extracted text) Records max 50K Fields max 300 Append only Duration 11 min	Size 200GB (natives) Records max 250K Extracted text - null Fields max 200 Append only Duration 8hrs
	Size 51-1500 MB (extracted text) Records max 50K Fields max 300 Overlay only Duration 4.5hrs	Size 12.5GB (natives) Records max 250K Extracted text - null Fields max 100 Overlay only Duration 4 hrs

Set Number	Copy Natives Files Setting	Overwrite Setting	Total Size	Natives	Metadata	Extracted Text	Records	Fields	Duration (seconds)
Small 1c	Physical Files	Append Only	50 MB	50 MB	3 MB	1 MB	100	190	58.33
Small 2b	Physical Files	Append Only	30 MB	30 MB	3 MB	1 MB	200	85	58.45
Small 3c	Physical Files	Append Only	10 MB	10 MB	3 MB	1 MB	100	190	61.87
Medium 1	Links Only	Append Only	600 MB	0	600 MB	0	30,000	200	319.81
Medium 2	No	Append Only	~1.5 GB	0	~5 MB	1.5 GB	50,000	2	15644.38
Medium 2a	No	Append Only	~1.5 GB	0	~5 MB	1.5 GB	50,000	2	663.66
Medium 3	Physical Files	Append Only	~1.55 GB	1.5 GB	~50 MB	0	5,000	100	124.13
Medium 4	Physical Files	Append Only	~1.65 GB	1.5 GB	~150 MB	0	5,000	300	133.48
Large 1	Links Only	Append Only	~9 GB	0	~9 GB	0	250,000	350	7569
Large 3	Physical Files	Append Only	~30 GB	25 GB	~5 GB	0	500,000	100	6909
Large 4	Physical Files	Append Only	~37.5 GB	12.5 GB	~2.5 GB	0	250,000	100	14831
Large 5	Physical Files	Append Only	~205 GB	200 GB	~5 GB	0	250,000	200	29256