Promoting data between workspaces through Integration Points

Note: The title of this topic used to be "Sync with Integration Points." The content below remains the same.

You can use Integration Points to promote (move) data from one Relativity workspace to another Relativity workspace. This can be thought of as a one-way sync process from a source workspace to a destination workspace and not a back and forth syncing process between them.

Once you have tagged documents for inclusion in or exclusion from the data you want to promote to review, you can access Integration Points to start the job or jobs that will send those documents to the review workspace.

This topic provides details on Integration Points as it is used to promote data between workspaces. For information, see Integration Points

This page contains the following information:

Special considerations for large data promotion workflows

We recommend configuring integration point jobs into batches for large data promotion (sync) workflows. The following are recommendations when creating job batches:

  • Document count should be no more than 500,000 documents in a single batch.
  • When including extracted text as a mapped field in the integration point job, the sum of the extracted text in the batch should not exceed 25 GB.
  • The recommended number of fields to be mapped should be no greater than 100, but it is best to map as few long text fields as possible.
  • When using the append/overlay or overlay only configuration, the batch size recommendations should be half the amount described above, specifically:
    • 250,000 document count
    • 12.5 GB extracted text sum

Exporting to a Relativity workspace

In addition to exporting to a CSV and load file, you have the option of promoting (exporting) data from a source workspace to a destination workspace in Relativity. You can do this by selecting Relativity as the destination for the data that you intend to transfer and then selecting a specific workspace where you want that data to go. This is the most efficient way to move documents from one workspace to another, since it is a one-step process, and it does not require you to export and re-import your data.

    Notes:
  • You can transfer document objects and non-document object metadata to a destination workspace. The non-document object must already exist in the destination workspace for non-document object metadata to be promoted.
  • You can move original images and natives or metadata. You are not required to have Integration Points installed on the destination workspace.
  • The data promotion process supports transfer of non-document objects linked to the Relativity Applications installed in the destination workspace. For example, if you want to push entities object to another workspace, make sure that the proper application is installed there, for example Legal Hold. You need to have the Relativity Application permission in both the Workspace Security and Object Security sections set to View.
  • If promoted fields have associative objects, these objects need to be transferred first to ensure successful promotion of fields having references to them.
  • The data promotion process will not transfer custom objects created by users in a workspace.
  • The data promotion process cannot transfer complex objects that have relationships to other objects, such as Search Terms Report (STR), structured analytic sets, and Integration Points profiles, because related objects' ArtifactIDs may be different in the source and destination workspaces.
  • When the Copy Images radio button is selected, the field mapping section is disabled because only the control number is required and available in this scenario. If you want to transfer other field metadata, you must create a new integration point without choosing to copy images.
  • Integration Points uses the following delimiters to configure the Import API for the destination workspace. Other delimiters typically configured with the Import API, for example via the RDC, are not utilized:

    • Multi-value delimiter: ASCII 029

    • Nested-value delimiter: ASCII 030

Setup

To create an integration point specifically for exporting a workspace, follow the steps below.

  1. Navigate to the Integration Points tab.
  2. Click New Integration Point.
  3. Complete the following fields in the Setup category of the Create Integration Point layout:
    • Name — the name of your integration for reference purposes.
    • Type — select Export to designate this as an export job. Selecting this sets the Source field to be Relativity.
    • Source — this is automatically set to Relativity since you are exporting data out of a Relativity workspace.
    • Destination — select Relativity. This is because you do not want to export directly to another Relativity workspace, but instead want to leverage the RDC export functionality, which is available on a subsequent layout only if you select Load File here.
    • Transferred Object — select the available Document or non-document object meta data you want to promote to the destination workspace. The non-document object must already exist in the destination workspace. The data promotion process will not transfer custom objects created by users in a workspace. Refer to the Notes above for additional information.
        Notes: You will need to create a new integration point for each object type that you want to export to the destination workspace.
    • Profile — use this to complete the remaining Integration Points settings based on the settings of a saved profile. This includes all of the fields in the Connect to Source layout, as well as field mappings. If no profiles exist in the workspace, you do not have the option of selecting them. To apply a profile that you have already created, select it from the drop-down list. For more information on profiles, see Integration Points profiles.
    • Email Notification Recipients — enter the email addresses of those who should receive notifications of whether the integration point export succeeded or failed. Use semi colons to separate email addresses.
    • Log Errors — select Yes or No to denote whether Relativity tracks item level errors.
      • If you select Yes, each job also logs any item level errors.
      • If you select No, Relativity does not log these item level errors.
      • Regardless of your selection, job-level errors are always recorded in Relativity.
    • Enable Scheduler — gives you the option of scheduling additional exports. Selecting Yes makes the following fields available:
      • Frequency — the interval at which Relativity runs this integration point.
        • Daily — select this option to promote data once every day.
        • Weekly — select this option to promote data on a weekly basis. You can specify how often in the Reoccur field, in which you will provide a number value in the Every # week(s) choice. You can then specify on which day of the week the data promotion will take place by checking any of the days of the week listed.
        • Monthly — select the day of the month that you want this integration point to promote data once every month.

        • Reoccur — enter the number of month(s) in which this integration point repeatedly promote data.
        • Send On:
          • Day __ the month — select the day of the month that you want this integration point to promote data.
          • The __ of the Month — select this option for this integration point to promote data on the chosen day of every month. For example, "The Second Friday of the month
    • Log Errors — select Yes or No to denote whether Relativity tracks item level errors. Regardless of your selection, job-level errors are always recorded in Relativity.
      • If you select Yes, each job also logs any item level errors.
      • If you select No, Relativity does not log these item level errors.
    • Enable Scheduler — gives you the option of scheduling additional exports. Selecting Yes makes the following fields available:
      • Frequency — the interval at which Relativity runs this integration point.
        • Daily — select this option to promote data once every day.
        • Weekly — select this option to promote data on a weekly basis. You can specify how often in the Reoccur field, in which you will provide a number value in the Every # week(s) choice. You can then specify on which day of the week the data promotion will take place by checking any of the days of the week listed.
        • Monthly — select the day of the month that you want this integration point to promote data once every month.

        • Reoccur — enter the number of month(s) in which this integration point repeatedly promote data.
        • Send On:
          • Day _ the month — select the day of the month that you want this integration point to promote data.
          • The __ of the Month — select this option for this integration point to promote data on the chosen day of every month. For example, "The Second Friday of the month.
      • Start Date — the date that you want Integration Points to start promoting the data.
      • End Date — (Optional) the date that you want Integration Points to stop promoting the data. Leaving the End Date blank causes the Integration Point to run indefinitely at the scheduled interval.
      • Scheduled Time — the time at which this integration point promote data. This time is local to your PC, not to the server.

  4. Click Next to advance to the Source Information layout.

Connect to Source

Continue to create your import integration point by connecting Relativity to the data source by following the steps below.

  1. In the Connect to Source layout, complete the following fields:
    • Source — select Document as the transferred object during setup, then select Saved Search or Production from the drop-down list. If you select any other object as the transferred object during setup, this is automatically set to View.
    • Saved Search/Production/View — depending on the selected Transferred Object, one of the following fields displays:
      • Saved Search — elect an existing saved search to use from the drop-down list or click the ellipsis button for an alternative way to select a saved search.
      • Production Set — select an existing production set to use from the drop-down list.
      • View — select an existing View to use from the drop-down list.
    • Destination Workspace — select an existing workspace to export your saved search or production to.

      Note: Relativity will validate whether the selected object is available in the destination workspace and transfer only objects linked to Relativity Applications installed in the destination workspace. If the selected object is not available in the destination workspace, an error message will display.

    • Location — select a folder or production set as the destination location.
      • Folder — select the drop-down menu to expand the folder structure, locate the folder from which you want to export the data, and select it. This field is only available if you have selected Folder.
      • Note: When creating an integration point based on a profile, if you change the saved search, folder, or production set on the Setup layout, the fields listed in the Selected Fields window below are not updated to reflect the new source that you selected. If you're not using a profile to creating the integration point, these fields are updated to reflect the newly selected source.

      • Production Set — select a production set to export the data to. Click the plus sign to create a new production set.

    • Create Saved Search — select Yes to create a saved search in the destination workspace. This saved search's name will take the name given to the integration point.

  2. Click Next to advance to the Destination Information layout.

Map Fields

Map the attributes or fields so that Integration Points imports the targeted data into specific Relativity fields by following the steps below.

  1. In the Map Fields wizard, you have the following options for mapping fields:
    • Use the Shift + click and Ctrl + click method to select multiple fields at a time, similar to field mapping in the Relativity Desktop Client.
    • Use the single and double arrows or double-click a field to move selected fields between columns.
    • The field names in the Destination column include the type of each field listed.
    • Click the Map Fields button between the Source and Destination columns to automatically map all fields with matching names. If you have Destination fields that are mapped to Fields in the Field Catalog, Relativity tries to find name matches between these Catalog Fields, as well.

        Notes:
      • If the WebAPIPath instance setting in the kCur.IntegrationPoints section is not configured correctly after upgrade or installation, the Source field list will not be able to display any attributes and will be empty. As a result, you will not be able to map fields.

      • You do not need to map all attributes or fields. Only the Unique Identifier and Object Identifier are required. The Unique Identifier should contain a value that no other item in the workspace contains. For example, use the GUID or distinguishedName attribute. The Object Identifier is the specific field on the object that holds the displayed identifier, which might not be unique. For example, the Full Name field is the Object Identifier of the Entity RDO but it might not be unique. These two identifier values can be the same.

  2. Complete the following Import Settings on the Map Fields layout:
    • Overwrite — determines how the system overwrites records once you promote data to the review workspace. This field provides the following choices:
      • Append Only — promote only new records into the review workspace.
      • Overlay Only — update existing records only in the review workspace. Any documents with the same workspace identifier are overlaid. This field acts as a link indicating to Relativity where to import the data. When you select this or Append/Overlay, you must complete the Multi-Select Field Overlay Behavior field described below.
      • Append/Overlay — adds new records to the review workspace and overlays data on existing records. When you select this or Overlay Only, you must complete the Multi-Select Field Overlay Behavior field described below. You are not able to create folders or re-folder documents when you select Append/Overlay.
    • Copy Images — choose to copy, or not copy, images to the destination workspace. Selecting Yes will copy images to your destination workspace. When yes is selected, an Image Precedence field needs to be completed. Selecting No will not copy images to your destination workspace. When no is selected, a Copy Native Files needs to be completed.
      • Image Precedence — select Original Images or Produced Images to copy to the destination workspace.
      • Copy Native Files — select Physical Files, Links Only, or No to determine how to copy native files to the destination workspaces.
        • Physical files — select to copy any native files from the source workspace to the destination workspace.
        • Links Only — select to add only links to the destination workspace that direct back to the source workspace documents.

          Note: When choosing this option to promote documents using only links from repository to review workspace, keep in mind that if you delete the native files in the repository workspace or if you delete or archive the repository workspace itself, the links to the promoted documents will be broken in the review workspace. In this situation, if you wanted to maintain the files in the review workspace, you first need to promote the physical files to that workspace before deleting them from the repository workspace or before deleting or archiving the repository workspace. For details on promoting subsets of data, see Repository workspace special considerations.

          Note: Be aware that when a non-administrator user copies native files by selecting the Links Only option, the System Created by column on the Job History tab will list the system administrator’s name instead of the non-administrator user's name.

        • No — select to not create additional copies of your native files. No native files will be pushed to the destination workspace. Only metadata, such as extracted text or coding decisions, will be pushed. For example, if you have a document with native files and coding decisions, and you select the No option, only metadata (the coding decisions) will be promoted to the destination workspace. Refer to additional notes below regarding this choice. 
          • Doing this maintains a single copy of that data file no matter how many times you use the document in your workspace.
          • You have the option of selecting No for an initial run of an integration point and then Yes for a subsequent run. Doing this saves you time on the initial job while then retaining copies of the native files on the final job.
          • The benefit of selecting No for this field is that you save on storage and speed, as it tells the system to look for a file path in the source workspace.

      • Use Folder Path Information — use a metadata field to build the folder structure for the documents that you promote to the review workspace.
        • Select Read From Field or Read From Folder Tree to use a metadata field to build the folder structure for the documents that you promote to the review workspace. Selecting either makes the Folder Path Information field required below.
        • Select No if you do not want to build a folder structure based on metadata. In this case, Relativity loads all documents directly into the folder indicated by the promote destination, and you create no new folders in the destination workspace.
        • You have the option of creating folders or re-foldering documents when you select Append/Overlay for the Overwrite field.
        • You have the option of re-foldering documents for the Overwrite field through the Move Existing Documents field below.
      • Folder Path Information — specify a metadata field to build the folder structure for the documents that you promote to the review workspace.
      • Move Existing Documents — re-folder documents that were previously imported into the destination workspace, but were only placed in the root case folder and not to any subfolders. This field is useful for situations in which you want to add new data to the destination workspace while overlaying existing data.
        • Select Yes to move existing documents into the folders provided in the Folder Path Information field. For example, you previously imported custodian Sally Smith's files into the destination workspace's root folder; now, in addition to placing the documents from the saved search specified in this integration point in their subfolder, you also want to move those previously migrated documents into Sally Smith's subfolder, you would select Yes.
        • Select No if you don’t want to re-folder existing documents.
      • Copy Files to Repository - determines whether or not Relativity will copy files from the location in the image load file field to a selected document repository.
        • Yes — select this to copy files to a document repository that you select in the File Repository field below.
        • No — select this if the native files already reside in their final location, which is accessible by Relativity.
  1. Click Save to save this integration point with these export settings.

During the mapping process, Relativity validates whether the selected objects are available in the destination workspace in the final step of the wizard. When fields cannot be mapped from the Source to the Destination, a pop-up modal displays them. You can either choose to keep the fields in the Source column or skip them and remove them.

Another solution that Integration Points data promotion provides is automatically remapping fields when the destination workspace is changed. When you change the destination workspace in either the integration point or integration point profile, fields can vary between the source and destination workspace. When the destination workspace is changed, the new fields in the workspace are automatically mapped.

The artifacts are mapped based on their names only if the ArtifactIDs are different in the new destination workspace. They are mapped so they the integration points job will not fail. This means that if the source workspace fields cannot match to the new destination workspace fields, those unmatched fields are presented at the end of the mappings list for easy identification. Relativity also informs the you that the original source fields from the profile were mapped with corresponding fields from the destination workspace. The message will be similar to this, "We restored the fields mapping as destination workspace has changed."

Mapping saved searches

The Map Saved Searches button automatically maps the fields from the saved search. It also maps the object identifier fields, even when such field is not selected in the saved search. If any unsupported fields are manually mapped, clicking the Save button displays a pop-up modal stating that the job might fail.

Once Relativity saves the integration point, you will be able to run it and export the data based on the settings you specified. See Running the export job for details.

Working with promoted documents

To view the documents you promoted to the review workspace, follow the steps below.

  1. Navigate to the destination workspace.
  2. Select the Saved Search browser and select the search you created to bring back documents that were promoted from the source workspace.
  3. Note the following fields on the saved search view:
    • Relativity Source Case — the name of the workspace in which you tagged documents for inclusion and exclusion and from which you promoted your tagged documents to the review workspace.
    • Relativity Source Job — the name of the Integration Point that you used to promote tagged documents to the review workspace.

You can now review these documents and apply coding decisions for responsiveness and/or issues designation.

Reusing coding decisions

You can re-use the coding decisions you made on reviewed documents and promote them back into the source workspace through another Integration Point. For example, you could run another promote job to conduct a privilege overlay on documents in the source workspace.

To do this, follow the steps below.

  1. Select the saved search you created to promote documents back to the source workspace.
  2. Navigate to the Integration Points tab.
  3. Create a new integration point that specifies the following values, which differ from those you entered for the promote job you ran previously:
    • Destination Workspace — select the original source workspace, specifically the workspace from which you previously promoted documents to the review workspace.
    • Saved Search — select the saved search you created to promote documents back to the source workspace.
    • Field Mappings — map only Control Number (Object Identifier) and Privilege Designation.
    • Overwrite — select Overlay Only.
  4. Click Run.

Performance baselines

The following table presents performance baselines for using the Relativity provider in Integration Points to promote documents and metadata between workspaces.

To better identify Integration Points capabilities of exporting data between workspaces, the testing sets of Integration Points jobs were divided into three bucket categories: small, medium and large.

The jobs which yielded the following metrics were performed on a basic test environment of Relativity. Due to differences in data, infrastructure, and configuration, these should not be used as a benchmark of what you expect to see in a production environment. The results may not scale linearly.

Small Medium Large
Size 10-50 MB
Records max 200
Fields max 200
Append Onl
Duration 1-2 minutes
Size 51-1500 MB (natives)
Extracted text - null
Records max 5K
Fields max 300
Append only
Duration 2.30 min
Size 25GB (natives)
Records max 500K
Extracted text - null
Fields max 100
Append only
Duration 115min
  Size 51-1500 MB (extracted text)
Records max 50K
Fields max 300
Append only
Duration 11 min
Size 200GB (natives)
Records max 250K
Extracted text - null
Fields max 200
Append only
Duration 8hrs
  Size 51-1500 MB (extracted text)
Records max 50K
Fields max 300
Overlay only
Duration 4.5hrs
Size 12.5GB (natives)
Records max 250K
Extracted text - null
Fields max 100
Overlay only
Duration 4 hrs
Set Number Copy Natives Files Setting Overwrite Setting Total Size Natives Metadata Extracted Text Records Fields Duration (seconds)
Small 1c Physical Files Append Only 50 MB 50 MB 3 MB 1 MB 100 190 58.33
Small 2b Physical Files Append Only 30 MB 30 MB 3 MB 1 MB 200 85 58.45
Small 3c Physical Files Append Only

10 MB

10 MB

3 MB 1 MB 100 190 61.87
Medium 1 Links Only Append Only 600 MB 0 600 MB 0 30,000 200 319.81
Medium 2 No Append Only ~1.5 GB 0 ~5 MB 1.5 GB 50,000 2 15644.38
Medium 2a No Append Only ~1.5 GB 0 ~5 MB 1.5 GB 50,000 2 663.66
Medium 3 Physical Files Append Only ~1.55 GB 1.5 GB ~50 MB 0 5,000 100 124.13
Medium 4 Physical Files Append Only ~1.65 GB 1.5 GB ~150 MB 0 5,000 300 133.48
Large 1 Links Only Append Only ~9 GB 0 ~9 GB 0 250,000 350 7569
Large 3 Physical Files Append Only ~30 GB 25 GB ~5 GB 0 500,000 100 6909
Large 4 Physical Files Append Only ~37.5 GB 12.5 GB ~2.5 GB 0 250,000 100 14831
Large 5 Physical Files Append Only ~205 GB 200 GB ~5 GB 0 250,000 200 29256