General recommendations for structured data import and export jobs

The following are recommendations for importing and exporting load files.

Load file import recommendations

Use the following recommendations to optimize performance for load file imports:

• Use Pre-check Load File feature to validate load file data before executing the import job. We strongly recommend using this feature to make sure that the import job does not come across metadata errors. If your workflow uses Staging, the Pre-check Load File step will not be available. In such case, validate your load file manually before proceeding.
• When using Import/Export in web only mode, make sure that the full path listed in the load file is captured in the ZIP file.
• Try to limit the number of fields in a single import job to 100. This minimizes the risk of job failure due to SQL exception related issues.
• When importing new choice fields, you may experience unexpected behavior, such as import errors or slow job performance for choice fields. To help prevent these issues, try to limit the number of choice values in a single import job to 100. This is the maximum number of new values across all choice type fields in the record set.
• Extracted text import:
  • There is no limitation or performance impact related to size of extracted text when the import is performed to Data Grid.
  • Depending on the job configuration, the extracted text import to Data Grid may be 60-80% faster than similar import to SQL.
  • Performance improves when extracted text files use UTF-16 encoding.
  • Performance and stability are affected for the extracted text import to SQL. If SQL is the only option, import extracted text separately from other data fields.

Load file export recommendations

Use the following recommendations to optimize performance for load file exports:

• For some combinations of job configuration and data, Windows system on your local machine may not be able to open ZIP archives due to too long file paths. You may receive an error message: “The compressed (zipped) folder is invalid.” In such case, you can do the following:
  • Use Express Transfer to export the dataset to your local disk without creating an archive.
  • Use other tools to extract the archive or move the archive file to a different location on your local machine.
  • Group files by “File Type” and not “Workspace Structure”. This may shorten the folder structure inside the archive file.
• Depending on the exported data structure, a Workspace folder structure can degrade performance significantly, up to 30 times slower in edge cases. We recommend using the File Type folder structure.
• Exporting many fields with complex data structures may affect performance or cause job errors. We recommend exporting only necessary fields.
• Increasing the Max Files to over 10,000 per folder may lead to significant export performance degradation. We recommend using the default setting of the Max Files per folder=2,000.
• The default Max (Volume) Size is set to 6,000 MB. You can adjust this setting to any value that works best for you. Setting it higher than the default value does not impact the export performance.
• Exporting images as PDFs or multi-page TIFFs is slower than exporting them as single-page images.
• Exporting to a ZIP archive using Import/Export in web only mode is significantly slower than exporting data with Express Transfer or exporting data to the Staging only.
• Exporting text files in Unicode (UTF-16) encoding is faster than using other encodings.