Document TypeㅣTroubleshooting
CategoryㅣApp Development
Applicable Product Versionㅣ6FS06
Document NumberㅣTDETS011
Issue
During DPI execution, memory usage continuously increases, and after a certain point, an out of memory issue occurs.
NoteDPI (Data Processing Interface) is an interface used in a series of batch operations that extract, process, and store large amounts of data. It is mainly utilized for data migration or interface processing with connected systems.
Cause
During DPI execution, a structure variable is used to store information for each row, and within this variable, there is an array variable that stores the actual data. The issue is that the length of this array variable is fixed based on "maximum number of columns × maximum length each column can have" (approximately 98MB). Because of this, the structure size is excessively set, leading to excessive memory usage and out-of-memory problems.
Solutions
To resolve the out-of-memory issue occurring during DPI execution, the size of the array variable that stores the actual data within the structure is improved to be dynamically allocated only as much as the number of columns × the maximum length each column can have.
Below is a comparison of memory usage before and after the improvement.
[Before Patch]
- Structure size: dop × maximum number of columns × maximum length of columns
- Example: 16 × 1500 × 65536 = approximately 1.6GB
[After Improvement Patch]
- Structure size: dop × actual number of columns × maximum length of each column
- Example (based on 5 columns): 16 × 5 × 65536 = approximately 5MB
If necessary, memory usage can also be reduced by lowering the dop value.
Note