Hi Seon je Kim,
After going through the question, I understand that you want to handle your large nD dataset without overloading your computer's memory, to do so you can try two solutions; one is Tall Arrays and other is memory mapping.
Tall Arrays in MATLAB
Tall arrays in MATLAB are designed to handle large datasets that are too big to fit into memory. They allow you to work with data that can be as large as the available disk space on your system. When you use tall arrays, MATLAB performs operations in chunks, processing only a portion of the data at a time, which prevents memory overflow errors. To use tall arrays with your nD dataset in the dm4 format, you'll first need to convert the data into a format that MATLAB can efficiently process in chunks, such as a collection of CSV files or a MAT-file. Once the data is in a compatible format, you can create a tall array using the `tall` function. MATLAB will manage the memory and disk usage for you, allowing you to perform operations on the dataset as if it were entirely in memory. For example, you can calculate statistics, filter data, and even perform machine learning tasks without loading the entire dataset into RAM at once.
Memory Mapping in MATLAB
Memory mapping is another technique for handling large datasets that don't fit into RAM. It allows you to map a portion of a file directly into your address space and access it as if it were loaded in memory. This can be done in MATLAB using the “memmapfile” class, which creates a memory map to a file containing binary data. This is particularly useful for large dm4 files, as you can access and process parts of the data without reading the entire file into memory. With memory mapping, you can specify the format and dimensions of the data to interpret the binary file correctly. You can then access the data using indexing operations, which MATLAB translates into file I/O operations behind the scenes. This method is efficient because it only reads the parts of the file you need into memory, keeping the memory footprint low and preventing 'out of memory' errors while allowing you to work with data slices that fit within your available RAM.
You can refer to following documents for more information regarding Memory mapping Operations and can also find related MATLAB Answers for better understanding.
Hope this helps!
