Best way to work with large file (viewing and manipulating the data) - memory mapped temp files?

1 次查看（过去 30 天）
Justin
Justin 2011-11-28
Background
I routinely work with large files from different lab equipment setups - sizes can run from 50 MB to 1-2 GB. Analyzing this data requires both viewing it - looking through 5-10 parameters at once for a test period - and manipulating it - mathematical operations, combining parameters, etc. So far I've been doing this by loading the file into the workspace as a data array (up to a file size of 3-400 MB or so), and poking it/plotting it as needed.
My current project is to build a large GUI for scrolling through the parameters in my data files and occasionally doing operations to them while they're on the screen. For example, I might have 2000 parameters in a 1 GB file that my GUI plots 10 at a time. I can then combine two of them into one channel, take the derivative of another, etc etc.
To do this, my thought is to load the file (out of its custom file type) and immediate save it into a 'temporary' working file - probably a standard binary file type or a .mat file. Then I'd access it in a memory-map type fashion (either with the build in memory map or a custom logic type thing). I'll keep records of the operations performed, GUI settings, etc in the active workspace, but only dig into the 'real' data when I have to.
My question - does this sound like the right way to approach this? I'm worried about speed (how fast can I read/write out of the large memory mapped file?) and about running out of memory (workspace memory, not file space).
Any ideas?
Thanks! - J
  2 个评论
Sven
Sven 2011-11-28
Do you have the latest MATLAB version? There was an enhancement to the load() function which allows you to load a certain subset of a file at one time. It might help you avoid too many temporary file issues... just save the whole data to a .mat file, then load pieces of the saved matrix on demand.
Justin
Justin 2011-11-28
Personally I do, but this app would be put out company-wide - some of the folks could potentially be a version or two behind. I'm not against relying on only the newest stuff, but being backwards compatible as much as possible is a big plus.

请先登录，再进行评论。

请先登录，再回答此问题。

回答（0 个）

类别

在 Help Center 和 File Exchange 中查找有关 Workspace Variables and MAT-Files 的更多信息

标签

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by