More efficient way to load data
显示 更早的评论
Hello,
Is there a more efficient way to load in data(array), I have a 1x10^7 which is roughly 4mb but takes almost 70s to load. Am using the
load('filename','varname')
command
6 个评论
Walter Roberson
2013-4-1
It might end up highly compressed.
Matt J
2013-4-1
@Natialol: If 4MB is the (compressed) file size, it is the uncompressed size that dictates how long it takes to load.
Walter Roberson
2013-4-1
Matt, decompression is done at CPU speeds, whereas reading from disk is done at disk speeds, which can be 1000 times slower. The usual grouping of speeds is
CPU registers >> Primary Cache >> Secondary Cache >> RAM >> Disk
where each step to the right is roughly 10 times slower than the one before it.
Therefore if you take an average of (say) 50 decompression instructions per disk byte loaded (and it would normally be far less), then you are still far ahead compared to reading from disk.
Walter Roberson
2013-4-1
Loads from disk to RAM generally use DMA, and thus proceed in parallel. The decompression would mostly happen in CPU register and primary cache, with the flushes to secondary cache possibly in parallel (depending on the memory controller and the CPU properties.)
回答（0 个）
类别
在 帮助中心 和 File Exchange 中查找有关 Standard File Formats 的更多信息
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!
Translated by 
