How to efficiently integrate big data without using memory / (How to create big data)

7 次查看（过去 30 天）

显示更早的评论

Mehmet OZC 2015-8-18

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/234416-how-to-efficiently-integrate-big-data-without-using-memory-how-to-create-big-data

评论： Mehmet OZC 2015-8-19

采纳的回答： JMP Phillips

in a study i will produce large arrays.
Each array will have at least 500 MB size.
Each array will have the same number of rows.
the total size of dataset will be approximately 20 GB or over.
Somehow I have to create a single variable/array which includes all data and size of 20 GB.

matfile seems a good solution. However when the size of file increases, it gets slower. How can i handle this problem?

9 个评论
显示 7更早的评论隐藏 7更早的评论

Mehmet OZC 2015-8-18

编辑：Mehmet OZC 2015-8-18

It works to a degree. When i try to append a 2 GB file to a 4GB file it gets slower. MATLAB does wonderful things. I believe it can handle this or is it impossible to create a really large file with using ordinary computers?

Walter Roberson 2015-8-18

I wonder if compression is leading to slowdowns? I do not know whether -v7.3 with matfile uses compression; see discussion http://www.mathworks.com/matlabcentral/answers/15521-matlab-function-save-and-v7-3 and http://www.mathworks.com/matlabcentral/answers/137592-compress-only-selected-variables-when-saving-to-mat

请先登录，再进行评论。

请先登录，再回答此问题。

采纳的回答

JMP Phillips 2015-8-19

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/234416-how-to-efficiently-integrate-big-data-without-using-memory-how-to-create-big-data#answer_189713

编辑：Walter Roberson 2015-8-19

Here are some things you could try:

Use the matfile function, which allows you to access and change variables directly in MAT-files, without loading into memory: http://au.mathworks.com/help/matlab/large-mat-files.html http://au.mathworks.com/help/matlab/ref/matfile.html

Structure your data differently: - if you are representing the data as doubles, maybe you can afford less accuracy e.g. use int32. For example, you can use scaling of 1e4 to represent a double value such as 100.3425 as an integer 1003425.

With MATLAB:

use 64 bit matlab version
try disabling compression when saving the files, with the -v6 option

Optimize your PC for your task:

in task manager, close any unnecessary processes running at the same time, including taskbar junk (adobe update, java update etc)
disable your anti-virus which might be trying to scan the file and slowing it down
under task manager, give higher priority to the MATLAB process (see http://www.sevenforums.com/tutorials/83361-priority-level-set-applications-processes.html)
increase your virtual memory or page file size http://windows.microsoft.com/en-au/windows/change-virtual-memory-size#1TC=windows-7
defragment your hard drive
run MATLAB from your local hard drive and not a network drive or external harddrive
save the .mat file to your local hard drive where it has plenty of space, not a network drive or external harddrive.
For faster hard drive access, use a Solid State Drive (SSD)

2 个评论
显示无隐藏无

Walter Roberson 2015-8-19

The -v6 option is incompatible with matfile and with objects over 2 Gb.

Mehmet OZC 2015-8-19

在 MATLAB Online 中打开

In one of the links provided above I have run across following code

example = matfile('example.mat','Writable',true);
[nrowsB,ncolsB] = size(example,'B');
for row = 1:nrowsB
  example.B(row,:) = row * example.B(row,:);
end