load specified column in matfile too slow

Question

Yu Li 2018-10-3

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/422057-load-specified-column-in-matfile-too-slow

评论： Walter Roberson 2018-10-3

I have a matfile with size of: 5e6*50.

I want to write a code to load the specific column into my memory, but I found that the time reading a specified column is nearly the same with that reading the whole matfile.

below is the test code:

    A=rand(5e6,50);
    save A A
    f=matfile('A');
    tic
    tmp=f.A;
    toc
    tic
    tmp=f.A(:,1);
    toc

is there anyway to improve the performance?

Thanks!

Yu

2 个评论
显示无隐藏无

Walter Roberson 2018-10-3

I notice you did not specifically save with -v7.3, so you might be getting a -v7 file.

When I test on my system with -v7.3, selecting one column comes out roughly 10% faster. Not as good as one might hope, though.

Yu Li 2018-10-3

Yes, the expected result should be 2% of reading the whole file, since I have totally 50 columns.

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Walter Roberson 2018-10-3

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/422057-load-specified-column-in-matfile-too-slow#answer_339628

在 MATLAB Online 中打开

In this case, you can do much better by using -nocompression

Save -v7.3 -nocompression
Elapsed time is 17.814604 seconds.
Done save -v7.3 -nocompression
Start matfile -v7.3 -nocompression
Elapsed time is 0.016278 seconds.
Done matfile -v7.3 -nocompression
Start recall entire variable -v7.3 -nocompression
Elapsed time is 2.195975 seconds.
Done recall entire variable -v7.3 -nocompression
Start recall one column -v7.3 -nocompression
Elapsed time is 1.089280 seconds.
Done recall one column -v7.3 -nocompression
Save -v7.3
Elapsed time is 58.543461 seconds.
Done save -v7.3
Start matfile -v7.3
Elapsed time is 0.077814 seconds.
Done matfile -v7.3
Start recall entire variable -v7.3
Elapsed time is 10.139135 seconds.
Done recall entire variable -v7.3
Start recall one column -v7.3
Elapsed time is 9.118167 seconds.
Done recall one column -v7.3

Source code:

A=rand(5e6,50);
time_it(A, {'-v7.3' '-nocompression'})
time_it(A, {'-v7.3'})
function time_it(A, saveoptions)
    savedesc = strjoin(saveoptions, ' ');
    fprintf('Save %s\n', savedesc);
    tic
    save('A', 'A', saveoptions{:});
    toc
    fprintf('Done save %s\n', savedesc);
      fprintf('Start matfile %s\n', savedesc);
      tic
      f = matfile('A');
      toc
      fprintf('Done matfile %s\n', savedesc);
      fprintf('Start recall entire variable %s\n', savedesc);
      tic
      tmp=f.A;
      toc
      fprintf('Done recall entire variable %s\n', savedesc);
      fprintf('Start recall one column %s\n', savedesc);
      tic
      tmp=f.A(:,1);
      toc
      fprintf('Done recall one column %s\n', savedesc);
  end

7 个评论
显示 5更早的评论隐藏 5更早的评论

Walter Roberson 2018-10-3

在 MATLAB Online 中打开

Version #3 adds per-point timing (that is, cost per matrix element retrieved). It times both A and A.' to try to account for differences in row / column sizes. Taking into account the different sizes, the summary seems to be that for this data, -nocompression is much faster, and that when compression is on, reading a row might be slightly faster than reading a column but there is probably not enough difference to make it worth rewriting code for rows vs columns

A=rand(5e6,50);
time_it(A, {'-v7.3' '-nocompression'}, '7.3 no compression')
time_it(A, {'-v7.3'}, '7.3 w/ compression')
Atrans = A.';
time_it(Atrans, {'-v7.3' '-nocompression'}, 'flipped 7.3 no compression')
time_it(Atrans, {'-v7.3'}, 'flipped 7.3 w/ compression');
function time_it(A, saveoptions, savedesc)
    fprintf('Save %s\n', savedesc);
    tic
    save('A', 'A', saveoptions{:});
    toc
    fprintf('Done save %s\n', savedesc);
      fprintf('Start matfile %s\n', savedesc);
      tic
      f = matfile('A');
      toc
      fprintf('Done matfile %s\n', savedesc);
      fprintf('Start recall entire variable %s\n', savedesc);
      tic
      tmp=f.A;
      t = toc;
      fprintf('%f seconds elapsed, %g per point\nDone recall entire variable %s\n', t, t/numel(tmp), savedesc);
      fprintf('Start recall one column %s\n', savedesc);
      tic
      tmp=f.A(:,1);
      t = toc;
      fprintf('%f seconds elapsed, %g per point\nDone recall one column %s\n', t, t/numel(tmp), savedesc);
      fprintf('Start recall one row %s\n', savedesc);
      tic
      tmp=f.A(1,:);
      t = toc;
      fprintf('%f seconds elapsed, %g per point\nDone recall one row %s\n', t, t/numel(tmp), savedesc);
      fprintf('\n');
  end

Yu Li 2018-10-3

Hi:

Thanks for your test, I got much deeper understanding about this.

I think there should have some bottom neck here, I will contact Mathworks for further investigation.

Thanks!

Yu

Walter Roberson 2018-10-3

You can refer them to this post and my test code.

One thing they are likely to point out is that the default of compression is intended for "real" data, not for rand(), and that when you read/write with compression, the performance would be expected to vary with how compressible the data is. Thus you should probably run this code with the rand() replaced by load of one of your actual matrices.

请先登录，再进行评论。

load specified column in matfile too slow

2 个评论
显示无隐藏无

回答（1 个）

7 个评论
显示 5更早的评论隐藏 5更早的评论

另请参阅

类别

标签

Community Treasure Hunt

load specified column in matfile too slow

2 个评论 显示 无隐藏 无

回答（1 个）

7 个评论 显示 5更早的评论隐藏 5更早的评论

另请参阅

类别

标签

Community Treasure Hunt

2 个评论
显示无隐藏无

7 个评论
显示 5更早的评论隐藏 5更早的评论