您正在关注此问题

您将在关注的内容源中看到更新。
您可能会收到电子邮件，具体取决于您的通信预设项。

Faster way of loading data than loading mat file with array? / Faster way than logical indexing or find?

4 次查看（过去 30 天）

显示更早的评论

Gerrit 2020-1-22

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/501499-faster-way-of-loading-data-than-loading-mat-file-with-array-faster-way-than-logical-indexing-or-f

⋮

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/501499-faster-way-of-loading-data-than-loading-mat-file-with-array-faster-way-than-logical-indexing-or-f

评论： Gerrit 2020-1-23

Hello everybody,

I need to load a really big amount of data into matlab. I could speed it up by moving from a table to an array and by using v6 to save it. But still it takes almost 50% of the time of the program. Is there an even faster method?

After that, I need to find certain values in this array. This takes about 40% of the time. Is there a faster emthod than logical indexing?

13 个评论
显示 11更早的评论隐藏 11更早的评论

Star Strider 2020-1-22

此评论的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/501499-faster-way-of-loading-data-than-loading-mat-file-with-array-faster-way-than-logical-indexing-or-f#comment_788796

⋮

链接

此评论的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/501499-faster-way-of-loading-data-than-loading-mat-file-with-array-faster-way-than-logical-indexing-or-f#comment_788796

Depending on the options available to you, installing a solid-state drive (SSD) in your computer or as an external drive would be the best option. Lacking that (and depending on how large the file is), saving it to an SD card (that currently can have reasonably large capacities) would be something to experiment with.

Guillaume 2020-1-22

此评论的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/501499-faster-way-of-loading-data-than-loading-mat-file-with-array-faster-way-than-logical-indexing-or-f#comment_788798

⋮

链接

此评论的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/501499-faster-way-of-loading-data-than-loading-mat-file-with-array-faster-way-than-logical-indexing-or-f#comment_788798

What format is the data stored as? text file? excel? If you mention -v6 does that mean it's stored as a (binary) mat file? If it's a mat file, I doubt that the storage method has much of an impact.

"I need to find certain values in this array" This is too vague for us to answer. Find according to what rule/criteria? To do what with the values afterward?

Gerrit 2020-1-22

此评论的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/501499-faster-way-of-loading-data-than-loading-mat-file-with-array-faster-way-than-logical-indexing-or-f#comment_788799

⋮

链接

此评论的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/501499-faster-way-of-loading-data-than-loading-mat-file-with-array-faster-way-than-logical-indexing-or-f#comment_788799

You are right, but I need to speed it up on every computer.

Gerrit 2020-1-22

此评论的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/501499-faster-way-of-loading-data-than-loading-mat-file-with-array-faster-way-than-logical-indexing-or-f#comment_788802

⋮

链接

此评论的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/501499-faster-way-of-loading-data-than-loading-mat-file-with-array-faster-way-than-logical-indexing-or-f#comment_788802

在 MATLAB Online 中打开

Yeah, it's stored as a binary mat file. At first, I used tables for my program. There I had a loading time of about 2.4s. Now I use arrays and the loading time decreased to 1.1s. Thats why I ask.

This is what I meant by I need to find certain values. For my 4D Interpolation afterwards, I only need values in a specific area.

new_upper_airfoil_all=new_upper_airfoil_all(new_upper_airfoil_all(:,4) < (Ma+delta_ini) & new_upper_airfoil_all(:,4) > (Ma-delta_ini) & new_upper_airfoil_all(:,5) < (C_l+delta_ini) & new_upper_airfoil_all(:,5) > (C_l-delta_ini),2:7);
new_lower_airfoil_all=new_lower_airfoil_all(new_lower_airfoil_all(:,4) < (Ma+delta_ini) & new_lower_airfoil_all(:,4) > (Ma-delta_ini) & new_lower_airfoil_all(:,5) < (C_l+delta_ini) & new_lower_airfoil_all(:,5) > (C_l-delta_ini),2:7);

Guillaume 2020-1-22

此评论的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/501499-faster-way-of-loading-data-than-loading-mat-file-with-array-faster-way-than-logical-indexing-or-f#comment_788806

⋮

链接

此评论的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/501499-faster-way-of-loading-data-than-loading-mat-file-with-array-faster-way-than-logical-indexing-or-f#comment_788806

If you're using plain arrays, that's as fast as you'll get with matlab. Similarly, your current indexing is the best you can do.

On the other hand, the problem may come from your algorithm itself, which may be (very) inefficient regardless of the storage method. Do you know the complexity of your algorithm? If it's exponential or worse, then you may need to find something more efficient.

Gerrit 2020-1-22

此评论的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/501499-faster-way-of-loading-data-than-loading-mat-file-with-array-faster-way-than-logical-indexing-or-f#comment_788807

⋮

链接

此评论的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/501499-faster-way-of-loading-data-than-loading-mat-file-with-array-faster-way-than-logical-indexing-or-f#comment_788807

编辑：Gerrit 2020-1-22

Ok, thank you. What do you mean by algorithm? The rest of my code? Yeah I still try to vectorize it, but I can't do it till now. But this would only get me some millisecounds. But this programm needs to run for 7500 cases, so it would add up.

Walter Roberson 2020-1-22

此评论的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/501499-faster-way-of-loading-data-than-loading-mat-file-with-array-faster-way-than-logical-indexing-or-f#comment_788882

⋮

链接

此评论的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/501499-faster-way-of-loading-data-than-loading-mat-file-with-array-faster-way-than-logical-indexing-or-f#comment_788882

If you have plain numeric arrays then the fastest way to load them is to use fread() from a binary file.

Walter Roberson 2020-1-22

此评论的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/501499-faster-way-of-loading-data-than-loading-mat-file-with-array-faster-way-than-logical-indexing-or-f#comment_788884

⋮

链接

此评论的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/501499-faster-way-of-loading-data-than-loading-mat-file-with-array-faster-way-than-logical-indexing-or-f#comment_788884

The data that you are currently doing the searching for ranges: is that data in sorted order? Because if it is, then binary search can be used to find the boundary in log2 time instead of linear time. I seem to recall that there is a File Exchange contribution to do binary search at the mex level.

James Tursa 2020-1-22

此评论的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/501499-faster-way-of-loading-data-than-loading-mat-file-with-array-faster-way-than-logical-indexing-or-f#comment_788887

⋮

链接

此评论的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/501499-faster-way-of-loading-data-than-loading-mat-file-with-array-faster-way-than-logical-indexing-or-f#comment_788887

A mex routine could avoid extracting the temporary arrays such as new_upper_airfoil_all(:,5) etc, but I'm not sure how much time that would save overall. Is the data in the arrays sorted? If so, a mex routine could use a binary search to find the edges and speed up the sub-array extraction.

Gerrit 2020-1-22

此评论的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/501499-faster-way-of-loading-data-than-loading-mat-file-with-array-faster-way-than-logical-indexing-or-f#comment_788960

⋮

链接

此评论的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/501499-faster-way-of-loading-data-than-loading-mat-file-with-array-faster-way-than-logical-indexing-or-f#comment_788960

I use load at the moment, I will try fread tomorrow. The data is sorted by the AeroDataId. For every AerodataId there are 70 points in the array. Is plain numeric array something else than a normal array?

Walter Roberson 2020-1-22

此评论的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/501499-faster-way-of-loading-data-than-loading-mat-file-with-array-faster-way-than-logical-indexing-or-f#comment_788962

⋮

链接

此评论的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/501499-faster-way-of-loading-data-than-loading-mat-file-with-array-faster-way-than-logical-indexing-or-f#comment_788962

In MATLAB, many kinds of arrays are "normal", including cell arrays. "plain numeric array" is emphasizing on a simple real-valued double array especially one that is vector or 2d.

Guillaume 2020-1-23

此评论的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/501499-faster-way-of-loading-data-than-loading-mat-file-with-array-faster-way-than-logical-indexing-or-f#comment_789128

⋮

链接

此评论的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/501499-faster-way-of-loading-data-than-loading-mat-file-with-array-faster-way-than-logical-indexing-or-f#comment_789128

If the data is saved in a mat file, fread won't help (unless you're planning to write your own mat file parser).

Gerrit 2020-1-23

此评论的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/501499-faster-way-of-loading-data-than-loading-mat-file-with-array-faster-way-than-logical-indexing-or-f#comment_789135

⋮

链接

此评论的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/501499-faster-way-of-loading-data-than-loading-mat-file-with-array-faster-way-than-logical-indexing-or-f#comment_789135