Faster way of loading data than loading mat file with array? / Faster way than logical indexing or find?
4 次查看(过去 30 天)
显示 更早的评论
Gerrit
2020-1-22
Hello everybody,
I need to load a really big amount of data into matlab. I could speed it up by moving from a table to an array and by using v6 to save it. But still it takes almost 50% of the time of the program. Is there an even faster method?
After that, I need to find certain values in this array. This takes about 40% of the time. Is there a faster emthod than logical indexing?
13 个评论
Star Strider
2020-1-22
Depending on the options available to you, installing a solid-state drive (SSD) in your computer or as an external drive would be the best option. Lacking that (and depending on how large the file is), saving it to an SD card (that currently can have reasonably large capacities) would be something to experiment with.
Guillaume
2020-1-22
What format is the data stored as? text file? excel? If you mention -v6 does that mean it's stored as a (binary) mat file? If it's a mat file, I doubt that the storage method has much of an impact.
"I need to find certain values in this array" This is too vague for us to answer. Find according to what rule/criteria? To do what with the values afterward?
Gerrit
2020-1-22
Yeah, it's stored as a binary mat file. At first, I used tables for my program. There I had a loading time of about 2.4s. Now I use arrays and the loading time decreased to 1.1s. Thats why I ask.
This is what I meant by I need to find certain values. For my 4D Interpolation afterwards, I only need values in a specific area.
new_upper_airfoil_all=new_upper_airfoil_all(new_upper_airfoil_all(:,4) < (Ma+delta_ini) & new_upper_airfoil_all(:,4) > (Ma-delta_ini) & new_upper_airfoil_all(:,5) < (C_l+delta_ini) & new_upper_airfoil_all(:,5) > (C_l-delta_ini),2:7);
new_lower_airfoil_all=new_lower_airfoil_all(new_lower_airfoil_all(:,4) < (Ma+delta_ini) & new_lower_airfoil_all(:,4) > (Ma-delta_ini) & new_lower_airfoil_all(:,5) < (C_l+delta_ini) & new_lower_airfoil_all(:,5) > (C_l-delta_ini),2:7);
Guillaume
2020-1-22
If you're using plain arrays, that's as fast as you'll get with matlab. Similarly, your current indexing is the best you can do.
On the other hand, the problem may come from your algorithm itself, which may be (very) inefficient regardless of the storage method. Do you know the complexity of your algorithm? If it's exponential or worse, then you may need to find something more efficient.
Walter Roberson
2020-1-22
If you have plain numeric arrays then the fastest way to load them is to use fread() from a binary file.
Walter Roberson
2020-1-22
The data that you are currently doing the searching for ranges: is that data in sorted order? Because if it is, then binary search can be used to find the boundary in log2 time instead of linear time. I seem to recall that there is a File Exchange contribution to do binary search at the mex level.
James Tursa
2020-1-22
A mex routine could avoid extracting the temporary arrays such as new_upper_airfoil_all(:,5) etc, but I'm not sure how much time that would save overall. Is the data in the arrays sorted? If so, a mex routine could use a binary search to find the edges and speed up the sub-array extraction.
Gerrit
2020-1-22
I use load at the moment, I will try fread tomorrow. The data is sorted by the AeroDataId. For every AerodataId there are 70 points in the array. Is plain numeric array something else than a normal array?
Walter Roberson
2020-1-22
In MATLAB, many kinds of arrays are "normal", including cell arrays. "plain numeric array" is emphasizing on a simple real-valued double array especially one that is vector or 2d.
Guillaume
2020-1-23
If the data is saved in a mat file, fread won't help (unless you're planning to write your own mat file parser).
Gerrit
2020-1-23
Yeah, I just tested that. I think, the way I have it, is the best way to do it. Now I try to focus on vectorizing my loops.
If you want, you can take a look at my question. Link: https://de.mathworks.com/matlabcentral/answers/501586-can-i-vectorize-one-of-the-loops
回答(0 个)
另请参阅
类别
在 Help Center 和 File Exchange 中查找有关 Logical 的更多信息
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!发生错误
由于页面发生更改,无法完成操作。请重新加载页面以查看其更新后的状态。
您也可以从以下列表中选择网站:
如何获得最佳网站性能
选择中国网站(中文或英文)以获得最佳网站性能。其他 MathWorks 国家/地区网站并未针对您所在位置的访问进行优化。
美洲
- América Latina (Español)
- Canada (English)
- United States (English)
欧洲
- Belgium (English)
- Denmark (English)
- Deutschland (Deutsch)
- España (Español)
- Finland (English)
- France (Français)
- Ireland (English)
- Italia (Italiano)
- Luxembourg (English)
- Netherlands (English)
- Norway (English)
- Österreich (Deutsch)
- Portugal (English)
- Sweden (English)
- Switzerland
- United Kingdom (English)
亚太
- Australia (English)
- India (English)
- New Zealand (English)
- 中国
- 日本Japanese (日本語)
- 한국Korean (한국어)