- Documentation for Tall Arrays: https://in.mathworks.com/help/matlab/import_export/tall-arrays.html
- Using Tall Arrays: https://in.mathworks.com/help/matlab/tall-arrays.html
- Deferred Evaluation of Tall Arrays: https://in.mathworks.com/help/matlab/import_export/deferred-evaluation-of-tall-arrays.
How to store and deal with very very large matrices?
82 次查看(过去 30 天)
显示 更早的评论
% input variables
T = 2274;
N = 58;
B = 10000;
H = 250;
K = 3;
% I simulate matrices of similar size to give the idea of the problem
rng('default') % even if replicability is not necessary in this case
% first set (based on dimension)
P0 = lognrnd(0, 1, T, N);
R0 = rand(T, N) - rand(T, N);
% second set (based on dimension)
R = rand(T, N, K) - rand(T, N, K);
V0 = rand(T, N, K);
Z0 = randn(T, N, K);
% third and foruth set (based on dimension)
Z1 = randn(H, B, N, K);
R1 = rand(H, B, N, K) - rand(H, B, N, K);
V1 = rand(H, B, N, K);
P1 = lognrnd(0, 1, H, B, N, K);
Z2 = randn(H, B, N, K);
R2 = rand(H, B, N, K) - rand(H, B, N, K);
V2 = rand(H, B, N, K);
P2 = lognrnd(0, 1, H, B, N, K);
The above code is not executable due to the large size of the matrices involved. While MATLAB can handle the computations, saving and loading these big matrices is extremely slow, and loading the '.mat' files often results in crashes.
I have attempted to use the matfile function to access the variables without loading them, but it doesn't seem to provide a satisfactory solution in terms of speed.
I would greatly appreciate any suggestions or improvements on how to efficiently deal with such big matrices in MATLAB. Is there a more optimized way to handle these large datasets, particularly for saving and loading operations?
0 个评论
回答(1 个)
Animesh
2023-8-22
Hello Barbab,
I understand that you are encountering difficulty while trying to load and store large-sized matrices.
In this case, I would recommend you to use ‘Tall Arrays’ in MATLAB. Tall Arrays are used to work with out-of-memory data that is backed by a datastore. Datastores enable you to work with large data sets in small blocks that individually fit in memory, instead of loading the entire data set into memory at once.
Please refer the following MathWorks Documentation link on ‘Tall Arrays’ for better understanding on how to use them for large matrices: https://www.mathworks.com/help/matlab/tall-arrays.html
Furthermore, for additional information on ‘Tall Arrays’ in MATLAB, consider going through the reference links below:
I hope this helps!
0 个评论
另请参阅
类别
在 Help Center 和 File Exchange 中查找有关 Tall Arrays 的更多信息
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!