Remove rows from large MAT file

7 次查看（过去 30 天）
Zion Nahisi
Zion Nahisi 2021-8-26
回答： Vedant Shah 2025-6-20
Hi, I have a large matrix stored in Mat FIle. I loaded the matfile:
db = matfile(myFile)
and got:
db =
matlab.io.MatFile
Properties:
Properties.Source: '.MyFile'
Properties.Writable: true
Properties.ProtectedLoading: false
hs: [216817664x532 uint32]
I want to delete the lat row in hs field:
db.hs(216817664, :) = [];
But I got an error:
Requested 216817664x532 (429.7GB) array exceeds maximum array size preference. Creation of arrays greater than this limit may
take a long time and cause MATLAB to become unresponsive.
Is there any other way to remove rows from such large files?
  4 个评论
显示 2更早的评论
Ive J
Ive J 2021-8-26
Have you tried tall arrays?
db = matfile(myFile)
t = tall(db.hs);
t(216817664, :) = []; % see write doc for saving this tall array
Zion Nahisi
Zion Nahisi 2021-8-26
Same error, filed on
t = tall(db.hs);
since MATLAB cannot access the whole matrix for convert it to tall array.

请先登录，再进行评论。

请先登录，再回答此问题。

回答（1 个）

Vedant Shah
Vedant Shah 2025-6-20
Hi @Zion Nahisi,
MATLAB attempts to load the entire variable ‘hs’ into memory when a row deletion is requested but given the size of the matrix, this operation is not feasible.
A possible workaround is to create a new MAT-file that excludes the last row of the original matrix. To avoid memory overflow issues, the data should be copied in manageable chunks. The following code snippet demonstrates how this can be achieved:
src = matfile('MyFile.mat');
dst = matfile('NewFile.mat', 'Writable', true);
chunkSize = 10000;
numRows = size(src, 'hs', 1);
numCols = size(src, 'hs', 2);
for i = 1:chunkSize:(numRows - 1)
lastRow = min(i + chunkSize - 1, numRows - 1);
dst.hs(i:lastRow, :) = src.hs(i:lastRow, :);
end
This approach ensures that only a portion of the data is loaded into memory at any given time, making the process efficient and scalable for very large datasets.
For more information, please refer to the documentation using the following commands in the MATLAB command line: 
web(fullfile(docroot, " /matlab/ref/matlab.io.matfile.html"));

类别

在 Help Center 和 File Exchange 中查找有关 Logical 的更多信息

标签

产品


版本

R2020b

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by