What's the upper limit for how much data can be stored in a structure?

16 次查看（过去 30 天）
Ibro Tutic
Ibro Tutic 2017-5-15
评论： Ibro Tutic 2017-5-15
I am working with about 20 datasets, each with a size of about 2500x300. Is it possible to open these datasets, store them, and save off the resulting structure as a mat file without running into memory issues? This is my current code, and I run into an "Out of memory" error when I run it.
for i=1:length(fieldnames(dat))
a=fieldnames(dat); %files to work with
a=a{i}; %selects upper structure
a2=fieldnames(foldername); %foldername where .dat files are held
a2=a2{i}; %select which folder
for j=1:length(fieldnames(dat.(strcat(a))))
b=fieldnames(dat.(strcat(a))); %list of filenames in upper structure
b=b{j}; %select file to open
for k=1:length(dat.(strcat(a)).(strcat(b)))
filepath = strcat(foldername.(strcat(a2)),'\', dat.(strcat(a)).(strcat(b))); %sets file path
fid = fopen(filepath{k}); %opens file
fmt = repmat('%s', 1, 360); %reads file, text string delimter
C= textscan(fid,fmt,'Delimiter','\t','CollectOutput',true); %records csv data to cell array C
fclose(fid); %closes file
DLG.(strcat(a)).(strcat(b)){k}=C{1}; %writes data to structure
end
end
end
  4 个评论
显示 2更早的评论
Guillaume
Guillaume 2017-5-15
编辑：Guillaume 2017-5-15
"I can't call Dat.a"
Indeed, the proper syntax is simply:
dat.(a)
No need for the strcat that does absolutely nothing.

请先登录，再进行评论。

请先登录，再回答此问题。

采纳的回答

Guillaume
Guillaume 2017-5-15
Your code is a bit confusing with all these apparently useless strcat (see comment to the question).
Anyway, to answer your question, you at least need 2500*300*20*8 bytes ~= 120 MB to store your datasets. Add to that the overhead of the structure fields (around 20*(numberofcharactersinfieldname + constant) and of the structure itself (not much).
Unless you've got very little memory on your computer or the data size is vastly different from what you've stated, it should all fit in memory.

更多回答（1 个）

Jan
Jan 2017-5-15
A cleaner version of the code:
fmt = repmat('%s', 1, 360); %reads file, text string delimter
datFields = fieldnames(dat);
folderFields = fieldnames(foldername); % "foldername" is a bad name for a struct
for i = 1:length(datFields)
a = datFields{i}; %selects upper structure
a2 = folderFields{i}; %select which folder
aFields = fieldnames(dat.(a));
for j = 1:length(aFields)
b = aFields{j}; %select file to open
filepath = fullfile(foldername.(a2), dat.(a).(b)); %sets file path
DLG.(a).(b) = cell(1, length(dat.(a).(b))); % Pre-allocate
for k = 1:length(dat.(a).(b))
fid = fopen(filepath{k}); %opens file
C = textscan(fid,fmt,'Delimiter','\t','CollectOutput',true); %records csv data to cell array C
fclose(fid); %closes file
DLG.(a).(b){k} = C{1}; %writes data to structure
end
end
end
I tried to move all repeated work before the loops, removed the useless strcat and there is still much potential to improve the readability. Fieldnames like "a" and "b" are not clear.

类别

在 Help Center 和 File Exchange 中查找有关 Structures 的更多信息

标签

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by