Data structure for mixed-type arrays - cellarray, dataset, structarray, struct of arrays, or other?
44 次查看(过去 30 天)
显示 更早的评论
I need to process large amounts of tabular data of mixed type - strings and doubles. A standard problem, I would think. What is the best data structure in Matlab for working with this?
Cellarray is definitely not the answer. It is extremely memory inefficient. (tests shown below). Dataset (from stats toolbox) is horribly time and space inefficient. That leaves me with structarray or struct of arrays. I did a test across all four different options for both time and memory below and it seems to me the struct of arrays is the best option for the things I tested for.
I am relatively new to Matlab and this is a bit disappointing, frankly. Anyway - looking for advice on whether I am missing something, or if my tests are accurate/reasonable. Am I missing other considerations besides access/conversion/memory usage that are likely to come up as I code more using this stuff. (fyi am using R2010b)
**** Test #1: Access speed Accessing a data item.
cellarray:0.002s
dataset:36.665s %<<< This is horrible
structarray:0.001s
struct of array:0.000s
**** Test #2: Conversion speed and memory usage I dropped dataset from this test.
Cellarray(doubles)->matrix:d->m: 0.865s
Cellarray(mixed)->structarray:c->sc: 0.268s
Cellarray(doubles)->structarray:d->sd: 0.430s
Cellarray(mixed)->struct of arrays:c->sac: 0.361s
Cellarray(doubles)->struct of arrays:d->sad: 0.887s
Name Size Bytes Class Attributes
c 100000x10 68000000 cell
d 100000x10 68000000 cell
m 100000x10 8000000 double
sac 1x1 38001240 struct
sad 1x1 8001240 struct
sc 100000x1 68000640 struct
sd 100000x1 68000640 struct
================== CODE: TEST#1
%%cellarray
c = cell(100000,10);
c(:,[1,3,5,7,9]) = num2cell(zeros(100000,5));
c(:,[2,4,6,8,10]) = repmat( {'asdf'}, 100000, 5);
cols = strcat('Var', strtrim(cellstr(num2str((1:10)'))))';
te = tic;
for iii=1:1000
x = c(1234,5);
end
te = toc(te);
fprintf('cellarray:%0.3fs\n', te);
%%dataset
ds = dataset( { c, cols{:} } );
te = tic;
for iii=1:1000
x = ds(1234,5);
end
te = toc(te);
fprintf('dataset:%0.3fs\n', te);
%%structarray
s = cell2struct( c, cols, 2 );
te = tic;
for iii=1:1000
x = s(1234).Var5;
end
te = toc(te);
fprintf('structarray:%0.3fs\n', te);
%%struct of arrays
for iii=1:numel(cols)
if iii/2==floor(iii/2) % even => string
sac.(cols{iii}) = c(:,iii);
else
sac.(cols{iii}) = cell2mat(c(:,iii));
end
end
te = tic;
for iii=1:1000
x = sac.Var5(1234);
end
te = toc(te);
fprintf('struct of array:%0.3fs\n', te);
================== CODE: TEST #2
%%cellarray
% c - cellarray containing mixed type
c = cell(100000,10);
c(:,[1,3,5,7,9]) = num2cell(zeros(100000,5));
c(:,[2,4,6,8,10]) = repmat( {'asdf'}, 100000, 5);
cols = strcat('Var', strtrim(cellstr(num2str((1:10)'))))';
% c - cellarray containing doubles only
d = num2cell( zeros( 100000, 10 ) );
%%matrix
% doubles only
te = tic;
m = cell2mat(d);
te = toc(te);
fprintf('Cellarray(doubles)->matrix:d->m: %0.3fs\n', te);
%%structarray
% mixed
te = tic;
sc = cell2struct( c, cols, 2 );
te = toc(te);
fprintf('Cellarray(mixed)->structarray:c->sc: %0.3fs\n', te);
% doubles
te = tic;
sd = cell2struct( d, cols, 2 );
te = toc(te);
fprintf('Cellarray(doubles)->structarray:d->sd: %0.3fs\n', te);
%%struct of arrays
% mixed
te = tic;
for iii=1:numel(cols)
if iii/2==floor(iii/2) % even => string
sac.(cols{iii}) = c(:,iii);
else
sac.(cols{iii}) = cell2mat(c(:,iii));
end
end
te = toc(te);
fprintf('Cellarray(mixed)->struct of arrays:c->sac: %0.3fs\n', te);
% doubles
te = tic;
for iii=1:numel(cols)
sad.(cols{iii}) = cell2mat(d(:,iii));
end
te = toc(te);
fprintf('Cellarray(doubles)->struct of arrays:d->sad: %0.3fs\n', te);
clear iii cols te;
whos
5 个评论
Matt J
2013-4-26
编辑:Matt J
2013-4-26
I'm a bit surprised to discover that sortrows works on cells. Nothing in the documentation about it.
Anyway, the fact that it requires multiple lines of code wouldn't make it "painful" in my book. You can always encapsulate the multiple lines in your own mfile and just reuse that.
The issue you point out is also only an issue when the columns being sorted are of mixed type. My approach might be to convert all the numeric data to strings and then concatenate the columns being sorted into one big string matrix. Then I can run sortrows on that.
per isakson
2013-4-26
[num2cell(sac.Var1(1:10)), sac.Var2(1:10), ...]
This operation represents a cost. What is the benifit of storing Var1, Var2, etc. in separate fields compared to a double array?
采纳的回答
Marc
2016-2-29
I thought about asking a question but was wondering if the "table" structure fits this. I cannot confirm when this data type was released (looks like it wasn't in 2010a or 2012a) but I have been using this now for a wide range of stuff that I dump into excel files and want to do some serious analysis on. In 2015b, the help doc refers you to table when looking at dataset, so it looks like the table data type will be replacing this in the future. Works pretty well if you do a lot of design of experiments and like me, have to dump the data somewhere like excel or some database due to work requirements.
If you format the excel file correctly, reading in tabular data to a table is really easy.
2 个评论
per isakson
2016-2-29
编辑:per isakson
2016-2-29
- table, Create table from workspace variables says: Introduced in R2013b
- dataset is in the Statistical Toolbox. However, it seems as if they decided to move the functionality to table in Matlab itself.
Guillaume
2016-2-29
In the context of this thread, a table is equivalent to a cell of arrays in term of memory overhead. Access is a bit more convenient at the expense of performance.
更多回答(2 个)
Matt J
2013-4-26
编辑:Matt J
2013-5-18
As best I can tell, you haven't tested a "cell of arrays", i.e., instead of having a 100000x10 cell array, have a 1x10 cell array where each c{i} contains an array of a column of data. Should be similar to "struct of arrays", but with easier indexing.
Beyond that, nothing in your tests is very unexpected. You have a large amount of data and have to be careful not to scatter it discontiguously in memory. Successive cell/struct elements cannot be held contiguously in memory, because they hold non-homogeneous data types. Numeric and string arrays are contiguous, however, so by grouping things into large numeric/string sub-arrays where possible, you maximize data contiguity, which leads to efficiencies both in access speed and memory usage.
As for "dataset", I cannot comment, since I don't have the Stats Toolbox. However, a mixed data table with 100000 rows is uncommonly large in my experience. I don't think you would ever see it in an Excel spreadsheet, for example. If dataset was meant to be "Excel-like", I can imagine 100000 rows being usage outside of what the designers anticipated.
3 个评论
matal
2013-5-17
1 个评论
Cedric
2013-5-17
编辑:Cedric
2013-5-17
It would not be too difficult to create your own class for that if you can precisely define what you need, moreover if you know OOP but never used it in MATLAB. Let us know if you are interested, this would be a good "case/pretext/application" to make the step towards OOP. It would not be more efficient than managing numeric arrays and cell arrays, as you would essentially build a wrapper around these structures with proper methods to manage size/indexing, but it would make the whole clean and consistent.
另请参阅
类别
在 Help Center 和 File Exchange 中查找有关 Structures 的更多信息
产品
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!