concatenate array of structures with same field names

Question

gujax 2023-6-5

1
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1978719-concatenate-array-of-structures-with-same-field-names

评论： gujax 2023-6-6

采纳的回答： Matt J

在 MATLAB Online 中打开

Hi

I am reading datafile from a camera which spits out data in a certain format - about several Gb's in size.

The output format is more complex than the example below:

m = memmapfile(def_file,'Format',{'uint8',[8,1],'identifier';'uint8',[504,1],'Header';'uint16',[2^16,1],'Pix'},'Repeat',N,'Offset',0,'writable',false);

where N=8000

and this is the output

m.Data

8000×1 struct array with fields:

identifier

Header

Pix

so that m.Data.Pix has 8000 fields each with 65536 element vector

and m.Data.identifier also has 8000 fields each with 8 element vector

I want to concatenate each structure array example for N=20 is a script I adapted from the forum

M = CatStructFields(m.Data,1,1);
tic
Dat = CatStructFields(m.Data,1,3);
toc

To concatenate N=8000 for Pix (j=3) takes ~ half an hour.

There must be a faster way to do this. Any ideas here?

%%
function M = CatStructFields(S, dim, j)
fields = fieldnames(S);
M=[];
for k = 1:numel(S)
    aField     = fields{j};
    M = cat(dim, M, S(k).(aField));
end
end

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

gujax 2023-6-5

Its way faster (like 7 seconds) if I dump the whole file using memmap as a uniform *uint16 into a Matlab variable and then sort out different fields.

But I ran the same using C++ and fstream where you can read a structure directly (where the struct is {identifier, header, Pix, ...}) and it was 10x faster than the uniform uint16 bit dumped version of my code.

So why is it that Matlab cannot use fread to read a structured file? Not sure how fread is implemented (does it use C++ fread or fstream?)

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Matt J 2023-6-5

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1978719-concatenate-array-of-structures-with-same-field-names#answer_1250829

编辑：Matt J 2023-6-5

So why is it that Matlab cannot use fread to read a structured file? Not sure how fread is implemented (does it use C++ fread or fstream?)

I think it may have to do with the fact that Matlab does not assume that the field data in a struct array are of a fixed size. Therefore, structs are neither stored nor read in contiguously.

9 个评论
显示 7更早的评论隐藏 7更早的评论

gujax 2023-6-6

Oh wow! Matt, your function worked like a charm. But I do not get it because I tried something similar and it led me astray.

Let me explain

When I tried this below (just like your function statement)

M2=cat(1,m.Data.Pix);

I get the following error message:

Error using memmapfile/subsref (line 790). A subscripting operation on the Data field attempted to create a comma-separated list. The memmapfile class does not support the use of comma-separated lists when subscripting.

But I copy pasted your function definition and tried this

tic

Dat = CatStructFields(m.Data,1,'Pix');

toc

And it worked (took 1.3 secs versus my original CatStructFields with for-loop which took more than half hour and now I understand whyis that based on your solution though I fail to understand based on the error message above).

Can you please explain why it works your way?

Thank you for the solution.

And I also found a simple solution which surprised me!

tic

M=zeros(65536,8000);

for jk=1:8000

M(:,jk)=m.Data(jk).Pix;

end

toc

This took 4 seconds still slow compared to yours

Matt J 2023-6-6

在 MATLAB Online 中打开

I think it's because m is not of type struct, and so the indexing syntax I was using does not apply. You could probably do,

S=m.Data;
M2=cat(1,S.Pix);

gujax 2023-6-6

yes that worked! Of course m is a filetype. The 'dot' in m.Data led me off thinking its a structure! Should have paid more attention

请先登录，再进行评论。

concatenate array of structures with same field names

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

采纳的回答

9 个评论
显示 7更早的评论隐藏 7更早的评论

更多回答（0 个）

另请参阅

类别

标签

产品

版本

Community Treasure Hunt

concatenate array of structures with same field names

1 个评论 显示 -1更早的评论隐藏 -1更早的评论

采纳的回答

9 个评论 显示 7更早的评论隐藏 7更早的评论

更多回答（0 个）

另请参阅

类别

标签

产品

版本

Community Treasure Hunt

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

9 个评论
显示 7更早的评论隐藏 7更早的评论