Reduce function if concatenated intermediate results doesn't fit in memory

2 次查看（过去 30 天）

Peng Li 2020-5-9

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/524196-reduce-function-if-concatenated-intermediate-results-doesn-t-fit-in-memory

评论： Peng Li 2020-5-11

I'm trying to decode a huge dataset of rougly 500k*10k size. What I'm doing is actually load the file into a datastore, replace the tall table variable names with a meaningful name, and decode the content of all columns with a meaningful sentences in my dictionary.

I have some wiered issues that didn't let my program go through using a tall table. So I'm now trying to use mapreduce.

I find an example here

https://www.mathworks.com/help/matlab/import_export/simple-data-subsetting-using-mapreduce.html

function subsettingReducer(~, intermValList, outKVStore)
  % get all intermediate results from the list
  outVal = {};
  while hasnext(intermValList)
    outVal = [outVal; getnext(intermValList)];
  end
  % Note that this approach assumes the concatenated intermediate values (the
  % subset of the whole data) fit in memory.
    
  add(outKVStore, 'Null', outVal);
end