Reduce function if concatenated intermediate results doesn't fit in memory

1 次查看（过去 30 天）
Peng Li
Peng Li 2020-5-9
评论： Peng Li 2020-5-11
I'm trying to decode a huge dataset of rougly 500k*10k size. What I'm doing is actually load the file into a datastore, replace the tall table variable names with a meaningful name, and decode the content of all columns with a meaningful sentences in my dictionary.
I have some wiered issues that didn't let my program go through using a tall table. So I'm now trying to use mapreduce.
I find an example here
https://www.mathworks.com/help/matlab/import_export/simple-data-subsetting-using-mapreduce.html
function subsettingReducer(~, intermValList, outKVStore)
% get all intermediate results from the list
outVal = {};
while hasnext(intermValList)
outVal = [outVal; getnext(intermValList)];
end
% Note that this approach assumes the concatenated intermediate values (the
% subset of the whole data) fit in memory.
add(outKVStore, 'Null', outVal);
end
My question is, in this example, if the contanenated intermediate results doesn't fit in memory, is there a better workaround for this?
Thanks!

请先登录，再回答此问题。

回答（0 个）

类别

在 Help Center 和 File Exchange 中查找有关 MapReduce 的更多信息

标签

产品


版本

R2020a

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by