Anonymous Functions and MapReduce

1 次查看(过去 30 天)
Hello there,
I have successfully created a home-made cluster with a couple of Macbooks. Everything seems alright, I've been doing a few tests and such. Now I want to run a MapReduce script which is fully working on my 'local' cluster profile and I have been using some anonymous functions for the Mapper and the Reducer because I need to handle several variables within the tasks.
I change the parallel pool from 'local' to 'homecluster', I've also set the Mapreducer environment but the following errors appear
Error using mapreduce (line 101)
An error occurred building a parallel.pool.Constant.
Error in SelectInitialCentroids (line 24)
outDataStore=mapreduce(DataStore,mapper,reducer,MapReduceEnv);
Error in main (line 35)
centroids=SelectInitialCentroids(DS,K,MapReduceEnv);
Caused by:
An error occurred interpreting function call.
An error occurred interpreting function call.
Where 'mapper' and 'reducer' are my anonymous functions and 'MapReduceEnv' is the environment.
I think there might be a problem with my anonymous functions, although it works fine with the 'local' profile.
Any ideas? Thank you very much indeed.
PS: running Matlab R2015b on Mac OS X El Capitain
Update: the problem is indeed with the anonymous functions. I was able to run MapReduce tasks in my home cluster with Map and Reduce functions declared standard (i.e. not anonymous).

采纳的回答

AlessioX
AlessioX 2015-12-31
Allright. I guess I figured this out.
When opening the parallel pool, you must specify all the files with such functions: that file(s) must be available to all the workers/nodes in the pool:
myCluster=parpool('HomeCluster','AttachedFiles',{'file1.m'....'fileN.m'});
and then force the MapReduce Environment to that cluster (as in the original topic)
MapReduceEnv=mapreducer(myCluster);
Every time you call the MapReduce tasks, also, that environment must be specified (as in the original topic)
outDataStore=mapreduce(DataStore,mapper,reducer,MapReduceEnv);

更多回答(0 个)

类别

Help CenterFile Exchange 中查找有关 MapReduce 的更多信息

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by