MapReduce cannot find dataset

1 次查看（过去 30 天）
AlessioX
AlessioX 2015-12-31
回答： AlessioX 2016-2-8
Hello there,
I was using MapReduce to elaborate a (relatively) large dataset, which is a simple matrix stored in a .csv format (~250MB). I am running this MapReduce in a home-made cluster with 2 computers.
The cluster is working fine, no problems whatsoever.
As always I create the datastore from the csv file
ds=datastore({'DS.csv'},'ReadVariableNames',false);
I open the pool and set the MapReduce environment
myCluster=parpool('HomeCluster');
MRE=mapreducer(myCluster);
I start the procedure and the following error appears:
Error using matlab.io.datastore.TabularTextDatastore/partition (line 44)
Cannot find files or folders matching: '<path for CSV>'
Weird thing is...the csv file is inside the current folder, which also contains my scripts/functions.
I also tried attaching that file to the pool (even if sounds pretty stupid to me, correct me if I'm wrong), but still no luck.
Any help is appreciated. Thanks!
Update: I was able to run this program by creating a folder on both computers with the same name with the same path and copying the csv file in both folders. But that's rather stupid, even because most of the times is impossible to create a perfectly matching path name (e.g. the user name in the operating system might be different). Is there any smarter way to avoid this? Can the master node be the only one with the dataset file in it?

请先登录，再回答此问题。

采纳的回答

AlessioX
AlessioX 2016-2-8
How did I actually solved it: by attaching such files to the pool didn't work out, so I had to share via network an external hard drive in order to have these files in the same path/folder.

更多回答（0 个）

类别

在 Help Center 和 File Exchange 中查找有关 MapReduce 的更多信息

标签

产品

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by