How to deal with a single tsv-file whose size is out of the memory?

Question

ziyou teng 2020-10-17

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/616938-how-to-deal-with-a-single-tsv-file-whose-size-is-out-of-the-memory

评论： ziyou teng 2020-11-10

Hi, guys.

I have a tsv file whose size is 20GB while my pc's memory is only 16GB.

When I read the file, it always shows errors.

I tried tall array as follows but fails due to tsv file cannot be recognized.

So I am looking for advice.

ds = tabularTextDatastore('D:\database1\scipatlinkage\paperauthoridaffiliationname.tsv');
ds.TreatAsMissing = 'NA';
ds.SelectedVariableNames = {'paperid','authorid','affiliationame'};
ds.SelectedFormats(2:3) = {'%s','%s'};
pre = preview(ds)
My matlab is R2020a

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

Athul Prakash 2020-10-24

What is the exact error you're getting? If it s a File Not Found issue, you may use the 'isfile' function to confirm the existence of your file before running the script. See this doc for 'isfile': https://in.mathworks.com/help/matlab/ref/isfile.html

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Athul Prakash 2020-10-24

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/616938-how-to-deal-with-a-single-tsv-file-whose-size-is-out-of-the-memory#answer_523598

I have noticed that you have not created the tall array in the code attached. Perhaps . . .

t = tall(ds);

might be missing.

.As alternatives to tall array, you may try using this datastore with 'mapreduce'. See this doc:

https://in.mathworks.com/help/matlab/mapreduce.html

Alternatively, you may also write your own code to 'read' from the datastore iteratively and process the data in chunks that can fit on your RAM.

Hope it Helps!

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

ziyou teng 2020-11-10

Thank you！It‘s really helpful. My problem is the file is too large that goes out of my pc's memory. Your answers have gaven me useful hints on that. Thanks again!

请先登录，再进行评论。