TabularTextDataStore wont store more than 20000 rows

Question

SkarletSam 2020-10-21

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/621923-tabulartextdatastore-wont-store-more-than-20000-rows

评论： Walter Roberson 2020-10-21

Im trying to import a large csv file (60,000 x 171). tabularTextDataStore is only storing up to 20,000 rows. Is there a more efficient way to import all of the data?

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Kevin Gurney 2020-10-21

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/621923-tabulartextdatastore-wont-store-more-than-20000-rows#answer_520818

在 MATLAB Online 中打开

The amount of data returned on each call to the read method of a TabularTextDatastore is controlled by the value of the ReadSize property.

If you want to read more than 20,000 rows on each call to read (20,000 is the default value), then you can adjust the value of ReadSize like so:

trainSet = tabularTextDatastore("data\trainingSet.csv");
trainSet.ReadSize = 60000; % You can adjust this value to your liking.
trainSet.TreatAsMissing = 'na';
trainSetR = read(trainSet)

2 个评论
显示无隐藏无

SkarletSam 2020-10-21

It seems like TabularTextDatastore's read size has a max value of 45140

Walter Roberson 2020-10-21

There seems to be an internal BUFFER_UPPERLIMIT property that is set to 32 megabytes, with 64 megabytes used for HADOOP . I do not see any method to set that property.

请先登录，再进行评论。