Index and combine datastores tables

4 次查看（过去 30 天）
Russell Shomberg
Russell Shomberg 2021-11-15
回答： Salman Ahmed 2021-11-19
I have two datastores, ds1 and ds2 consisting of large CSV files. ds1 has some bad lines and ds2 has the corrections. I need to combine the two tables, but delete the bad lines from ds1 and save in a single datastore. This simple code seems like it should work.
t1 = tall(ds1);
t2 = tall(ds2);
t3 = [t1(idx,:); t2];
write('test/out_*.csv',t3);
However, I get an error saying that write does not work with column indexing.
Evaluating tall expression using the Local MATLAB Session:
- Pass 1 of 2: 48% complete
Evaluation 21% complete
Error using tall/write (line 222)
Unable to access intermediate data in the temporary folder, most likely because it ran out of space. Clear space on the local drive, or avoid operations that reorder
tall arrays (such as SORT or indexing with a tall numeric column vector).
Caused by:
Error in adding keys and values.
Error during serialization
Any ideas on how I can get around this error?

请先登录，再回答此问题。

回答（1 个）

Salman Ahmed
Salman Ahmed 2021-11-19
Hi Russell,
To combine tall arrays with different underlying datastores, it is recommended that you write the arrays (or indexing results) to disk and then create a single new datastore referencing those locations:
files = {'folder/path/to/file1','folder/path/to/file2'};
ds = datastore(files);

类别

在 Help Center 和 File Exchange 中查找有关 Tall Arrays 的更多信息

标签

产品


版本

R2019b

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by