Building tall table from tall arrays generates error

Question

Harry Cho 2023-3-14

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1928735-building-tall-table-from-tall-arrays-generates-error

评论： Harry Cho 2023-3-15

data.csv

clear
dataFile = 'data.csv';
ds = tabularTextDatastore(dataFile, FileExtensions='.csv');
ds.ReadVariableNames = true;
ds.Delimiter = ',';
ds.SelectedVariableNames = ["hash", "count"];
ds.SelectedFormats = {'%s', '%f'};
data = tall(ds);
Starting parallel pool (parpool) using the 'Processes' profile ...
Connected to the parallel pool (number of workers: 2).
[g, THash] = findgroups(data.hash);
TCount = splitapply(@(x) {x}, data.count, g);
%% This works but cannot use it because actual data file is far larger than memory
hash = gather(THash);
Evaluating tall expression using the Parallel Pool 'Processes':
- Pass 1 of 1: 0% complete
- Pass 1 of 1: 100% complete
- Pass 1 of 1: Completed in 1.9 sec
Evaluation completed in 2.8 sec
count = gather(TCount);
Evaluating tall expression using the Parallel Pool 'Processes':
- Pass 1 of 3: 0% complete
- Pass 1 of 3: 100% complete
- Pass 1 of 3: Completed in 0.54 sec
- Pass 2 of 3: 0% complete
- Pass 2 of 3: 100% complete
- Pass 2 of 3: Completed in 0.46 sec
- Pass 3 of 3: 0% complete
- Pass 3 of 3: 100% complete
- Pass 3 of 3: Completed in 0.58 sec
Evaluation completed in 2.3 sec
T1 = table(hash, count);
%% This is the intended code but doesn't work
TT = table(THash,TCount);
Error using tall/table
Incompatible non-scalar tall array arguments. Each of the tall arrays must be the same size in the first dimension, must be derived from a single tall array, and must not have been indexed
differently in the first dimension (indexing operations include functions such as VERTCAT, SPLITAPPLY, SORT, CELL2MAT, SYNCHRONIZE, RETIME and so on).
write(fullfile(pwd,'data'),TT,FileType="parquet");

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Oguz Kaan Hancioglu 2023-3-15

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1928735-building-tall-table-from-tall-arrays-generates-error#answer_1193410

在 MATLAB Online 中打开

Your code wasn't work because "gather(TCount)" returns cell array for each element. Therefore you are trying to write double array in to one single cell. You can find the length of each array into the cell. I hope this solves your problem.

%% This works but cannot use it because actual data file is far larger than memory
hash = gather(THash);
count = gather(TCount);
cellsz = cellfun(@size,count,'uni',false);
newCount = cellfun(@(x) x(1),cellsz,'UniformOutput',false)
T1 = table(hash, newCount);

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

Harry Cho 2023-3-15

Thank you for the reply. Unfortunately I have to collect cell array, in which each cell has different length of double array. My question is why it works in-memory table T1 but not in tall table TT.

请先登录，再进行评论。