problems when adding a column to tall array

Question

Benjamin Imbach 2017-8-28

1
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/354300-problems-when-adding-a-column-to-tall-array

回答： Josh Meyer 2017-8-28

I'm trying to add a new column to an already existing tall array.

The tall array is based on a datastore.

DS = datastore('Matlab_DS');
Table = tall(DS);

I gather a column of the table, do some calculations and save the results in a new cell array.

Column1 = gather(Table.Column1);   % 4975885x1 cell
ColumnNew = SomeFunction(Column1); % 4975885x1 cell (fully evaluated at this point)

Then I'm trying to add the newly calculated column to the original table.

Table.ColumnNew = tall(ColumnNew);

The following error message appears when I try to gather the Table:

Incompatible tall array arguments. The first dimension in each tall array must have the same size, and each tall array must be based on the same
datastore.

Thinking it is a problem with the datastore I try to concatenate a new tall array

Table2 = [Table tall(ColumnNew)];

When I gather Table2 now, a slightly different error message appears:

Incompatible tall array arguments. The first dimension in each tall array must have the same size, or have a size of 1.

I can gather ColumnNew, and it shows the same number of rows as the Table, so that shouldn't be a problem.

However, this works:

Table2 = [gather(Table.Column1) ColumnNew];

If I gather the whole Table first and then concatenate the new column it seems to work. However, the full Table is too big to fit into memory. It is not possible for me to gather the full table...

Am I doing something wrong, is this some kind of bug, or is it simply not possible to add a new column to a tall table this way?

My MATLAB release: 2016b

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Josh Meyer 2017-8-28

2
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/354300-problems-when-adding-a-column-to-tall-array#answer_279459

In the first case the reason you get an error is because you are attempting to add a tall variable to the table that was created from an in-memory array. Instead of gathering, working on in-memory data, then converting back to tall and trying to append to the table, you can simply do

Table.ColumnNew = SomeFunction(Table.Column1)

For example:

ds = datastore('airlinesmall.csv','TreatAsMissing','NA');
ds.SelectedVariableNames = 'ArrDelay';
t = tall(ds)
t.Sine= sin(t.ArrDelay) %works
t.Sine2= tall(gather(sin(t.ArrDelay))) %doesn't work