How can I get the indices of values meeting a condition in a tall array?

Question

Eva-Maria Weiss 2019-8-1

1
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/474439-how-can-i-get-the-indices-of-values-meeting-a-condition-in-a-tall-array

评论： Eva-Maria Weiss 2019-8-4

Hi everybody,

I have a tall array containing voltage data. I want to thresholding the trace to receive the signals. So with thresh is the threshold, data is the tall double array (Mx1), I perform

data = abs(data);
loc = tall(data > thresh);

to get loc, a tall logical, containing ones for samples above threshold and zeros for samples below, after evaluating.

I just want to have the crossings, so I apply

location = diff(loc,1,1);

So locations where the signal crosses the threshold are 1, all others are 0 or -1;

But now I need a vector listing the position values (indices), and i use

starts = find(gather(location == 1));

Is there any possibility to realize that without gather to leave the array unevaluated?

Many thanks in advance!!!!

Eva

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

madhan ravi 2019-8-2

编辑：madhan ravi 2019-8-2

在 MATLAB Online 中打开

Eva-Maria Weiss's answer moved here:

I found that in the MATLAB documentation:

https://de.mathworks.com/help/matlab/import_export/index-and-view-tall-array-elements.html

"The find function locates nonzero elements in tall column vectors, and can be useful to generate a vector of indices for elements that meet particular conditions. For example, k = find(X<0) returns the linear indices for all negative elements in X."

So I tried to convert a tall variable (location) of type tall double (unevaluated) to a Mx1 tall double (unevaluated) - to get a column vector,

location = location(:,1);

but still I got the error message

"undefined function 'find' for input arguments of type'tall' "

Isn't oit a column vector? How can I convert my variable to a column vector thats's a real one?

I really appreciate any help or hint!

Thank you very much in advance!

Eva

-----

Maybe find is not working on tall since I use 2017b?

So far I come up with some proceedings:

I aim to solve the issue by logical indexing;

so first I create this tall lgocal vector containing the crossings

loc = data(data > thresh); 
location = diff(loc,1,1);
location = location == 1;

Then I creat a tall column vector containing indices:

index = location;
index(:,1) = tall([1:1:numbSamples -1]');

Then I select indices meeting the condition

starts = index(location,1);
starts(starts == 0) = [];

To see if that approach is working, I want evaluate the result by using

starts(starts == 0) = [];

Evaluating starts properly, but after finishing pass 1 of 2, I was again left with an error message saying:

Incompatible tall array arguments. The tall arrays must be based on the same datastore and

be indexed in the first dimension using the same logical index vector.

Learn more about errors encountered during GATHER.

How can I overcome that? Could anyone please help, meanwhile I really feel like circling around!

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Guillaume 2019-8-2

2
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/474439-how-can-i-get-the-indices-of-values-meeting-a-condition-in-a-tall-array#answer_385916

编辑：Guillaume 2019-8-2

在 MATLAB Online 中打开

Maybe find is not working on tall since I use 2017b

Indeed, support for using find on tall arrays was added in R2018a.

As for your error with your workaround, there's a big difference between

[1:1:numbSamples -1]

and

[1:1:numbSamples-1]  
%or
[1:1:numbSamples - 1]

The former is equivalent to

[1:1:numbSamples, -1] %concatenation! not subtraction

and will therefore have numbSamples + 1 elements when you meant to have numbSamples - 1 elements. So, of course, the length doesn't match that of your logical array and you get an error.

To avoid this sort of issue:

be consistent with your spacing. Either no space at all on both sides of the operator, or spaces on both sides.
Don't use [] when you don't mean concatenation. If you want to transpose a vector surround it with (). This is (negligibly) faster to process as well.

So:

index(:,1) = tall((1:numbSamples-1)');

One potential problem with your workaround is that if numbSamples is sufficiently large, the temporary non-tall array 1:numbSamples-1 may not fit in memory.

5 个评论
显示 3更早的评论隐藏 3更早的评论

Guillaume 2019-8-3

在 MATLAB Online 中打开

Indeed, I see the same issue in the latest version of matlab.

I don't know enough about tall arrays to explain why it errors and the error would appear to contradict the documentation so I'll raise a bug report with Mathworks to get an explanation,

Digging a bit in the tall array code, the incompatibitly is caused because location stems from a Mx1 tall array (even thought it is now (M-1)x1) whereas index starts as as (M-1)x1 array. That's enough to cause an error, but I don't know why.

So, a workaround is to create |index as a Mx1 tall array and then reduce ot to (M-1)x1. The following works:

assert(numbSamples <= 134217728, 'flintmax overflow, see explanation below');
index = diff(tall(cumsum(0:numbSamples-1)'), 1, 1);
starts = index(location);

The major limitation with that is that if numbSamples is greater than 134,217,728, the cumulative sum will go over flintmax and you won't be storing exact numbers anymore.

If numbSamples is greater than that, you can use 64-bit integers instead to push the limit to 6,074,000,999:

assert(numbSamples <= 6074000999, 'this is going to use too much memory! (and it will overflow uint64)'
index = diff(tall(cumsum(uint64(0:0:numbSamples-1)'), 1, 1);
starts = index(location);

Since it would take about 45 GB of memory to store the temporary non-tall array with 6,074,000,999 elements, I don't think that limit is a problem.

Guillaume 2019-8-4

Indeed, I initially tried with in-memory tall arrays and my workaround worked fine for that. Having no tried with a datastore backed tall array, the workaround doesn't work.

There appears to be some restrictions on logical indexing that are not documented. Unfortunately, this is beyond my knowledge of tall arrays, so I don't think I can help further. If you can, I suggest you raise a support request with mathworks (or upgrade to 2018a so you can use find).

Note that I've raised my own bug report to at least get the documentation explaining why this does not work.

Eva-Maria Weiss 2019-8-4

Well, my next try is to expand the datastore with this index vector. It's far away from being elegant, since you have to create tall arrays, then write them back to disk and then again create new tallDatastores and tall arrays.

And 'write' takes a time, too. But my guess is, that I will just do it one time per experiment. Evaluation 'location' will be for each electrode...380 times per experiment.

But still I don't know if it's working at all. So lets see, I will report.

And yes, I guess I should also contact the matlab support. Version 2018 would be great, but I'm lmited to what I can get from university :).

Thank you again for your help and ideas!

Eva

请先登录，再进行评论。

How can I get the indices of values meeting a condition in a tall array?

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

回答（1 个）

5 个评论
显示 3更早的评论隐藏 3更早的评论

另请参阅

类别

标签

产品

版本

Community Treasure Hunt

How can I get the indices of values meeting a condition in a tall array?

1 个评论 显示 -1更早的评论隐藏 -1更早的评论

回答（1 个）

5 个评论 显示 3更早的评论隐藏 3更早的评论

另请参阅

类别

标签

产品

版本

Community Treasure Hunt

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

5 个评论
显示 3更早的评论隐藏 3更早的评论