Performing matrix subtraction for huge data

1 次查看（过去 30 天）
MSDataSpl1
MSDataSpl1 2017-8-11
编辑： José-Luis 2017-8-15
I have a matrix say A = [2,5,7]' and another matrix B = [2,7,9]', now i want to subtract B from A in such a way like each element of B will get subtracted from all elements of A, like c = A - B then c should be like (Expected outcome) = [0 3 5, -5 -2 0, -7 -4 -2]
Its like each element of B will get deducted from all elements of A and result will be stored in row wise in new matrix row wise.I dont want to use any loops but only vectorization.So i have used bsxfun but it is good for small matrix but if the dimension of A is 230600 * 1 and B is 1400000 * 1 then bsxfun show out of memory error so how to solve this issue without using loops??
  4 个评论
显示 2更早的评论
Stephen23
Stephen23 2017-8-11
编辑：Stephen23 2017-8-11
@MSDataSpl1: what optimization? You asked about an arithmetic operation (which will be impossible for any desktop today PC to perform), not an optimization.
José-Luis
José-Luis 2017-8-11
编辑：José-Luis 2017-8-11
You could perform that operation, but a more appropriate question is: should you?
How about just performing the actual operation you need only when you need it? No need to have such a big array in memory.
When you start talking about this much data then you need a database. Loop and save to a database every now and then.
I really really doubt that you need to keep such an array in memory or in a database. I would go out of my way to avoid that. It's just a waste of disk space.

请先登录，再进行评论。

请先登录，再回答此问题。

回答（3 个）

José-Luis
José-Luis 2017-8-11
编辑：José-Luis 2017-8-11
The problem is not bsxfun, it is the sheer amount of data:
dataSize = 230600 * 1400000 * 8 (size of a double in bytes)
more or less equals
2.6e12 Bytes
2.6e9KB
2.6e6MB
2.6e3GB
That strikes me as quite a bit of data for the kind of computers that exist nowadays.
You need to divide your problem in smaller chunks. Better yet, you need to carefully evaluate if there are better ways of achieving what you are trying to do.
  4 个评论
显示 2更早的评论
MSDataSpl1
MSDataSpl1 2017-8-11
@John & @Jose could you answer this one,
https://in.mathworks.com/matlabcentral/answers/351074-i-want-to-find-the-minimum-value-of-the-matrices-so-i-am-using-min-function-but-the-problem-is-the-d
José-Luis
José-Luis 2017-8-15
编辑：José-Luis 2017-8-15
I had a look at that question. Still not clear to me what you want to achieve. Please learn to format your post. If it's hard to read, people might be less willing to help.
Please accept the answer that best solves your problem.

请先登录，再进行评论。

Jan
Jan 2017-8-11
Explain, what the actual problem is you are working on. It is not likely that you really need to create this output matrix explicitly, because it is such easy to access the output dynamically: The element at the index [n,m] is A(n) - B(m). Then you can and should omit the creation of C = A.' - B. Where ever you want to write C(n,m) write A(n) - B(m) instead.
I'm sure there will be further methods to reduce the need to create C explicitly. If you want to search the minimum only, or what ever. So please explain the mathematical problem you want to solve.
  1 个评论
MSDataSpl1
MSDataSpl1 2017-8-11
@Jan Simon I am trying to take the difference of the time stamps successive traces(messages) which are coming in a particular time gap so like in 1 hr 1500000 messages comes so i want to take the difference of the timestamps and then calculate the minimum and plot the data. like x = trace(i) - trace(i-1) where i = 1500000.
or
you can answer my this question then it will also help,
https://in.mathworks.com/matlabcentral/answers/351074-i-want-to-find-the-minimum-value-of-the-matrices-so-i-am-using-min-function-but-the-problem-is-the-d

请先登录，再进行评论。

alice
alice 2017-8-11
You can avoid to loop using the minus operator comportment:
A = [2,5,7]';
B = [2,7,9]';
result = A'-B;
  1 个评论
José-Luis
José-Luis 2017-8-11
And that still won't solve the problem even if OP is sitting on top of a Google data center.

请先登录，再进行评论。

类别

在 Help Center 和 File Exchange 中查找有关 Logical 的更多信息

标签

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by