Performing matrix subtraction for huge data
显示 更早的评论
I have a matrix say A = [2,5,7]' and another matrix B = [2,7,9]', now i want to subtract B from A in such a way like each element of B will get subtracted from all elements of A, like c = A - B then c should be like (Expected outcome) = [0 3 5, -5 -2 0, -7 -4 -2]
Its like each element of B will get deducted from all elements of A and result will be stored in row wise in new matrix row wise.I dont want to use any loops but only vectorization.So i have used bsxfun but it is good for small matrix but if the dimension of A is 230600 * 1 and B is 1400000 * 1 then bsxfun show out of memory error so how to solve this issue without using loops??
4 个评论
Your final matrix would have 230600*1400000 = 3.2284e11 elements in it, and would require atleast 230600*1400000*8 = 2.5827e+012 = 2.349 tebibytes of memory just for MATLAB to be able to have it in the workspace. Do you have that much memory in your computer?
MSDataSpl1
2017-8-11
You could perform that operation, but a more appropriate question is: should you?
How about just performing the actual operation you need only when you need it? No need to have such a big array in memory.
When you start talking about this much data then you need a database. Loop and save to a database every now and then.
I really really doubt that you need to keep such an array in memory or in a database. I would go out of my way to avoid that. It's just a waste of disk space.
回答（3 个）
The problem is not bsxfun, it is the sheer amount of data:
dataSize = 230600 * 1400000 * 8 (size of a double in bytes)
more or less equals
2.6e12 Bytes
2.6e9KB
2.6e6MB
2.6e3GB
That strikes me as quite a bit of data for the kind of computers that exist nowadays.
You need to divide your problem in smaller chunks. Better yet, you need to carefully evaluate if there are better ways of achieving what you are trying to do.
4 个评论
John D'Errico
2017-8-11
编辑：John D'Errico
2017-8-11
+1. Regardless, tomorrow, or next week, month or year, if and when someone gets a large enough computer to solve the current problem, then they will decide they need to to solve a problem that is 1000 times as large.
John's law: Computational problems always expand to be larger than the capabilities of the available computing resources.
The art of computing and of mathematics in general is to solve intractable problem not by brute force, but by more subtle means. There will always be many who will choose to use brute force anyway. :(
José-Luis
2017-8-11
I like John's law. :)
MSDataSpl1
2017-8-11
Jan
2017-8-11
1 个投票
Explain, what the actual problem is you are working on. It is not likely that you really need to create this output matrix explicitly, because it is such easy to access the output dynamically: The element at the index [n,m] is A(n) - B(m). Then you can and should omit the creation of C = A.' - B. Where ever you want to write C(n,m) write A(n) - B(m) instead.
I'm sure there will be further methods to reduce the need to create C explicitly. If you want to search the minimum only, or what ever. So please explain the mathematical problem you want to solve.
类别
在 帮助中心 和 File Exchange 中查找有关 Loops and Conditional Statements 的更多信息
2017-8-11
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!
Translated by 
