How to speed up making large matrices, and reduce memory usage

4 次查看（过去 30 天）
In-chan Kim
In-chan Kim 2020-5-6
评论： In-chan Kim 2020-5-6
How can I speed up making a large matrix, and also cut down memory usage?
I need to make a matrix which turns out to be in the order of 2,000 by 30,000 ish.
I can run it, but it'd be great to be able to speed it up, and also reduce memory use.
At the moment my code is as below, there length of C is 1800.
Creating Aineq is my final aim!
Aineq=[ones(length(C))]; %size of matrix to represent size one of the decision variables Q or G
Aineq1=[tril(Aineq),-tril(Aineq),tril(Aineq),-tril(Aineq),Aineq*0,Aineq*0,Aineq*0,zeros((length(C)),length(F))]; %1 matrix for first DV, then the next DV
Aineq2=[-tril(Aineq),tril(Aineq),-tril(Aineq),tril(Aineq),Aineq*0,Aineq*0,Aineq*0,zeros((length(C)),length(F))]; %2
Aineq3=[eye(length(C)),eye(length(C)),Aineq*0,Aineq*0,Aineq*0,Aineq*0,Aineq*0,zeros((length(C)),length(F))]; %3
Aineq4=[tril(Aineq,-1)*-1,tril(Aineq),tril(Aineq,-1)*-1,tril(Aineq,-1),Aineq*0,Aineq*0,Aineq*0,zeros((length(C)),length(F))]; %4
Aineq5=[Aineq*0,Aineq*0,Aineq*0,eye(length(C)),Aineq*0,Aineq*0,Aineq*0,zeros((length(C)),length(F))]; %5
Aineq6=[Aineq*0,Aineq*0,tril(Aineq,-1)*-1,tril(Aineq,-1),tril(Aineq,-1),tril(Aineq,-1)*-1,tril(Aineq,-1)*-1,zeros((length(C)),length(F))]; %6
Aineq7=[Aineq*0,Aineq*0,tril(Aineq)*-1,tril(Aineq),tril(Aineq),tril(Aineq)*-1,tril(Aineq)*-1,zeros((length(C)),length(F))]; %7
Aineq8=[Aineq*0,Aineq*0,tril(Aineq,-1),tril(Aineq,-1)*-1,tril(Aineq,-1)*-1,tril(Aineq,-1),tril(Aineq,-1),zeros((length(C)),length(F))]; %8
Aineq9=[Aineq*0,Aineq*0,tril(Aineq),tril(Aineq)*-1,tril(Aineq)*-1,tril(Aineq),tril(Aineq),zeros((length(C)),length(F))];%9
Aineq10=[Aineq*0,Aineq*0,eye(length(C)),Aineq*0,Aineq*0,Aineq*0,Aineq*0,zeros((length(C)),length(F))]; %10
Aineq11=[Aineq*0,Aineq*0,Aineq*0,eye(length(C)),Aineq*0,Aineq*0,Aineq*0,zeros((length(C)),length(F))]; %11
Aineq12=[Aineq*0,Aineq*0,Aineq*0,Aineq*0,Aineq*0,eye(length(C)),Aineq*0,zeros((length(C)),length(F))]; %12
Aineq13=[Aineq*0,Aineq*0,Aineq*0,Aineq*0,eye(length(C)),Aineq*0,Aineq*0,zeros((length(C)),length(F))]; %13
Aineq14=[Aineq*0,Aineq*0,Aineq*0,Aineq*0,Aineq*0,Aineq*0,eye(length(C)),zeros((length(C)),length(F))]; %14
Aineq15=[Aineq*0,Aineq*0,Aineq*0,Aineq*0,Aineq*0,Aineq*0,eye(length(C))*-1,zeros((length(C)),length(F))]; %15
clear Aineq
% Aineq=[Aineq1;Aineq2;Aineq3;Aineq4;Aineq5;Aineq6;Aineq7;Aineq8;Aineq9;Aineq10;Aineq11;Aineq12;Aineq13;Aineq14;Aineq15];
Aineq=[Aineq1]; clear Aineq1
Aineq=[Aineq;Aineq2]; clear Aineq2
Aineq=[Aineq;Aineq3]; clear Aineq3
Aineq=[Aineq;Aineq4]; clear Aineq4
Aineq=[Aineq;Aineq5]; clear Aineq5
Aineq=[Aineq;Aineq6]; clear Aineq6
Aineq=[Aineq;Aineq7]; clear Aineq7
Aineq=[Aineq;Aineq8]; clear Aineq8
Aineq=[Aineq;Aineq9]; clear Aineq9
Aineq=[Aineq;Aineq10]; clear Aineq10
Aineq=[Aineq;Aineq11]; clear Aineq11
Aineq=[Aineq;Aineq12]; clear Aineq12
Aineq=[Aineq;Aineq13]; clear Aineq13
Aineq=[Aineq;Aineq14]; clear Aineq14
Aineq=[Aineq;Aineq15]; clear Aineq15

请先登录，再回答此问题。

采纳的回答

Bjorn Gustavsson
Bjorn Gustavsson 2020-5-6
This looks like a very sparse matrix, right? Then I would first figure out how to assign the blocks of Aineq from its components, something like this:
blksz = [3,13]; % or whatever they become
nBlks = 15;
Aineq = zeros(blksz*nBlks);
for i1 = 1:nBlks
for i2 = 1:nBlks
% Some logics here for selecting which block to fill Aineq with
Aineq((1:blksz(1))+blksz*(i1-1),(1:blksz(2))+blksz(2)*(i2-1)) = CurrBlk;
end
end
Once you've figured that one out, you should look at what components of every block is non-zero and start to build sparse blocks, and then concatenate them instead, that would be much more memmory efficient. Then the last step (or preferably the firts, but I find it easier to move to sparse representations of comlex matrices step-by-step), is to calculate vectors with the row and column-indices and the corresponding matrix element and then simply concatenate those into three arrays i_rows, i_cols, val and then call sparse:
Asparse = sparse(i_rows,i_cols,vals,nRows,nCols);
HTH
  8 个评论
显示 6更早的评论
In-chan Kim
In-chan Kim 2020-5-6
Thanks everyone, real good outputs! About 3 times less memory and faster!
Name Size Bytes Class Attributes
A 27003x12603 2722550472 double
spr 27003x12603 829912320 double sparse
Elapsed time is 4.461101 seconds.
Elapsed time is 15.440768 seconds.
In-chan Kim
In-chan Kim 2020-5-6
Is there any way to see how much memory is being used by a code being run? And which parts are taking up a lot of memory?
I know with profiler you can do that for the time it takes.
Also, I thought I'd share what code I went with
Aineqinit=zeros(15*length(C)+length(F),7*length(C)+length(F)); % full size of Aineq based on the make up of the 16 constraints
Aineqbase=[ones(length(C))];%size of matrix to represent size one of the decision variables Q or G
Aineqinit(1:length(C),1:7*length(C)+length(F))=[tril(Aineqbase),-tril(Aineqbase),tril(Aineqbase),-tril(Aineqbase),Aineqbase*0,Aineqbase*0,Aineqbase*0,zeros((length(C)),length(F))]; %1 matrix for first DV, then the next DV
Aineqinit(1*length(C)+1:2*length(C),1:7*length(C)+length(F))=[-tril(Aineqbase),tril(Aineqbase),-tril(Aineqbase),tril(Aineqbase),Aineqbase*0,Aineqbase*0,Aineqbase*0,zeros((length(C)),length(F))]; %2
Aineqinit(2*length(C)+1:3*length(C),1:7*length(C)+length(F))=[eye(length(C)),eye(length(C)),Aineqbase*0,Aineqbase*0,Aineqbase*0,Aineqbase*0,Aineqbase*0,zeros((length(C)),length(F))]; %3
Aineqinit(3*length(C)+1:4*length(C),1:7*length(C)+length(F))=[tril(Aineqbase,-1)*-1,tril(Aineqbase),tril(Aineqbase,-1)*-1,tril(Aineqbase,-1),Aineqbase*0,Aineqbase*0,Aineqbase*0,zeros((length(C)),length(F))]; %4
Aineqinit(4*length(C)+1:5*length(C),1:7*length(C)+length(F))=[Aineqbase*0,Aineqbase*0,Aineqbase*0,eye(length(C)),Aineqbase*0,Aineqbase*0,Aineqbase*0,zeros((length(C)),length(F))]; %5
Aineqinit(5*length(C)+1:6*length(C),1:7*length(C)+length(F))=[Aineqbase*0,Aineqbase*0,tril(Aineqbase,-1)*-1,tril(Aineqbase,-1),tril(Aineqbase,-1),tril(Aineqbase,-1)*-1,tril(Aineqbase,-1)*-1,zeros((length(C)),length(F))]; %6
Aineqinit(6*length(C)+1:7*length(C),1:7*length(C)+length(F))=[Aineqbase*0,Aineqbase*0,tril(Aineqbase)*-1,tril(Aineqbase),tril(Aineqbase),tril(Aineqbase)*-1,tril(Aineqbase)*-1,zeros((length(C)),length(F))]; %7
Aineqinit(7*length(C)+1:8*length(C),1:7*length(C)+length(F))=[Aineqbase*0,Aineqbase*0,tril(Aineqbase,-1),tril(Aineqbase,-1)*-1,tril(Aineqbase,-1)*-1,tril(Aineqbase,-1),tril(Aineqbase,-1),zeros((length(C)),length(F))]; %8
Aineqinit(8*length(C)+1:9*length(C),1:7*length(C)+length(F))=[Aineqbase*0,Aineqbase*0,tril(Aineqbase),tril(Aineqbase)*-1,tril(Aineqbase)*-1,tril(Aineqbase),tril(Aineqbase),zeros((length(C)),length(F))];%9
Aineqinit(9*length(C)+1:10*length(C),1:7*length(C)+length(F))=[Aineqbase*0,Aineqbase*0,eye(length(C)),Aineqbase*0,Aineqbase*0,Aineqbase*0,Aineqbase*0,zeros((length(C)),length(F))]; %10
Aineqinit(10*length(C)+1:11*length(C),1:7*length(C)+length(F))=[Aineqbase*0,Aineqbase*0,Aineqbase*0,eye(length(C)),Aineqbase*0,Aineqbase*0,Aineqbase*0,zeros((length(C)),length(F))]; %11
Aineqinit(11*length(C)+1:12*length(C),1:7*length(C)+length(F))=[Aineqbase*0,Aineqbase*0,Aineqbase*0,Aineqbase*0,Aineqbase*0,eye(length(C)),Aineqbase*0,zeros((length(C)),length(F))]; %12
Aineqinit(12*length(C)+1:13*length(C),1:7*length(C)+length(F))=[Aineqbase*0,Aineqbase*0,Aineqbase*0,Aineqbase*0,eye(length(C)),Aineqbase*0,Aineqbase*0,zeros((length(C)),length(F))]; %13
Aineqinit(13*length(C)+1:14*length(C),1:7*length(C)+length(F))=[Aineqbase*0,Aineqbase*0,Aineqbase*0,Aineqbase*0,Aineqbase*0,Aineqbase*0,eye(length(C)),zeros((length(C)),length(F))]; %14
Aineqinit(14*length(C)+1:15*length(C),1:7*length(C)+length(F))=[Aineqbase*0,Aineqbase*0,Aineqbase*0,Aineqbase*0,Aineqbase*0,Aineqbase*0,eye(length(C))*-1,zeros((length(C)),length(F))]; %15
% formulate LHS of constraint 16
u = zeros((numel(stcounter)+1)/2, sum(stcounter));
for i = 1:2:numel(stcounter)
u(((i+1)/2), :) = repelem([0, 1, 0], [sum(stcounter(1:i-1)), stcounter(i), sum(stcounter(i+1:end))]);
end
Aineqinit(15*length(C)+1:15*length(C)+length(F),1:7*length(C)+length(F))=[u,zeros(length(F),length(C)),zeros(length(F),length(C)),zeros(length(F),length(C)),zeros(length(F),length(C)),zeros(length(F),length(C)),zeros(length(F),length(C)),eye(length(F))*-1*dtih.*PEVmaxstay.*staycounter]; %16
Aineq=sparse(Aineqinit);

请先登录，再进行评论。

更多回答（0 个）

类别

在 Help Center 和 File Exchange 中查找有关 Sparse Matrices 的更多信息

标签

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by