- data is passed as a 1200000 x 1 column vector
- you create lambda as a 1 x 1200000 row vector
- data.*lambda will then do implicit expansion to 1200000 x 1200000 because the two arrays have compatible sizes (either the same or 1, in this case 1), repeating the columns in data and the rows in lambda and multiplying the two matrices element-wise.
- The implicit expansion fails because MATLAB has built-in protection against creating arrays that are too large.
mle memory error with custom negative log-likelihood function
1 次查看(过去 30 天)
显示 更早的评论
I am trying to use 'mle' with a custom negative log-likelihood function, but I get the following error:
Requested 1200000x1200000 (10728.8GB) array exceeds maximum array size preference (15.6GB). This might cause MATLAB to become unresponsive.
The data I am usng is a 1x1200000 binary array (which I had to convert to double), and the function has 10 arguments: one for the data, 3 known paramenters, and 6 to be optimized. I tried setting 'OptimFun' to both 'fminsearch' and 'fmincon'. Also, optimizing the parameters using 'fminsearch' and 'fminunc' instead of 'mle' works fine.
The problem happens in the 'checkFunErrs' functions, inside the 'mlecustom.m' file (call at line 173, actuall error at line 705).
I am quite new to MATLAB, so I am guessing I do not understand how to use this properly, or doesn't mle support that ammount of data?
Thanks for the help.
data = randi([0 1], 1, 1200000);
T_1 = 50000;
T_2 = 100000;
npast = 10000;
start = [0 0 0 0 0 0];
func = @(x, data, cens, freq)loglike(data, [x(1) x(2) x(3) x(4) x(5) x(6)],...
T_1, T_2, npast);
params = mle(data, 'nloglf', func, 'Start', start, 'OptimFun', 'fmincon');
% Computes the negative log likehood
function out = loglike(data, params, T_1, T_2, npast)
size = length(data);
if npast == 0
past = 0;
else
past = zeros(1, size);
past(npast+1:end) = movmean(data(npast:end-1),[npast-1, 0]); % Average number of events in the previous n years
end
lambda = params(1) + ...
(params(2)*cos(2*pi*(1:size)/T_1)) + ...
(params(3)*sin(2*pi*(1:size)/T_1)) + ...
(params(4)*cos(2*pi*(1:size)/T_2)) + ...
(params(5)*sin(2*pi*(1:size)/T_2)) + ...
params(6)*past;
out = sum(log(1+exp(lambda))-data.*lambda);
end
0 个评论
采纳的回答
Mihaela Jarema
2022-2-1
Hi Jose,
The error you mention is thrown when mle calls your custom function loglike, in the last line of the custom function loglike:
out = sum(log(1+exp(lambda))-data.*lambda);
It’s the data.*lambda part which throws the error, since
To avoid implicit expansion, you might want to try out working with column vectors instead of row vectors, e.g., by changing the order of the arguments in the first line from 1, 1200000 to 1200000,1 and transposing others like 1:size to (1:size)’. By the way, you might also want to rename the variable size to something else, like len, since size is a built-in MATLAB function you are overwriting.
Hope this helps,
Mihaela
0 个评论
更多回答(0 个)
另请参阅
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!