Do the boxplot stats without boxplot

2011 8 13
4 个回答
4 次查看（30 天）

0 个投票

Dear all, I have an enormous number of numbers and I want to plot the box plot
Actually I have 7 sets of 125.000.000 numbers (luckily I am running this in a system with huge ram). As boxplot with 7 inputs takes so much time I was thinking if there might be a way to calculate for every set all the statistics boxplot calculates.. and then feed them in a boxplot_type function to do the stupid plotting.
Is there something like that in matlab?
I would like to thank youf or your help
Best Regards Alex

请先登录，再回答此问题。

回答（4 个）

Oleg Komarov
Oleg Komarov 2011-8-13

1 个投票

Time boxplot:
tic
boxplot(...)
toc
against:
tic
prctile(...)
max(...)
min(...)
toc
and also against:
tic
sort()
toc
for one of your datasets, if there's significant gain for one of the alternatives then yes you could improve boxplotting...but I doubt it.
EDIT
Avoiding to call boxplot on the big database I create a fake boxplot and adjust it with the stats calculated from the real database:
% Suppose truedata is your dataset A
truedata = rand(1e6 + 123423,7);
sz = size(truedata);
% Create fakedata and boxplot it
fakedata = rand(10,sz(2));
h = boxplot(fakedata,'labels',10:10:10*sz(2));
% Now sort your truedata and calculate min,max,25,50,75 percentile
truedata = sort(truedata);
s.mins = truedata(1,:);
s.maxs = truedata(end,:);
xi = bsxfun(@plus,sz(1).*[0.25; 0.5; 0.75], sz(1) * (0:sz(2)-1));
x = [floor(xi(:)); ceil(xi(:))];
s.ptiles = reshape(interp1(x,truedata(x),xi(:)),3,sz(2));
% Readapt the fake boxplot:
% 1.Adjust upper whisker
set(h(1,:),{'Ydata'}, num2cell([s.ptiles(3,:); s.maxs].',2));
set(h(3,:),{'Ydata'}, num2cell(repmat(s.maxs.',1,2),2));
% 2. Adjust lower whisker
set(h(2,:),{'Ydata'}, num2cell([s.mins; s.ptiles(1,:);].',2));
set(h(4,:),{'Ydata'}, num2cell(repmat(s.mins.',1,2),2));
% 3. Adjust body and median
set(h(5,:),{'Ydata'}, num2cell(s.ptiles([1 3 3 1 1],:).',2));
set(h(6,:),{'Ydata'}, num2cell(repmat(s.ptiles(2,:).',1,2),2));
% 4. Delete outlier marking
delete(h(end,:))

7 个评论
显示 5更早的评论 隐藏 5更早的评论

Alex
Alex 2011-8-13
Thanks a lot I will check and I will report back
Alex
Alex 2011-8-15
Thanks a lot. I have notirced something really strange.
After doing sort in the A matrix... boxplot returns much faster back.
Could you please explain why is this behaviour? For me it is something so strange..
Oleg Komarov
Oleg Komarov 2011-8-15
I don't observe the same behaviour. What code did you call?
Alex
Alex 2011-8-16
size(A)
ans =
131072000 7
A=sort(A);
boxplot(A,'symbol','','labels',[1.14 3.43 5.72 8.01 10.3 12.6 14.9]);
set(gca,'YScale','log');
this takes 30 mins and plots. If I do not sort the data I do not get results
Oleg Komarov
Oleg Komarov 2011-8-16
It's possible that the internal sort of boxplot "recognizes" that the array is already sorted and do not call othe internal routines or it may be that the subfunction don't end modifying A and copy is created keeping memory consumpution at a "reasonable" level.
Anyways, the fastest method should be the one I indicated in my edit. Consider that you can create the fake boxplot with all the features you want and adjust it later keeping all of those. Right now I am eliminating the red outliers but it's very simple to keep them and update.
Alex
Alex 2011-8-16
just to make it more "formal"
tic;
A=sort(A);
boxplot(A,'symbol','','labels',[1.14 3.43 5.72 8.01 10.3 12.6 14.9]);
set(gca,'YScale','log');
toc;
Elapsed time is 1123.973379 seconds.
I comment out the sort
tic;
%A=sort(A);
boxplot(A,'symbol','','labels',[1.14 3.43 5.72 8.01 10.3 12.6 14.9]);
set(gca,'YScale','log');
toc;
Elapsed time is 1502.582405 seconds.
why it does more time in the second casE?
Oleg Komarov
Oleg Komarov 2011-8-16
Use the profiler to check the differences.
Have you tried my other solution?

请先登录，再进行评论。

Alex
Alex 2011-8-13

0 个投票

Hmm... what I tried is the following: I only did boxplot for the first set only 1*125.000.000 sets and matlab replied after 15 minutes. Then I tried boxplot with the 7 sets (7*125.000.000 sets) which unfortunately has not retuned after 3 hours..
So far matlab consumed 110 Gb of memory and keeps going on....
If there was a way to do 7*boxplots(for only one set) and then put together I think I will be finished by now. Is there any way to do that?
B.R Alex

1 个评论
显示 -1更早的评论 隐藏 -1更早的评论

Oleg Komarov
Oleg Komarov 2011-8-13
You said 1*125.000.000 sets, what do you mean?
Sorting 1e8 elements takes 11 seconds on my laptop, after that most of the calculations can be done in instants. I really don't see how boxplot is taking som much time.
Please post the code you're using.

请先登录，再进行评论。

Alex
Alex 2011-8-14

0 个投票

I wouldl ike to thank you for your answer
size(A)
ans =
131072000 7
so I have a matrix of 131072000*7 and I want a box plot
this boxplot(A','symbol','');
will crash after 6 hours after consumerd 300GB (three hundred gigabytes of ram!)

3 个评论
显示 1更早的评论 隐藏 1更早的评论

Oleg Komarov
Oleg Komarov 2011-8-14
Please don't create additional answers but use the comment facility (I know it doesn't allow code formatting but it keeps the logical track of question-answer).
Have you tried to
A = sort(A);
If you can sort it in a reasonable amount of time then we could find a workaround. You can also try to sort column by column (to break the memory requirements:
for n = 1:size(A,2)
A(:,n) = sort(A(:,n));
end
It takes so much time because boxplot uses uniquerows and unique. What happens is that it fills up all of your memory (300 GB who do you work for?) and starts swapping on hard disk.
Alex
Alex 2011-8-15
(I hope this time I did reply correct!)
tic; test=sort(A); toc
Elapsed time is 35.341047 seconds.
As you can see that was extremely fast. There is no more swap space and the system administrator is not here to ask him to add more!
One solution is to try to do 7*different Box plots and then with some magic way to make them one. As you can read below , yesterday I have explained what I have tried but all the boxplots get stuck the one over it.
B.R
Alex
Oleg Komarov
Oleg Komarov 2011-8-15
Give a try to my EDIT in my answer.

请先登录，再进行评论。

Alex
Alex 2011-8-14

0 个投票

I then tried the following I wrote
>> boxplot(A(:,1)','symbol','','labels',[10]); hold on; boxplot(A(:,2)','symbol','','labels',[20]); boxplot(A(:,3)','symbol','','labels',[30]); boxplot(A(:,4)','symbol','','labels',[40]); boxplot(A(:,5)','symbol','','labels',[50]); boxplot(A(:,6)','symbol','','labels',[60]); boxplot(A(:,7)','symbol','','labels',[70]); hold off;
which makes it run but unfortunately all the 7 boxplots are plotted the one over the other. As you might have guessed already these 7 sets are for the 10,20,30,40,50,60,70 cases.
I would like to thank you for your answers
Best Regards Alex

1 个评论
显示 -1更早的评论 隐藏 -1更早的评论

Oleg Komarov
Oleg Komarov 2011-8-14
Please see the edit on my answer and stop creating additional answers, use comments.

请先登录，再进行评论。

类别

在 帮助中心 和 File Exchange 中查找有关 Box Plots 的更多信息

标签

提问:

Alex
Alex
2011-8-13

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by