Matlab is significantly slower than Julia on simple evaluation

91 次查看（过去 30 天）
Michal
Michal 2021-11-9
编辑： Bobby Cheng 2022-1-11
More and more frequently some people open the question about speed comparison between Matlab and Julia. The main Julia developers goal was to create high-level language for scientific computing with computing speed comparable to C/C++.
I just read the interesting blog: Testing Julia: Fast as Fortran, Beautiful as Python which shows simple exp() evaluation comparison between Python, Julia and Fortran (details are here).
The corresponding Matlab functions to compare Julia with Matlab:
function [M] = test_eval_exp(N)
disp('Loop-for version:')
for m = 1:10
n = m*N;
tic;
M = eval_exp_loop(n);
tcpu = toc;
disp(['N = ' num2str(n) ' time = ' num2str(tcpu)]);
end
disp('Vectorized version:')
for m = 1:10
n = m*N;
tic;
M = eval_exp_vec(n);
tcpu = toc;
disp(['N = ' num2str(n) ' time = ' num2str(tcpu)]);
end
end
function M = eval_exp_loop(n)
a = linspace(1,2*pi,n);
M = complex(zeros(n,n));
for j = 1:n
for i = 1:n
M(i,j) = exp(1j * (100+1j) * sqrt( a(i)^2 + a(j)^2 ));
end
end
end
function M = eval_exp_vec(n)
a = linspace(1,2*pi,n);
a2 = a.^2;
M = exp(1j*(100+1j)*sqrt(a2 + a2'));
end
And results:
>> test_eval_exp(1e3);
Loop-for version: %... sigle Thread
N = 1000 time = 0.067138
N = 2000 time = 0.2546
N = 3000 time = 0.54324
N = 4000 time = 0.96516
N = 5000 time = 1.5097
N = 6000 time = 2.161
N = 7000 time = 2.9356
N = 8000 time = 3.8305
N = 9000 time = 4.8475
N = 10000 time = 5.9773
Vectorized version: % ... multi Thread (8 threads)
N = 1000 time = 0.036739
N = 2000 time = 0.03424
N = 3000 time = 0.073091
N = 4000 time = 0.12321
N = 5000 time = 0.18877
N = 6000 time = 0.26854
N = 7000 time = 0.36426
N = 8000 time = 0.47832
N = 9000 time = 0.61006
N = 10000 time = 0.75371
% Julia loop-for (1 thread) for N= 1e4 ... 3sec
% Julia vectorized (8 thread) for N= 1e4... 0.6 sec
% Julia vectorized + optimized (8 thread) for N= 1e4... 0.3 sec
These results really supports the general answer, that Julia (at least on this specific task) is significantly faster in a case when Julia and Matlab code are realised simply by for-loop (1-thread ~ 2x). In a case when Matlab code is vectorized and run on 8 core CPU the Julia vectorized for-loop is still faster (~1.25x) . Finaly, the highly optimized Julia code is 2x faster than Matlab vectorized code.
My questions are:
  1. Is there any possibility how to speed up (optimize) loop-for and vectorized Matlab code to get similar speed as Julia?
  2. MATLAB JIT engine produce typically significantly slower simple for-loop code than Julia LLVM. Why? Is possible in future MATLAB releases expect some significant additonal speed up?
My questions are motivated to get the right answers for those people who argue by simply comparing the speed of Matlab vs Julia and completely avoid the other benefits of Matlab.
  3 个评论
显示 1更早的评论
Jon
Jon 2021-11-9
Oh, I guess you are showing those functions with your commented out code below the function call. Is that correct?

请先登录，再进行评论。

请先登录，再回答此问题。

采纳的回答

Bobby Cheng
Bobby Cheng 2021-11-17
First of all, thank you for creating the benchmark and raising this performance issue. Indeed, we are seeing more and more performance comparison like this, what I would call a focused micro-benchmark. At MathWorks, we found this very informative. More often than not, it highlights the less optimized part of the MATLAB language, and it is exactly what this benchmark is picking up. The bad news is that, like Jan, I can confirm that Julia is doing better than MATLAB in the current release for this benchmark, and there is not much you can do within MATLAB to optimize it further. The good news is that the issues discovered here are on our plan to be addressed in a future release.
When I took a look at your benchmark, the most time consuming items are:
  1. Complex scalar execution overhead in MATLAB . Complex is an attribute of certain classes, not a datatype of its own. MATLAB math tends to return the most mathematically correct answer, like sqrt(-1) returns 1i instead of NaN as required by the IEEE standard, and in MATLAB, 1i – 1i returns real 0, instead of 0+0i. This makes MATLAB easy to use, but it does add overhead in execution. We need to minimize this overhead.
  2. There are faster algorithms to compute the exponential function in the literature. But using those algorithms is not a matter of simply ripping out one piece of code, inserting new code, compiling it, and saying "Ship it to users!" We need to ensure that the performance, accuracy, stability, and portability (among other considerations) meet our standards before we publish them as part of a new release. Sometimes this decision is not easy: for example, when the faster and more accurate SUM for single precision data was released, as my colleague Christine noted on Loren Shure's blog (link to https://blogs.mathworks.com/loren/2021/09/07/a-faster-and-more-accurate-sum/) it created noise in our internal test suite and led us to watch for user feedback before we made the same modification to double precision SUM. Not everyone was happy with the change but we could justify why we made the change.
I can't tell you the exact timeline for each of these issues to be addressed because each issue has its own unique challenge, but rest assured that we plan to improve in these areas. Keep checking on MATLAB and keep us on our toes.
Thank you again for the benchmark.

更多回答（1 个）

Jan
Jan 2021-11-10
I can confirm your observation that Julia runs faster than Matlab for some problems. Answering your three questions is not easy due to the restriction "completely avoid the other benefits of Matlab". It is questionable if a comparison of two tools is useful, if only one specific feature is considered. This is like comparing a racing care with a limousine: Of course they have a different speed. As soon as I want to transport a cardboard box, this difference is not relevant.
  • Is there any possibility how to speed up (optimize) loop-for and vectorized Matlab code to get similar speed as Julia?
I tried it without success. I frequently post C-mex functions in this forum, because this is my way to squeeze bottlenecks in Matlab. But some C code cannot be a fair answer to a question about Matlab and Julia.
  • MATLAB JIT engine produce typically significantly slower simple for-loop code than Julia LLVM. Why?
Because I do not have the possibility to control the way Matlab's JIT is working, I cannot guess, why it is slower than Julia.
  • Is possible in future MATLAB releases expect some significant additonal speed up?
Yes. An acceleration is possible.
I've answered some questions concerning the acceleration of code in this forum. The speed of the used computer is less relevant. Even for a lot of money increasing the speed by a factor of 4 is really hard (if neither the number of cores or the RAM is the limitting factor). In opposite to this, accelerating some code by a factor of 100 to 1000 was possible in many cases just by avoiding repeated calculations and using a smarter representation of the data (like columnwise operations and avoiding the dynamic creation of variables by eval()).
Of course the speed of the programming language has an effect also. You have demonstrated an advantage of Julia a tiny test problem. This is interesting and it is a good strategy to use Julia, if it solves a problem efficiently. For my work this is not an option, because I use exhaustively tested tools, e.g. one for clinical decision making with more than 300'000 lines of Matlab code (plus comments), which was tested over 2 decaded with thousands of patients. Converting this to Julia would be a waste of time.
You see, "completely avoid the other benefits of Matlab" is a really hard limitation for writing a meaningful answer to your question. The time to solve a problem consists of:
t_solution = t_design + t_programming + t_debugging + t_documenting + ...
t_runtime
Therefor re-using existing toolboxes has a much more effect than the actaul run time.
But your point is important: If I create a new program e.g. to calculate collisions of galaxies using some simple loops to obtain the accelerations (and not a leap-frog-toolbox, which performs the actual calculations in an assembler routine, such that Matlab is responsible for defining the initial values only and to visualize the results), I'd try this in Julia. Then it would not help to discuss, if a future version of Matlab might be faster, because waiting for years cannot compete with minutes of runtime.
For my daily work, the power of Matlab is more important than its speed. Therefore I'm not the right one to answer your question, but I wanted to make a start at least.
  10 个评论
显示 8更早的评论
Oliver Josephs
Oliver Josephs 2021-12-17
Hi Bobby,
You mentioned on the 10th that the R2022a prerelease was out. Normally I can download this but it seems not at the moment.
Should it be available?
Oliver
Bobby Cheng
Bobby Cheng 2022-1-11
编辑：Bobby Cheng 2022-1-11
Sorry Oliver for the delay response. I was on vacation. You should try download again. There were some issues over the holiday. It should be available now. If you have any issue, you should contact support directly for faster response.

请先登录，再进行评论。

类别

在 Help Center 和 File Exchange 中查找有关 GPU Computing 的更多信息

标签

产品


版本

R2021b

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by