Help reducing huge time overhead when executing system() or unix() commands from Matlab function

Question

Nick Counts 2017-11-17

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/367548-help-reducing-huge-time-overhead-when-executing-system-or-unix-commands-from-matlab-function

评论： Walter Roberson 2017-11-17

I have a data processing program that goes through gigabytes of .csv text. The approach I've been taking is to use grep (Mac or Linux) to break up my massive data files into digestible files.

Here's the catch: I have to do it a few hundred times. No biggie, grep is fast, I'll knock this out in no time. A test from bash:

Sweet! 1.6s per file and I'll have this done before I finish my coffee break.

Well....

Holy Toledo, that makes a huge difference. My pre-processing is now taking multiple hours per data file. I have tried to look at Matlab environment variables and even dipped my toe into calling through java (checkout jsystem on the exchange!)

Still no luck. I can have my function generate a bash script and then I can run the script manually, I suppose, but I need this to work for other people in the office who are less command line savvy (that's saying something!)

Can anyone shed some light on what's going on or point me towards an improved time?

Thanks a bunch!

% I ran this example on a 3.23 gig text file. 
% 16,706,909 lines
% 3,225,789,919 characters
[~, r] = unix('time LC_ALL=C grep -F "My Data Tag String" "/Path/To/Giant/Data/File.csv" > output.csv')

2 个评论
显示无隐藏无

Walter Roberson 2017-11-17

You could have it generate a script that had all of the commands in it to do all of the splitting, and then do a single system() -- thus getting the overhead only once.

Nick Counts 2017-11-17

True, the single script could be run from my function. I wasn't sure if the overhead is just on spawning each instance, or if there's something deeper at work that would put me in the same position.

Also, it hurts to not have user feedback - my current implementation has a progress bar so my users believe something is happening!

请先登录，再进行评论。

请先登录，再回答此问题。