Logging parellel pool worker diaries to single file

Question

Ben Wetherill 2018-1-31

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/380044-logging-parellel-pool-worker-diaries-to-single-file

编辑： Ben Wetherill 2018-2-2

I'm converting a procedural script into a parallel script. The existing script dumps the command window output using Diary. So each job's output was nicely lumped together. Now though, as I have tasks running in parallel my log file is very disjointed. Not ideal.

Is there a way I can collect each worker's diary and send it to a dataQueue for storage in a single file? In pseudo code it would be:

parfor ...
  run task (including command window outputs)
  send Diary content back to client dataPool for writing to log file
end

Is there some method within my parfor loop of saying 'send(q, thisWorker.Diary)'?

2 个评论
显示无隐藏无

Ben Wetherill 2018-1-31

编辑：Ben Wetherill 2018-1-31

While cycling home I thought of a possible solution: each worker just writes their own diary to a random filename report and passes the filename and the job name back to the dataQueue. The dataQueue can then use this to append the report content to the relevant overall log, deleting the temporary files as it goes.

Ben Wetherill 2018-2-1

编辑：Ben Wetherill 2018-2-1

在 MATLAB Online 中打开

Tried my idea above (see code below). It didn't work. The individual temp log files remained blank. So it seems that for parfor you can't get access to each workers command wondow output. So I'm getting all the workers log reports intermixed together in the main client window. Not helpful. :( Will try Edric's idea below.

    parfor i = casesToRun
        % setup worker diary
        tempLogfile = [tempname '.log'];
        diary(tempLogfile);
        doStuff (...); 
        % Send temp diary file to logQueue
        logData = {tempLogfile, logFileName};
        send(logQueue, logData);
    end

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Edric Ellis 2018-2-1

1
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/380044-logging-parellel-pool-worker-diaries-to-single-file#answer_302905

在 MATLAB Online 中打开

Instead of parfor, you could use parfeval to run stuff on the workers. This will require a bit of restructuring - you'll need to divide up your work into decent-sized chunks (this is something that parfor takes care of for you automatically). Then, you can use the Diary property of each parallel.Future instance.

For example

% request execution on a worker  
f = parfeval(@() fprintf('worker diary text: %d\n', rand()), 0);
wait(f), f.Diary

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

Ben Wetherill 2018-2-2

编辑：Ben Wetherill 2018-2-2

在 MATLAB Online 中打开

For interest, below is what I did. There seems to be an issue though with some of the logs not being saved. All cases are run and I have the results, but some of the diary logs are missing. I'll ask seperate question about this issue.

   diary on;
    for i = casesToRun
        f(i) = parfeval(p, @doStuff, 2, ...);
    end 
    for i = casesToRun
        [completedIdx, status, xlRow] = fetchNext(f);
        writeWorkerLog(f(i).Diary, logFileName);
        xlswrite(TestCasesFile,status,'Sheet1',xlRow);
    end
    diary off;