Cancel parfeval causes workers to throw errors

Question

Daniel Flisek 2024-6-12

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/2127921-cancel-parfeval-causes-workers-to-throw-errors

评论： Edric Ellis 2024-6-18

I'm writing a GUI in App Designer to run Simulink simulations in parallel, using parfeval because I need the ability to cancel in-progress tasks. The app has an Abort button that sets a global variable when it is pressed, which the fetchNext loop checks for. The function myFunc runs the Simulink models and does a lot of extra setup and processing as well. Here's some abbreviated code:

% get or create pool
pool = gcp;
% schedule tasks
futures(1:numIterations) = parallel.FevalFuture;
for p = 1:numIterations
    futures(p) = parfeval(@myFunc);
end
% collect results
numCompleted = 0;
timeout = 180;  % 3 minute timeout
while numCompleted < numIterations
    [completedIdx, results] = fetchNext(futures,timeout);
    numCompleted = numCompleted + 1;
    % collect results from tasks that didn't time out
    if ~isempty(completedIdx)
        workerTasks{completedIdx} = results;
    end
    % check for abort
    if app.abortRequested
        break;
    end
end
% check for abort
if app.abortRequested
    cancel(pool.FevalQueue.QueuedFutures);
    cancel(pool.FevalQueue.RunningFutures);
end

This runs fine, and the abort feature works the first time you use it.

Then if you immediately run the code again, several of the tasks come back Failed with the error message "Dot indexing is not supported for variables of this type." If you close down the pool and re-run the program with fresh workers, the problem disappears.

It seems like cancel doesn't fully delete the job from the threads, and there's something left over that interferes with new tasks. Has anyone run into this? Am I missing a step when canceling the futures?

Thanks in advance.

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Edric Ellis 2024-6-14

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/2127921-cancel-parfeval-causes-workers-to-throw-errors#answer_1471886

在 MATLAB Online 中打开

My guess is that whatever myFunc is doing is not safe against being interrupted with CTRL-C. When you cancel a parallel.Future, the execution of myFunc on the worker is (effectively) interrupted as if you had typed CTRL-C.

From the code you've posted, it's not completely clear what's going wrong, but my main suggestion would be: use onCleanup inside myFunc to make sure your code is "interrupt safe".

A secondary point - your "cancel" could be refined very slightly to this:

% check for abort
if app.abortRequested
    cancel(futures);
end

This will ensure you don't cancel anything other than the executions of myFunc

4 个评论
显示 2更早的评论隐藏 2更早的评论

Daniel Flisek 2024-6-14

在 MATLAB Online 中打开

Thanks for the reply. I apologize I can't be more detailed, but I'll describe as much as I can.

myFunc takes a list of Simulink models, and a table of simulation tasks to run. The table contains rows for different input parameter values, so one worker can receive multiple simulation sets.

The worker loads all models in fast restart mode. Then, it does some initialization to create appropriate input values, which get stored to the appropriate model workspace. The first models are executed, and then their outputs become the inputs of the last model in the list. This process is repeated for each entry in the task table.

After all tasks are complete, fast restart is disabled and all models are closed, and simulation results are returned.

My first thought was that being interrupted leaves models open in fast restart. So I added onCleanup to close any open models:

function workerCleanup(modelList)
loadedModels = find_system('SearchDepth',0);
names = modelList(matches(modelList, loadedModels));
for j = 1:numel(names)
    set_param(names(j), 'FastRestart','off');
    close_system(names(j), 0, 'SaveModelWorkspace','false');
end
end

However this didn't seem to help. I'm not sure what qualifies as "interrupt safe" code.

Thanks for your help.

Daniel Flisek 2024-6-17

Thanks for the replies. I added some print statements and the workers that fail, fail at the sim command. I verified that all of the inputs to sim are correct, the model's workspace variables are set properly, and the Simulink models are not still open from the previously-aborted run. Not sure what else to check at this point.

To speed up execution, I call start_simulink on each worker before it receives its task list. I can't seem to find a stop_simulink command, though. Is there a way to close Simulink on the workers, so I can get a fresh session?

If all else fails, I guess I'll just have to shut down and re-open the parallel pool whenever a run is Aborted.

Edric Ellis 2024-6-18

I don't think there is an equivalent of stop_simulink - although there is bdclose('all') which closes all models - that's my usual "big hammer" for this sort of situation.

I'm not sure what else to suggest. It might be worth contacting MathWorks Support as they might be able to help you go through more debugging steps.

请先登录，再进行评论。

Cancel parfeval causes workers to throw errors

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

回答（1 个）

4 个评论
显示 2更早的评论隐藏 2更早的评论

另请参阅

类别

标签

产品

版本

Community Treasure Hunt

Cancel parfeval causes workers to throw errors

0 个评论 显示 -2更早的评论隐藏 -2更早的评论

回答（1 个）

4 个评论 显示 2更早的评论隐藏 2更早的评论

另请参阅

类别

标签

产品

版本

Community Treasure Hunt

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

4 个评论
显示 2更早的评论隐藏 2更早的评论