How to I pass off a Simulink application command syntax to MATLAB Job Scheduler on a local cluster?

Question

Ewen Chan 2019-9-5

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/479051-how-to-i-pass-off-a-simulink-application-command-syntax-to-matlab-job-scheduler-on-a-local-cluster

评论： Ewen Chan 2019-9-10

I have a MATLAB/Simulink application that I normally would call from the MATLAB command window with:

app inputfile.m

And then it would run.

If I want to pass that off to the MATLAB Job Scheduler, how would I go about doing that?

I tried following the help documentation about using the cluster profile, creating the parallel job, and then creating the task, except that in the syntax for createTask(j, F, N...)

I tried to put in the "app inputfile.m" as the function into the createTask function and that didn't work.

(cf. https://www.mathworks.com/help/parallel-computing/program-a-job-on-a-local-cluster.html)

What would be the better way for me to pass off "app inputfile.m" to MJS?

Your help is greatly appreciated.

Thank you.

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Edric Ellis 2019-9-6

1
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/479051-how-to-i-pass-off-a-simulink-application-command-syntax-to-matlab-job-scheduler-on-a-local-cluster#answer_390635

在 MATLAB Online 中打开

Where do the results from running app inputfile.m go? Does this create variables directly in the base workspace?

To use app together with createTask, you'd probably need to call it like this:

createTask(job, @app, 0, {'inputfile.m'})

However, that might not do what you want. It might work to use the batch command, a bit like this:

job = batch('app inputfile.m');
wait(job), load(job)

If that does work, then you can simply call batch multiple times.

An alternative might be to use parfor to run app in a parallel loop. But that depends a bit on how the outputs of app are produced...

10 个评论
显示 8更早的评论隐藏 8更早的评论

Ewen Chan 2019-9-6

So...here is what I was thinking/what my idea was (again, drawing for my previous experience as a CAE/FEA engineer):

We used to pack up our simulations into a zip file and then upload it to the HPC cluster and then it will unpack said zip files (which will contain all of the input files it needs for the run) and then run it, and then zip up the result files back up, so that we can download it and post-process the results.

The runs that I've got now are almost all entirely self-contained within its own directory. The output files takes the name of the inputfile.m and writes it out to disk as inputfile.csv (for example) as the outputs. So if I want to perform different runs in the same directory, I would change the name of the inputfile from inputfile.m to inputfile1.m (for example) and then the outputs will be written to disk as inputfile1.csv.

I hope that helps and that makes sense.

If I don't change the name, then the app assumes that I am looking to overwrite the existing results, so that will be on me.

If I use batch, will I see the job/task in the MATLAB Job Monitor?

Yeah, I read through the simple batch processing page as well, but I couldn't make the connection between what it states there (where the syntax was showing the example of a function) to my running a Simulink app that was developed in-house.

Thanks.

Ewen Chan 2019-9-6

"but it might be necessary to attach additional files to the job so that the cluster workers can load all related models."

If it is in the search path of the local MATLAB installation/instance, will I still need to do that or will batch know to go look for those additional files via the PATH that has been set for the MATLAB instance?

Also, with batch, does this mean that I will be able to submit as many jobs to the MATLAB job scheduler as there are physical cores in the system?

So again, the way that I am currently doing things is that if there are multiple runs that I need to perform, I run them manually, sequentially.

All of the runs are independent from each other and they're also all serial runs.

Will I be able to use batch to submit multiple jobs to MJS so that the will be executed in parallel from the MJS?

(e.g. my system has a four core processor. Will I be able to submit four jobs to MJS and have MJS pick them all up immediately and run them simultaneously, four serial jobs running in parallel or will it just automate my manual, but serial, sequential execution process that I am currently doing?)

Also, will I need to start multiple instances of MATLAB to submit each job from each MATLAB instance or can I submit all four jobs with a single MATLAB instance?

Thank you.

Ewen Chan 2019-9-6

Edric:

I haven't tried running it with batch just yet.

There's a problem with my latest manual run, so I am re-running it now to try and figure out what the issue it. (I am 99.9999% sure that it's not MATLAB related, but related to an error state in my input file, but I'm checking that right now.)

Initially, I'll be using my local cluster.

What I've proposed to my management is that I will hopefully be able to eventually offload the runs from my local laptop to a server that sits in the office permanently and we just submit our jobs to that as a shared resource. The goal is to alleviate some of the problems that we are encountering with running the jobs locally on our laptop (which I can't get into, nor discuss). And being that I've been told that this is a legacy app, so they're not working on actively developing it/making changes to it anymore, I told my management that I am just going to see if I can put a "wrapper" around it so that the idea is that it would be sent to MJS and processed on that and then we just download the results back (either as the entire run directory or something else) so that the app itself no longer has to be developed to enable this. (It's an automated version of me, sitting on the server, submitting jobs.)

My idea is that if I can prove/demonstrate that it works on my local laptop, that that might help convince them to get a server that sits in the office and I just essentially repeat the process, except rather than submitting to the 'local' cluster profile, it would be sending the jobs to the server instead.

This is the idea.

This is also why I am somewhat fixated on the MJS idea because I presume that eventually/ultimately, if the jobs are sent to a remote server and the server is running the app (as though I was sitting in front of it, physically and doing what I'm currently doing, but manually), then it frees my system up for me to do other stuff.

Thank you.

Edric Ellis 2019-9-10

If you're interactively waiting for things to complete, then perhaps the Job Monitor might be the way forward - there you'll be able to see when things are complete.

Programattically - you can either call wait with a timeout, or check job.State to see if it is 'finished'.

Ewen Chan 2019-9-10

在 MATLAB Online 中打开

This would mean that I would have to create a M-script that will submit the job, correct?

The other question that I also have was that in the example (re: submitting a job as batch, by right-click on the M-script file), it automatically assigns the ID.

Is there a way to do that programmatically? i.e. instead of:

job_n = batch('app inputfile.m', ...)

that it will automatically assign an ID to it without having to manually specify the ID.

How would I go about doing that?

请先登录，再进行评论。

Answer 2

Jason Ross 2019-9-5

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/479051-how-to-i-pass-off-a-simulink-application-command-syntax-to-matlab-job-scheduler-on-a-local-cluster#answer_390568

Are the parsim or batchsim commands applicable to your workflow? They are designed to parallelize simulations.

12 个评论
显示 10更早的评论隐藏 10更早的评论

Ewen Chan 2019-9-9

Jason:

Thank you.

So this where I am a little bit confused or maybe I just don't understand it sufficiently.

Right now, on my console, when I type in path, it will give me all of the search folders and directories that have been set/added to the path.

But when I do it with the batch job, it doesn't do that.

Is there a way for me to pass that onto the batch job? I know that the AdditionalPath will be a way to add it, so my question is is there a way to pass off what I currently have set in my console onto that? (i.e. can I "export" it from the console as a character vector or cell array of characters and then have batch "import" it from that?)

In other words, my confusion lies in how would I pass information from my "console" to the batch workers if it dosen't automatically or already take that into consideration?

Would I have to create a M-script file where I set path=path; so that it would then be passed on to the worker clients?

Your help is greatly appreciated.

Thank you.

Ewen Chan 2019-9-9

在 MATLAB Online 中打开

Jason:

It was an error.

But rather than me defining an additional variable for the AdditionalPaths, I just did this:

job = batch('app inputfile.m', 'Profile', 'local', 'Pool', 0, 'AdditionalPaths', path);

and that seems to be working.

It'll take a little while for my app to run, so I'll just let it go, but it does look like that it is finally running.

I'll have to see if it will produce the outputs and also for me to be able to get them from the job, but at least it is starting the run, so this looks like it's promising progress.

Thank you.

Ewen Chan 2019-9-9

Yes, I am running this on WIndows, and to make matters worse, I think that some of the paths might be a mixture of OSes.

But again, it looks like that with the batch submission syntax that I have now, it appears to be working, so it looks like that it was able to take the mixed OS syntax.

Thanks.

请先登录，再进行评论。

Answer 3

Ewen Chan 2019-9-9

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/479051-how-to-i-pass-off-a-simulink-application-command-syntax-to-matlab-job-scheduler-on-a-local-cluster#answer_391088

Tangential question - my MATLAB R2015a installation is missing the mdce.bat from C:\Program Files\MATLAB\R2015a\toolbox\distcomp\bin.

That file is apparently required to start the mdce service so that I can start MJS.

How can I "backfill" that missing file?

I'm not really sure why it's missing, but it looks like that is needed to start MJS.

Once again, your help would be greatly appreciated.

Thank you.

2 个评论
显示无隐藏无

Jason Ross 2019-9-9

When you run the installer, make sure that MDCE is selected and it should be there. When you set up the MDCE installation it's a best practice to do an all products install so you can service any inbound request from any arbitrary user.

If you try and backfill files manually it can become a tedious nightmare very quickly -- the installer knows how to do "the right thing".

Ewen Chan 2019-9-9

Ahhh....I might not have much in the way of control over that then because that's handled by the IT administration and the installers are only downloadable by IT, and not by/for end users.

Hmmm....bummer.

Let me talk with my IT to figure out what's going on with that.

Thanks.

请先登录，再进行评论。

Answer 4

Ewen Chan 2019-9-10

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/479051-how-to-i-pass-off-a-simulink-application-command-syntax-to-matlab-job-scheduler-on-a-local-cluster#answer_391148

在 MATLAB Online 中打开

So the application ran and I was able to get the results that were written out by the application, but at the end, when MATLAB tries to load the results from the job, this is the error message that I get:

Error using parallel.Job/load (line 33)
Error encountered while running the batch job. The error was:
The task result was too large to be stored.
Caused by:
    Error using parallel.internal.cluster.FileSerializer>iSaveMat (line 281)
    Data too large to be saved.

Any ideas on how I might be able to resolve this and/or get around it?

Thank you.

4 个评论
显示 2更早的评论隐藏 2更早的评论

Edric Ellis 2019-9-10

It is indeed to do with how the 'local' cluster is saving data. But that is derived from your user-level MAT file preferences. If you change your default MAT file version to v7.3, then (unless you specify otherwise), all MAT files created by MATLAB - including those used by the 'local' cluster - will be able to store >= 2Gb. The link I provided should give you instructions as to how to do that.

The reason you don't see this when running your application locally is that the results are returned to you in RAM. Any cluster type has to store the results of a job in some persistent storage somewhere so that you can pick them up later. 'local' uses MAT files on disk for this.

Ewen Chan 2019-9-10

Edric:

Thank you.

Sorry, I had missed that.

I thought that I would have to edit my Simulink application code to enable that.

My apologies.

请先登录，再进行评论。

How to I pass off a Simulink application command syntax to MATLAB Job Scheduler on a local cluster?

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

回答（4 个）

10 个评论
显示 8更早的评论隐藏 8更早的评论

12 个评论
显示 10更早的评论隐藏 10更早的评论

2 个评论
显示无隐藏无

4 个评论
显示 2更早的评论隐藏 2更早的评论

另请参阅

类别

标签

产品

版本

Community Treasure Hunt

How to I pass off a Simulink application command syntax to MATLAB Job Scheduler on a local cluster?

0 个评论 显示 -2更早的评论隐藏 -2更早的评论

回答（4 个）

10 个评论 显示 8更早的评论隐藏 8更早的评论

12 个评论 显示 10更早的评论隐藏 10更早的评论

2 个评论 显示 无隐藏 无

4 个评论 显示 2更早的评论隐藏 2更早的评论

另请参阅

类别

标签

产品

版本

Community Treasure Hunt

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

10 个评论
显示 8更早的评论隐藏 8更早的评论

12 个评论
显示 10更早的评论隐藏 10更早的评论

2 个评论
显示无隐藏无

4 个评论
显示 2更早的评论隐藏 2更早的评论