Parallel Computing with Command Line Scripts
1 次查看(过去 30 天)
显示 更早的评论
I am using Matlab as a wrapper code to run a windows exe. Currently, I am writting input files for the exe with Matlab. Then writing a .bat file to submit all of the files to the exe. Then Matlab reads the output files. Then Matlab processes the data. Currently I am writing one .bat file and so each input file is submitted serially to the exe. However, I would like to make use of my multiple cores to speed up the process. How can I run these files in parallel? I was thinking I could create the same number of .bat files as number of cores and evenly divide up the input files across all of the .bat files and thus cores. Is there a better way to submit work in command line for parallel computing? Also, how do I determine how many cores I have? Is this the same as the number of workers Matlab displays when performing parallel loops?
Current steps:
- Matlab writes input files
- Matlab writes .bat file
- .bat file submits input files to exe and moves output files to correct folder
- Matlab reads output files
- Matlab plots data
Potential steps:
- Matlab writes m-input files using parfor
- Matlab writes n-.bat files (n = number of cores) with m/n input files per .bat file (how do I determine n?)
- Matlab submits n-.bat files using parfor to Windows command line
- Matlab reads in and processes data
3 个评论
Walter Roberson
2019-3-25
Outline:
- find/create an output directory.
- begin loop
- prepare an input file.
- create a bat that instructs running the .exe on the input file, writing to an output file in the output directory. In the step after the .exe is run (so after it is finished processing) have the bat create file [output_file_name '.ok'] or similar. That is, the .exe will probably start creating the output file early, but the output file will not be ready for post processing until the exe finishes executing, and monitoring for the end of the exe is a nuisance. So have the .bat create a file that will not even exist until the output file is completely ready
- system() the .bat using '&' at the end of the command line to cause the bat to be run asynchronously. The system() will not wait until the .bat finishes executing if you use & on the command line.
- Add the .ok file name to a list of files you are expecting to see later
- dir() the output directory for .ok files . Each of them represents a completed run. For each of them, run the post-processing phase on the corresponding output file and delete the .ok file (or rename it to something else if you want to keep track of which ones have already been processed.) Remove the .ok file name from the list of files you are expecting to see later
- end loop
- after all of the inputs have been prepared, then loop until the list of files you expect to see later is empty. The loop would be the same as the above "dir() the output directory for .ok files" step. At any given time that all of the existing .ok files have been handled but there are still files left in the list of what you expect to see later, then pause() for a time to give time for .exe to finish processing in the background. You might want to put in a failsafe execute of the loop in case some output file did not get created (e.g., a problem with the .exe)
回答(0 个)
另请参阅
类别
在 Help Center 和 File Exchange 中查找有关 File Operations 的更多信息
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!