minibatchqueue dispatch in background slow

3 次查看（过去 30 天）
Luke Hubbard
Luke Hubbard 2022-11-11
评论： Luke Hubbard 2022-11-15
Is there a reason using minibatchqueue with dispatchInBackground set to true would be slower than false?
I'm using a machine with A100 GPU (40Gig) and 50 cores
When I turn the dispatchInBackground option on, it's like the minibatchqueue preprocess function gets in an infinite loop prepping data. The parallel pool turns on, but it only a portion of the four bars on the parallel indicator ever turns green. If I turn the option off, it's slow but runs fine.
  2 个评论
Joss Knight
Joss Knight 2022-11-14
Hard to answer without more information really, like some example code. It might be slower if you're accidentally using a remote cluster instead of a local one or if there is otherwise some problem for your pool workers reading the files. Or perhaps your data is in memory rather than on the filesystem, in which case you usually shouldn't use dispatchInBackground. If file access and preprocessing is trivial in terms of performance then you may find that the real bottleneck is communication between the pool and your client MATLAB. Or it could be you have preprocessing code that is very compute intensive and the workers are competing with each other and the client for resources.
Luke Hubbard
Luke Hubbard 2022-11-15
My datastore preprocessing function is mostly file i/o that pulls in a table from a given file (the files are not local but rather on a shared network--maybe a source of issue?) with a couple of extra steps adding a few new columns (features) to the table, taking the log of features with large numbers, and then using the datasample() fcn to return a smaller portion of the table. Then I split the features into the predictor sequence array and responses categorical array for use in sequence training.
When I set up my minibatchqueue, I have another preprocessing function to pad batch sequences and return as dlarray. Like I said before, the network runs and learns as expected without the dispatchInBackground feature turned on but hangs otherwise.
I'm not at liberty to share actual code. I couldn't find any other community questions that were similar, so I was curious if this is a known issue.

请先登录，再进行评论。

请先登录，再回答此问题。

回答（0 个）

类别

在 Help Center 和 File Exchange 中查找有关 Image Data Workflows 的更多信息

标签

产品


版本

R2022b

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by