load file over network vs copyfile

7 次查看（过去 30 天）
Martin
Martin 2021-4-30
评论： Martin 2021-5-3
Hi,
So this is not a complex coding problem, more a basic matlab understanding issue.
I have access to huge amounts of data (in the range of 100s of terabytes) in a network folder. When I read this data (using load) on the actual network path i get a speed in the range of 10-30mbit/s (based on information in the task manager). This is quite slow and will not work.
When I instead copy the folders using "copyfile" from the network location to C:\ I get a speed in the range of 150-200 mbit/s.
Then I can continue reading the variables I'm interested in.
What I can't understand is why there is such a big difference in speed using load on the network location and copying the file to C:\ could anyone here please enlighten me? :)

请先登录，再回答此问题。

采纳的回答

Walter Roberson
Walter Roberson 2021-4-30
load() needs to seek() in the file, and potentially read sections of the same file multiple times.
When seeking around is done, the file cannot logically be cached because on a network drive it must be assumed that another process might be writing to the file. Though that is a detail that depends on the exact file system, as on some network file systems, the file could potentially be locked while it is open, even over the network.
  3 个评论
显示 1更早的评论
Walter Roberson
Walter Roberson 2021-5-3
I would not say "always", but that would be most common, yes.
Martin
Martin 2021-5-3
Sorry Walter but you seem to know this stuff, can I add one question?
Would it be possible to virtually run Matlab on the network path somehow? (Lets assume X is a mapped network folder, and that I would be permitted by the network admins.)
If the files are located at "X:\My_files\", could I create "X:\tempAnalysis\" and run the workspace buffer and cache from there? Or something similar to get around the "copy stuff to C:" but still be way faster than reading over the network as described in the initial question.
Or how would you have solved a similar problem? If you could just push me in the right direction that would be greatly appreciated!

请先登录，再进行评论。

更多回答（0 个）

类别

在 Help Center 和 File Exchange 中查找有关 Deep Learning Toolbox 的更多信息

标签

产品


版本

R2019b

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by