Can Matlab "Memory Map" actual memory addresses?

3 次查看（过去 30 天）
Anthony Barone
Anthony Barone 2017-12-9
评论： Walter Roberson 2017-12-18
Id like to be able to memory map a set of actual memory addresses and access data from those addresses from multiple instances of Matlab. Is this possible?
Note: This differs from standard memory mapping in the sense that I do NOT want to map some data on disk to virtual memory addresses, I want to map actual memory addresses and access the data stored in them from multiple instances of Matlab.
Thanks!
.
.
Note on Intended usage:
Without getting into too many details, I envision using this in a way such that one instance reads data into the memory-mapped memory addresses (perhaps with some zero padding) (some data headers will also get stripped out and delt with separately in this step), one or more instances processes the data (each instance would use the output aof the previous instance as an input), and one final instance takes data that has finished processing and writes it to disk in a new variable.
This same set of memory addresses would be cyclically reused multiple times to process the entire dataset (A bit over 2TB of data in my immediate use case). Each processing step only requires a local subset of the data that is saved sequentially in the data (no more than ~3000 uint32 values, and as little as a single uint32 value). Each instance would memory-map a "progress indicator" to ensure that other instances dont access data before it is ready for them as well as to ensure that data isnt overwritten until after the last instance has finished saving the processed data to disk.
This is conceptually a bit like "piping" the data in the sense that a given operation would start working on data before the previous operation has completely finished, though is different in the sense that the data is returned to main memory in between each step and each step is performed by different CPU cores (this makes it so that single threaded operations like fread and fwrite dont force you to not use the other CPU cores). It might make sense to lock each Matlab instance to the CPU core(s) you want it to use (to prevent cache thrashing and other potential problems).
.
.
Side Note: if this isnt directly possible, has anyone tried creating a ramdisk and then memory mapping that? Any chance Matlab is smart enough to realize that the data is already in memory and not try to remap the physical memory adresses to virtual ones?
(I dont know if this would affect performance much, but I have A LOT of data to run through and am concerned with overusing virtual memory addresses resulting in out of memory issues).
(The machine this will run on also has a rather unique NUMA related problem, where if I limit the Matlab instance to using 1 NUMA node and run out of memory (virtual+physical) on that node, the process of moving stuff (such as cached data) to another NUMA node can be very slow in some situations. This can slow things down by a factor of 20x during parts of my code where Matlab is requesting additional memory).

请先登录，再回答此问题。

采纳的回答

Walter Roberson
Walter Roberson 2017-12-9
  2 个评论
Anthony Barone
Anthony Barone 2017-12-11
I'll be damned, this is (almost) exactly what I wanted. I only say "almost" because it seems like (based on the comments) more of the code has stopped working on newer versions of Matlab (the original was from 2010ish), though it seems like there is a chance of getting it to work with cell arrays.
I dont suppose you know of anything similar that is a bit more up to date and more fully working, do you? Im guessing if you did you would have suggested that instead, but I figure it cant hurt to ask.
Thanks for the suggestion. If I can get it to work (even if only with cell arrays) this could save me a lot of time...
Walter Roberson
Walter Roberson 2017-12-18
Sorry, James is the go-to person for Mex and APIs; I would have to study what he has posted to have any hope at all of updating the code.

请先登录，再进行评论。

更多回答（0 个）

类别

在 Help Center 和 File Exchange 中查找有关 Data Type Conversion 的更多信息

标签

产品

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by