Better memory-mapped files in Matlab

版本 1.31.0.0 (120.8 KB) 作者: Dylan Muir
A better, transparent memmapfile, with complex number support.
5.0
(5)
1.2K 次下载
更新时间 2020/5/18
在 GitHub 上查看许可证

See also http://dylan-muir.com/articles/mapped_tensor/
If this function is useful to your academic work, please cite the publication in lieu of thanks:
Muir and Kampa, 2015. "FocusStack and StimServer: A new open source MATLAB toolchain for visual stimulation and analysis of two-photon calcium neuronal imaging data". Frontiers in Neuroinformatics.

This class transparently maps large tensors of arbitrary dimensions to temporary files on disk. Referencing is identical to a standard matlab tensor, so a MappedTensor can be passed into functions without requiring that the function be written specifically to use MappedTensors. This is opposed to memmapfile objects, which cannot be used in such a way. Being able to used MappedTensors as arguments requires that the tensor is indexed inside the function (as opposed to using the object with no indices). This implies that a function using a MappedTensor must not be fully vectorised, but must operate on the mapped tensor in segments inside a for loop.

MappedTensor also offers support for basic operations such as permute and sum, without requiring space for the tensor to be allocated in memory. memmapfile sometimes runs out of virtual addressing space, even if the data is stored only on disk. MappedTensor does not suffer from this problem.

Functions that work on every element of a tensor, with an output the same size as the input tensor, can be applied to a MappedTensor without requiring the entire tensor to be allocated in memory. This is done with the convenience function "SliceFunction".
An existing binary file can also be mapped, similarly to memmapfile. However, memmapfile offers more flexibility in terms of file format. MappedTensors transparently support complex numbers, which is an advantage over memmapfile.

Example:
mtVar = MappedTensor(500, 500, 1000, 'Class', 'single');
% A new tensor is created, 500x500x1000 of class 'single'.
% A temporary file is generated on disk to contain the data for this tensor.

for (i = 1:1000)
mtVar(:, :, i) = rand(500, 500);
mtVar(:, :, i) = abs(fft(mtVar(:, :, i)));
end

mtVar = mtVar';

mtVar(3874)

mtVar(:, 1, 1)

mfSum = sum(mtVar, 3);
% The sum is performed without allocating space for mtVar in
% memory.

mtVar2 = SliceFunction(mtVar, @(m)(fft2(m), 3);
% 'fft2' will be applied to each Z-slice of mtVar
% in turn, with the result returned in the newly-created
% MappedTensor mtVar2.

clear mtVar mtVar2
% The temporary files are removed

mtVar = MappedTensor('DataDump.bin', 500, 500, 1000);
% The file 'DataDump.bin' is mapped to mtVar.

SliceFunction(mtVar, @()(randn(500, 500)), 3);
% "Slice assignment" is supported, by using "generator" functions that accept no arguments. The assignment occurs while only allocating space for a single tensor slice in memory.

mtVar = -mtVar;
mtVar = 5 + mtVar;
mtVar = 5 - mtVar;
mtVar = 12 .* mtVar;
mtVar = mtVar / 5;
% Unary and binary mathematical operations are supported, as long as they are performed with a scalar. Multiplication, division and negation take O(1) time; addition and subtraction take O(N) time.

引用格式

Dylan Muir (2024). Better memory-mapped files in Matlab (https://github.com/DylanMuir/MappedTensor), GitHub. 检索来源 .

MATLAB 版本兼容性
创建方式 R2009a
兼容任何版本
平台兼容性
Windows macOS Linux
类别
在 Help Center 和 MATLAB Answers 中查找有关 Large Files and Big Data 的更多信息

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

@MappedTensor

@MappedTensor/private

无法下载基于 GitHub 默认分支的版本

版本 已发布 发行说明
1.31.0.0

Updated description
Updated description
Updated description
Updated formatting
Updated description
Updated description
Updated description
1.30.0.0

Moved to github hosting
Added "neuroscience" tag
1.29.0.0

Improved referencing for edge cases.
1.28.0.0

Improved referencing to make it more similar to matlab tensors. FIxed a referencing bug involving a confusion between "58" and ":".
1.27.0.0

Updated usage notes.
1.26.0.0

Added paper reference.
1.25.0.0

Indexing improvement
1.24.0.0

Accelerated reading of data, especially when accessing chunks of data in sequential order.
1.23.0.0

Updated description
1.22.0.0

Fixed bug where call to fopen failed
1.21.0.0

Removed compiled MEX files from archive
1.20.0.0

Fixed several mex compilation bugs.
1.19.0.0

Updated summary
1.18.0.0

Updated description
1.17.0.0

Updated description
1.16.0.0

Fixed a regression, such that SliceFunction no longer worked. Thanks to Stanislas Rapacchi for the bug report.
1.15.0.0

Minor bug fixes
1.13.0.0

MappedTensor now uses mex-accellerated internal functions, if possible. MappedTensor is now much faster.
1.11.0.0

Fixed a referencing bug, where repeated indices and multi-dimensional indices were not referenced correctly on reads.
1.10.0.0

Accelerated SliceFunction; SliceFunction now provides a slice index argument; better error reporting when too many dimensions were used for indexing; SliceFunction now provides feedback during operation
1.9.0.0

MappedTensor now does not rely internally on memmapfile, but performs optimised direct binary file reads. It is now much faster than memmapfile, for some tasks. You can now specify a header offset to skip, when mapping an existing file.
1.8.0.0

Updated description
1.7.0.0

Updated image
1.5.0.0

Added support for unary uplus, uminus; binary plus, minus, times, mtimes, m/l/r/divide (all with a scalar).
1.4.0.0

Fixed a bug in linear indexing of a permuted tensor; added support for slice assignment; added support for complex values.
1.3.0.0

Added support for "sum"; added SliceFunction.
1.2.0.0

Added a brief example, more details of restrictions.
1.0.0.0

要查看或报告此来自 GitHub 的附加功能中的问题，请访问其 GitHub 仓库。
要查看或报告此来自 GitHub 的附加功能中的问题，请访问其 GitHub 仓库。