Hi,
You can implement a custom datastore in MATLAB to shuffle frames within a single large binary file while maintaining the benefits of a datastore.
Custom Datastore class:Create a custom datastore class that extends the matlab.io.Datastore class. This class can be implemented to read and shuffle frames within a binary file.
Implementing Custom Read and Shuffle methods:
- Read Method: Implement the read method to read a specified number of frames from randomly shuffled positions in a binary file. Use "fseek" to move the file position pointer based on the shuffled frame order in the binary file.
- Shuffle Method: Implement a custom "shuffle" method that generates a random permutation of frame indices. This method should update the order in which frames are accessed during reading, without altering the binary file.
Example Custom Datastore class definition:
classdef CustomFrameDatastore < matlab.io.Datastore
properties
FileName
FrameSize
TotalFrames
CurrentIndex
FrameOrder
end
methods
function ds = CustomFrameDatastore(fileName, frameSize, totalFrames)
ds.FileName = fileName;
ds.FrameSize = frameSize;
ds.TotalFrames = totalFrames;
ds.CurrentIndex = 1;
ds.FrameOrder = randperm(totalFrames);
end
function data = read(ds)
if ds.CurrentIndex > ds.TotalFrames
error('No more data to read.');
end
fid = fopen(ds.FileName, 'rb');
frameIndex = ds.FrameOrder(ds.CurrentIndex);
fseek(fid, (frameIndex-1)*ds.FrameSize, 'bof');
data = fread(fid, ds.FrameSize, 'uint8');
fclose(fid);
ds.CurrentIndex = ds.CurrentIndex + 1;
end
function reset(ds)
ds.CurrentIndex = 1;
end
function tf = hasdata(ds)
tf = ds.CurrentIndex <= ds.TotalFrames;
end
function shuffle(ds)
ds.FrameOrder = randperm(ds.TotalFrames);
ds.CurrentIndex = 1;
end
end
end
Here is the example code to use the above custom datastore:
% Initialize datastore
frameSize = 1024 * 1024; % Example frame size
totalFrames = 50000; % Example total number of frames
ds = CustomFrameDatastore('largefile.bin', frameSize, totalFrames);
% Shuffle and read frames
ds.shuffle();
while hasdata(ds)
frameData = ds.read();
% Process frameData
end
I hope this helps!
