Writing a tall array / datastore to a .dat

Question

Stefan Oline 2021-3-9

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/767787-writing-a-tall-array-datastore-to-a-dat

评论： Divya Gaddipati 2021-5-13

Hello, I'm having trouble writing a large .dat file. It's composed of 64 x 1GB files. I attempted to follow the method here, however I'm having trouble writing the output (a 64 x 407297536 matrix) to a .dat which will end up being ~64GB.

Is there an easy way to write a giant .dat?

Thanks very much.

%% User inputs
channels = 1:64;
demed_flag = 1;
store_flag = 1;
% Choose a directory to store the files
outDir  = 'H:\Falkner_lab\Ephys\2020.07.16_Mouse2357\2020.08.28\tall_eg';
writeDir = 'H:\Falkner_lab\Ephys\2020.07.16_Mouse2357\2020.08.28\tall_eg\write';
%% Setup
n_channels = length(channels);
% Check how many samplesa are in a single channel
[Sample_check, Header] = Nlx2MatCSC(['CSC' num2str(channels(1),'%02.f') '.ncs'],...
        [0 0 0 0 1], 1, 1, [] );
temp_a = reshape(Sample_check,[],1)';
n_samples = length(temp_a);
clear Header Sample_check temp_a
%% Import .ncs data files from the channels list to individual .mat files
fprintf('*Importing data*\n')
for i=1:n_channels
    disp(['Importing channel ' num2str(i) ' of ' num2str(n_channels) ' (' ...
        num2str(i/n_channels*100,2) '%)'])
    %fprintf('i = %.0f\n', i )
    [Samples, Header] = Nlx2MatCSC(['CSC' num2str(channels(i),'%02.f') '.ncs'],...
        [0 0 0 0 1], 1, 1, [] );
    data = reshape(Samples,[],1)';
    % Choose a file name - ensure these progress in order
    fname = fullfile(outDir, sprintf('data_%05d.mat', channels(i)));
    % Save the data and increment counters
    save(fname, 'data', '-v7.3');
    
    clear Samples Header data fname 
end
clear i
fprintf('*Importing data complete*\n');
%% Create a datastore from the files
% Read the data back in as a tall array. First create a datastore ...
fprintf('*Creating a datastore*\n');
ds = fileDatastore(fullfile(outDir, '*.mat'), ...
    'ReadFcn', @(fname) getfield(load(fname), 'data'), ...
    'UniformRead', true);
fprintf('*Creating a datastore complete*\n');
% ... and then a tall array
fprintf('*Storing the datastore in a tall array*\n');
tdata = tall(ds);
fprintf('*Storing the datastore in a tall array complete*\n');
%% Write to file for KS2
if store_flag == 1
    
    fprintf('*Writing data*\n');
    fid = fopen('myNewFile_zeros.dat', 'w');
    fwrite(fid,tdata, 'int16'); % Code breaks here********************
    fclose(fid);
    fprintf('*Writing data complete*\n');
    beep
end