Skip reading a file with file datastore

9 次查看(过去 30 天)
I'm using a file datastore to read log files. I'm using a while loop to read files in datastore. In some cases the file reader is not able to read a file and will throw an error. If this happens, the file datastore does not move forward to the next file and is stucked trying to read the same file.
I would like to be able to continue reading the next file in the datastore if this happens.
Are there a nextfile(fds) function or similar that can change the progress of the datastore?
Alternatively I have to make a recursive call for a new file datastore based on the remaing subset of data and merge the output together. I would like to avoid that.
% Read through the file datastore and store data
ii=1;
reset(fds)
hwait = waitbar(0, '0', 'Name', 'Reading data');
while hasdata(fds)
try
Data{ii, :} = read(fds);
catch
% flag file and move to next
flaggedfiles{end+1}
end
ii = ii + 1;
waitbar(progress(fds), hwait, sprintf('%2.0f %%', progress(fds) * 100));
end

采纳的回答

praguna manvi
praguna manvi 2024-10-7
编辑:praguna manvi 2024-10-7
Hi Heine Hørup Pedersen,
As I understand, you are trying to achieve the behavior of "next(fds)" on a "FileDatastore" object "fds" during an "Exception" in a loop while updating the "waitbar". Although there is no built-in support to perform a file-skip, "@fcn" can be configured to "partialfile," which allows a "done" flag for this effect after error handling within the "@fcn" function.
Here is an example that simulates this behavior:
% Create a temporary directory
tempDir = tempname;
mkdir(tempDir);
% Create some text files in the directory
file1 = fullfile(tempDir, 'file1.txt');
file2 = fullfile(tempDir, 'file2.bin'); % Binary file to simulate error
file3 = fullfile(tempDir, 'file3.txt');
% Write data to the files
fid1 = fopen(file1, 'w');
fprintf(fid1, '1\n');
fclose(fid1);
fid2 = fopen(file2, 'w');
fwrite(fid2, 'double');
fclose(fid2);
fid3 = fopen(file3, 'w');
fprintf(fid3, '3\n');
fclose(fid3);
% Create a fileDatastore
fds = fileDatastore(tempDir, 'ReadFcn', @readTextFile, 'FileExtensions', {'.txt', '.bin'});
% Function to read text files
function [data,userdata,done] = readTextFile(filename, userdata)
fid = fopen(filename, 'r');
if fid == -1
error('File cannot be opened.');
end
data = fread(fid, '*char')';
fclose(fid);
try
validateattributes(str2double(data), "double", {"nonnan"});
catch ME
fprintf('Inner Error message: %s\n', ME.message);
end
done = 1;
end
% Initialize variables
ii = 1;
reset(fds);
hwait = waitbar(0, '0', 'Name', 'Reading data');
flaggedfiles = {}; % Initialize the flagged files list
% Read through the file datastore and store data
while hasdata(fds)
try
Data{ii, :} = read(fds);
ii = ii + 1; % Increment the index only if read is successful
catch ME
% flaggedfiles{end+1}
fprintf('Outer Error message: %s\n', ME.message);
end
% Update the waitbar and introduce a short pause
waitbar(progress(fds), hwait, sprintf('%2.0f %%', progress(fds) * 100));
pause(0.5); % Pause for half a second to make the waitbar visible longer
end
close(hwait);
% Display results
disp('Data read from files:');
disp(Data);
For more information about “FileDatastore” properties refer to this link below:
Hope this helps!
  1 个评论
Heine Hørup Pedersen
Thank you Praguna
The solution lies in handling the error with try, catch in the reader file as you suggest.

请先登录,再进行评论。

更多回答(0 个)

类别

Help CenterFile Exchange 中查找有关 Environment and Settings 的更多信息

产品


版本

R2023b

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by