Skip reading a file with file datastore
9 次查看(过去 30 天)
显示 更早的评论
I'm using a file datastore to read log files. I'm using a while loop to read files in datastore. In some cases the file reader is not able to read a file and will throw an error. If this happens, the file datastore does not move forward to the next file and is stucked trying to read the same file.
I would like to be able to continue reading the next file in the datastore if this happens.
Are there a nextfile(fds) function or similar that can change the progress of the datastore?
Alternatively I have to make a recursive call for a new file datastore based on the remaing subset of data and merge the output together. I would like to avoid that.
% Read through the file datastore and store data
ii=1;
reset(fds)
hwait = waitbar(0, '0', 'Name', 'Reading data');
while hasdata(fds)
try
Data{ii, :} = read(fds);
catch
% flag file and move to next
flaggedfiles{end+1}
end
ii = ii + 1;
waitbar(progress(fds), hwait, sprintf('%2.0f %%', progress(fds) * 100));
end
0 个评论
采纳的回答
praguna manvi
2024-10-7
编辑:praguna manvi
2024-10-7
Hi Heine Hørup Pedersen,
As I understand, you are trying to achieve the behavior of "next(fds)" on a "FileDatastore" object "fds" during an "Exception" in a loop while updating the "waitbar". Although there is no built-in support to perform a file-skip, "@fcn" can be configured to "partialfile," which allows a "done" flag for this effect after error handling within the "@fcn" function.
Here is an example that simulates this behavior:
% Create a temporary directory
tempDir = tempname;
mkdir(tempDir);
% Create some text files in the directory
file1 = fullfile(tempDir, 'file1.txt');
file2 = fullfile(tempDir, 'file2.bin'); % Binary file to simulate error
file3 = fullfile(tempDir, 'file3.txt');
% Write data to the files
fid1 = fopen(file1, 'w');
fprintf(fid1, '1\n');
fclose(fid1);
fid2 = fopen(file2, 'w');
fwrite(fid2, 'double');
fclose(fid2);
fid3 = fopen(file3, 'w');
fprintf(fid3, '3\n');
fclose(fid3);
% Create a fileDatastore
fds = fileDatastore(tempDir, 'ReadFcn', @readTextFile, 'FileExtensions', {'.txt', '.bin'});
% Function to read text files
function [data,userdata,done] = readTextFile(filename, userdata)
fid = fopen(filename, 'r');
if fid == -1
error('File cannot be opened.');
end
data = fread(fid, '*char')';
fclose(fid);
try
validateattributes(str2double(data), "double", {"nonnan"});
catch ME
fprintf('Inner Error message: %s\n', ME.message);
end
done = 1;
end
% Initialize variables
ii = 1;
reset(fds);
hwait = waitbar(0, '0', 'Name', 'Reading data');
flaggedfiles = {}; % Initialize the flagged files list
% Read through the file datastore and store data
while hasdata(fds)
try
Data{ii, :} = read(fds);
ii = ii + 1; % Increment the index only if read is successful
catch ME
% flaggedfiles{end+1}
fprintf('Outer Error message: %s\n', ME.message);
end
% Update the waitbar and introduce a short pause
waitbar(progress(fds), hwait, sprintf('%2.0f %%', progress(fds) * 100));
pause(0.5); % Pause for half a second to make the waitbar visible longer
end
close(hwait);
% Display results
disp('Data read from files:');
disp(Data);
For more information about “FileDatastore” properties refer to this link below:
Hope this helps!
更多回答(0 个)
另请参阅
类别
在 Help Center 和 File Exchange 中查找有关 Environment and Settings 的更多信息
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!