How do query timestamps when reading fileDatastore

Question

Dharmesh Joshi 2024-1-24

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/2074001-how-do-query-timestamps-when-reading-filedatastore

回答： Walter Roberson 2024-1-24

Hello,

I am saving a substantial amount of RAW sensor data at specific intervals into individual files. Then, I am using `fileDatastore` to read all those files and compile them into one variable, as demonstrated below. The issue I'm facing is that I have data spanning over a year, but I only want to read the last three months of data. How can I modify my code so that once it starts reading a file, if it encounters timestamps older than three months, it stops progressing through that datastore?

The purpose of this is to speed up my data compilation by avoiding unnecessary data reads.

fds = fileDatastore("Raw Sensor Data\"+deviceid,"ReadFcn",@load,"FileExtensions",".mat")
    readlist=readall(fds)
    readlist_size=size(readlist);
    temp_live_data_downloaded_data=[];
    temp_ref_data_timestamp=[];
    temp_ref_data_sensordata=[];
    for packet = 1:readlist_size(1)
    temp_live_data_downloaded_data =[temp_live_data_downloaded_data  readlist{packet}.live_data_downloaded_data];
    temp_ref_data_timestamp =[temp_ref_data_timestamp  readlist{packet}.ref_data_timestamp];
    temp_ref_data_sensordata =[temp_ref_data_sensordata readlist{packet}.ref_data_sensordata];
    end

3 个评论
显示 1更早的评论隐藏 1更早的评论

Dharmesh Joshi 2024-1-24

When ever i download new sensor data, it is saved in a new .mat file. So each sensor will have its own folder . This folder has many files with in it. But each sensor will have its own folder.

Star Strider 2024-1-24

Are the timestamps part of the file name?

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Walter Roberson 2024-1-24

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/2074001-how-do-query-timestamps-when-reading-filedatastore#answer_1396901

在 MATLAB Online 中打开

fds = fileDatastore("Raw Sensor Data\"+deviceid,"ReadFcn",@load,"FileExtensions",".mat")

Instead of passing in the directory to fileDatastore(), you would pass in a cell array of validated file names.

rdpath = "Raw Sensor Data";
dinfo = dir(fullfile(rdpath, deviceid, "*.mat"));
filenames_only = {dinfo.name};
%ADJUST THIS SECTION
%for the sake of arguement assume the first 8 characters of the file name
%are constant and the timestamp portion ends with "_"
timestamps = regexp(filenames_only, '^(?<=.{8})[^_]*', 'once');
ts_datetime = datetime(timestamps);
now = datetime('now');
now_minus_3 = now - calmonths(3);
mask = isbetween(timestamps, now_minus_3, now);
dinfo = dinfo(mask);
selected_fullnames = fullfile({dinfo.folder}, {dinfo.name});
fds = fileDatastore(selected_fullnames, "ReadFcn", @local);

The portion marked ADJUST THIS SECTION needs to be adjusted to suit your purposes. The code as-is assumes that the first 8 characters of the file name are to be ignored, and that then there is a timestamp that ends at the first "_", and that timestamp can be fed into datetime() directly. This is example code that needs to be changed according to the actual filename structure you have.