Processing Multiple CSV files one at a time and plotting them on the same graph/
50 次查看(过去 30 天)
显示 更早的评论
I wrote a code which takes in two CSV files and synchronizes their data and plots it on a graph. With bigger files, it shows a memory issue and the program does not run. I have tried to split the big CSV file into multiple files and I want the program to read it one by one (not storing the data since doing this also lead to program failure with memory overload). I have attached my very first code (which worked fine for smaller CSV files) and the second code that I modified for larger memory files but I still cant seem to run it.
INITIAL CODE
%Initial Code
%reading and modifying the ring Data
sleeponData = readtable('x.csv');
sleeponTimestamp = sleeponData(:,"time");
sleeponTimestamparray = table2array(sleeponTimestamp);
sleeponSportRaw = table2array(sleeponData(:,"sportRaw"));
sleeponSportRaw = 5000*sleeponSportRaw; %use plotyy left for sensor y for ring
%converting epoch time to datetime
sleep_on_dt = datetime(sleeponData.time, 'ConvertFrom','epochtime','TicksPerSecond',1e3, 'Format', 'dd-MM-yyyy HH:mm:ss.SSSSSSSSS','TimeZone','+09:00');
sleepon_dt_shifted = sleep_on_dt + hours(9);
sleeponData.time = sleepon_dt_shifted;
%reading the pillow
data = importdata('sensordata.csv');
data(:,2) = [];
baseline = data(1,:);
baseline(1) = 0;
minus_bl = data - baseline;
minus_bl(end,:) = [];
tabledata = array2table(minus_bl);
%pillowTime = array2table(data(:,1));
sleeponTT = table2timetable(sleeponData);
pillow_time = datetime(tabledata.minus_bl1, 'ConvertFrom','posixtime','Format', 'dd-MM-yyyy HH:mm:ss.SSSSSSSSS','TimeZone','Asia/Tokyo');
tabledata.data1 = pillow_time;
pilowTT = table2timetable(tabledata);
%synchronizing the timestamp
%new = synchronize(sleeponData, data, 'secondly','linear');
new = synchronize(sleeponTT, pilowTT);
% plotyy(new.time, new.sportRaw, new.time, new{:,3:end});
figure
hold on
x = new.time;
y = new.sportRaw;
yyaxis right;
plot(x,y,'-x','Color', [.9 .4 1], 'markersize',15 );
ylabel('Movement');
title('Participant x');
z = new{:,3:end};
yyaxis left;
plot(x,z, '-');
ylabel('Sensor Readings');
MODIFIED CODE BELOW:
%MODIFIED CODE
%reading and plotting the Ring Data
sleeponData = readtable('sleepring.csv'); %small file
sleeponTimestamp = sleeponData(:,"time");
sleeponTimestamparray = table2array(sleeponTimestamp);
sleeponSportRaw = table2array(sleeponData(:,"sportRaw"));
sleeponSportRaw = 5000*sleeponSportRaw; %use plotyy left for sensor y for ring
%converting epoch time to datetime
sleep_on_dt = datetime(sleeponData.time, 'ConvertFrom','epochtime','TicksPerSecond',1e3, 'Format', 'dd-MM-yyyy HH:mm:ss.SSSSSSSSS','TimeZone','+09:00');
sleepon_dt_shifted = sleep_on_dt + hours(9);
sleeponData.time = sleepon_dt_shifted;
%reading the multiple sensor files
data = [];
filename = "sensordata_%i.csv";
figure
title('Sensor Data 1');
hold on
for i = 1:100;
new_filename = sprintf(filename, i);
if isfile(new_filename)
newdata = importdata(new_filename);
newdata(:,2) = [];
data = [data; newdata];
baseline = data(1,:);
baseline(1) = 0;
minus_bl = data - baseline;
minus_bl(end,:) = [];
tabledata = array2table(minus_bl);
sleeponTT = table2timetable(sleeponData);
pillow_time = datetime(tabledata.minus_bl1, 'ConvertFrom','posixtime','Format', 'dd-MM-yyyy HH:mm:ss.SSSSSSSSS','TimeZone','Asia/Tokyo');
tabledata.data1 = pillow_time;
pilowTT = table2timetable(tabledata);
new = synchronize(sleeponTT, pilowTT);
x = new.time;
y = new.sportRaw;
yyaxis right;
plot(x,y,'-x','Color', [.9 .4 1], 'markersize',15 );
ylabel('Movement');
z = new{:,3:end};
yyaxis left;
plot(x,z, '-');
ylabel('Sensor Readings');
else
break;
end
end
2 个评论
Eric Sofen
2022-3-21
I'm not sure what's wrong with your new code. It will be easier to diagnose with some sample data.
That said, this sounds like a situation wheretall arrays and tables might be useful. They use lazy evaluation to work with data that doesn't fit in memory.
回答(0 个)
另请参阅
类别
在 Help Center 和 File Exchange 中查找有关 Timetables 的更多信息
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!