Plot exceeding time limit due to large dataset

Question

Aditya 2024-10-4

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/2157370-plot-exceeding-time-limit-due-to-large-dataset

评论： Voss 2024-10-5

采纳的回答： Voss

在 MATLAB Online 中打开

I have a dataset in which I have to categorize the rise of UV levels from a satellite.

As this satellite orbits the Earth there are periods where it cannot process the UV levels.

The raw data that I am getting is close to a square wave.

I have to categorize this data into rising where the satellite comes out of the influence of the Earth,

OnDuty where the satellite is able to record data properly and falling where it is going behind the Earth.

The dataset contains a timestamp of frequency 1Hz where one value is recorded every second.

I want to categorize these three classes into for atleast now, red, green and blue.

The dataset contains the said timestamp and the corresponding UV values.

I am currently using a threshold to categorize the data and is working for a smaller dataset, however with a dataset of 80000 entry points my code is not able to run efficiently.

Here is the code. Where states stores the ith values nature. The problem lies within the for loop that runs for the plotting.

function categorize_square_wave()
    y_foo = readtable("80000entries.csv", Range='uv_values');
    y_values = table2array(y_foo);
    k = 1;  % Sensitivity parameter
    differences = diff(y_values);
   
    % comments have been put to remove staticstical computation
    mu = mean(differences);
    sigma = std(differences);
    
    % Thresholds
    T_high = mu + k * sigma;
    T_low = mu - k * sigma;
    
    % coded directly to reduce computation
    % T_high = 500;
    % T_low = -100;
    % Categorizing based on threshold values
    % Initialize states with zeros (default to dutiful)
    states = zeros(1, length(differences));  
    
    % Assign states using vectorization
    states(differences > T_high) = 1;  % Rising
    states(differences < T_low) = -1;   % Falling
    
    % Collect dutiful values
    % dutiful_values = y_values(states == 0);
    
    
    figure;
    hold on;
    
    for i = 1:length(differences)
        if states(i) == 1
            plot([i, i+1], [y_values(i), y_values(i+1)], 'g', 'LineWidth', 2);  % Green for Rising
        elseif states(i) == -1
            plot([i, i+1], [y_values(i), y_values(i+1)], 'r', 'LineWidth', 2);  % Red for Falling
        else
            plot([i, i+1], [y_values(i), y_values(i+1)], 'b', 'LineWidth', 2);  % Blue for Dutiful
            % dutiful_values = [dutiful_values, y_values(i)];  % Collect Dutiful values
        end
    end
    
    % Plot original values in dashed lines
    plot(y_values, 'k--', 'LineWidth', 1);  % Black dashed line for original values
    yline(T_high, 'r--', 'T_{high}', 'LabelVerticalAlignment', 'bottom', 'LabelHorizontalAlignment', 'right'); % Dashed red line for T_high
    
    xlabel('Sample Number');
    ylabel('y(t)');
    title('Classification of Changes in Square Wave');
    hold off;
end
categorize_square_wave();
% we have a good estimation of what values are 
% is there a way to use flags on each state so that we can just plot
% without checking each value?
% implemetn flags

2 个评论
显示无隐藏无

Saurav 2024-10-4

Can you attach the .csv file for better debugging?

Aditya 2024-10-4

Im sorry that i couldnt attach it, the basics are there is a timestamp corresponding to a uv irradiance level

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Voss 2024-10-4

1
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/2157370-plot-exceeding-time-limit-due-to-large-dataset#answer_1526570

在 MATLAB Online 中打开

You can replace those ~80000 plotted red, green, and blue lines with 3 lines: one red, one green, and one blue.

Use NaNs in the plotted lines where the data is not pertinent to that line, e.g., the green "rising" line will have NaNs wherever the data is not "rising". NaNs don't render on a plotted line so can be used to create gaps in a line.

Here's an example with made-up data consisting of 80000 datapoints:

y_values = min(0.5,max(-0.5,sin(linspace(0,4*pi,80000))));

k = 1; % Sensitivity parameter

differences = diff(y_values);

% comments have been put to remove statistical computation

mu = mean(differences);

sigma = std(differences);

% Thresholds

T_high = mu + k * sigma;

T_low = mu - k * sigma;

% Categorizing based on threshold values

% Initialize states with zeros (default to dutiful)

states = zeros(1, length(differences));

% Assign states using vectorization

states(differences > T_high) = 1; % Rising

states(differences < T_low) = -1; % Falling

% Initialize three vectors of NaNs

N = numel(y_values);

y_rising = NaN(1,N);

y_falling = NaN(1,N);

y_on = NaN(1,N);

% populate the rising data line with y_values where the data is rising

idx = find(states == 1);

y_rising(idx) = y_values(idx);

y_rising(idx+1) = y_values(idx+1);

% populate the falling data line with y_values where the data is falling

idx = find(states == -1);

y_falling(idx) = y_values(idx);

y_falling(idx+1) = y_values(idx+1);

% populate the on-duty data line with y_values where the data is on-duty

idx = find(states == 0);

y_on(idx) = y_values(idx);

y_on(idx+1) = y_values(idx+1);

% plot the three lines

figure

hold on

plot(y_rising,'g','LineWidth',2)

plot(y_falling,'r','LineWidth',2)

plot(y_on,'b','LineWidth',2)

% Plot original values in dashed lines

plot(y_values, 'k--', 'LineWidth', 1); % Black dashed line for original values

yline(T_high, 'r--', 'T_{high}', 'LabelVerticalAlignment', 'bottom', 'LabelHorizontalAlignment', 'right'); % Dashed red line for T_high

xlabel('Sample Number');

ylabel('y(t)');

title('Classification of Changes in Square Wave');

4 个评论
显示 2更早的评论隐藏 2更早的评论

Aditya 2024-10-5

编辑：Aditya 2024-10-5

在 MATLAB Online 中打开

Hey there appears to be an issue when I use a few datasets where the cycles are not exactly square.

Because of this thresholding kinks are being mesured as duty states where they really have to be on the rising edge.

The timestamp is of x_values and a typical orbit is roughly about 90 minutes.

The logic I am thinking of is that after a sequence of consecutive rising states I will just change the next 15 minutes or 15*60 counts of data into the rising edge to factor in the kink and do the same for the kink for the trailng edge. I hope I am clear, I will try attaching an image for the exact issue.

Here is the new x and y values.

    y_foo = readtable("aug33_orbit236_31aug2024_UVA.csv");
    % count_uva = table2array(y_foo.Counts);
    counts_uva = y_foo.Counts;
    timestamp = datetime(y_foo.Time, 'InputFormat', 'dd MMMM yyyy, HH:mm:ss.SS');
    k = 1;  % Sensitivity parameter
    differences = diff(counts_uva);
    

I am so sorry for taking up your time again. Also if there is any other approach to categorize such values?

Voss 2024-10-5

Whatever method you decide to use to categorize the data into the three groups, you can use the approach in my answer to plot three lines.

You might want to post a new question about selecting the best method for categorizing that data.

请先登录，再进行评论。

Plot exceeding time limit due to large dataset

2 个评论
显示无隐藏无

采纳的回答

4 个评论
显示 2更早的评论隐藏 2更早的评论

更多回答（0 个）

另请参阅

类别

标签

产品

版本

Community Treasure Hunt

Plot exceeding time limit due to large dataset

2 个评论 显示 无隐藏 无

采纳的回答

4 个评论 显示 2更早的评论隐藏 2更早的评论

更多回答（0 个）

另请参阅

类别

标签

产品

版本

Community Treasure Hunt

2 个评论
显示无隐藏无

4 个评论
显示 2更早的评论隐藏 2更早的评论