Data normalization with threshold

Question

MattC 2023-3-5

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1923235-data-normalization-with-threshold

评论： Star Strider 2023-3-5

Hi, I am interested in normalizing my data between 0 and 1. However, I do have a threshold value/limit value that I would like to use while normalizing. Example: data: [10,30,50,75], threshold value: [70]. Good data is when it does not exceed 70. I am not sure how to incorporate this threshold while normalizing data. Please help

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Star Strider 2023-3-5

1
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1923235-data-normalization-with-threshold#answer_1185415

在 MATLAB Online 中打开

I am not certain what you want to do.

Perhaps one of these —

data = [10,30,50,75]; 
threshold_value = [70];
Result = normalize(data(data<=threshold_value), 'range',[0 1])                              % Scale Only The Data To Be Considered
Result = 1×3
         0    0.5000    1.0000
B  = [70 1; 0 1] \ [1; 0];                                                                  % Scaling Parameters
Result = [data(data<=threshold_value).' ones(size(data(data<=threshold_value).'))] * B      % Scale [0 70] —> [0 1]
Result = 3×1
    0.1429
    0.4286
    0.7143

The first scales only the data that are less than 70 to [0 1]. The second scales [0 70] to [0 1], regardless of the data, although both limit the data to [0 70].

.

20 个评论
显示 18更早的评论隐藏 18更早的评论

Star Strider 2023-3-5

在 MATLAB Online 中打开

I would definitely call this scaling, not normalising, however I doubt that it makes much of a difference.

Plotting it is straightforward.

Try omething like this —

Scaled = @(data,Th) [data(:) ones(size(data(:)))] * ([Th 1; 0 1] \ [1; 0]); % Scale Data By Threshold

A = [10,30,50,75];

B = [30,50,75,95];

C = [15,35,50,75];

A_Threshold = [70];

B_Threshold = [80];

C_Threshold = [85];

A_Scaled = Scaled(A,A_Threshold);

B_Scaled = Scaled(B,B_Threshold);

C_Scaled = Scaled(C,C_Threshold);

y = [A_Scaled, B_Scaled, C_Scaled];

x = ones(size(y,1),1)*(1:size(y,2));

rgb = 'bgr';

cmy = 'cmy';

xtl = {'A','B','C'};

cgttt = [0.9 0.5 0.3];

figure

hold on

for k = 1:size(y,2)

L = y(:,k)<=1;

hp1{k} = plot(x(L,k), y(L,k), 's', 'Color',rgb(k), 'MarkerSize', 10, 'MarkerFaceColor',rgb(k), 'DisplayName',sprintf('%s <= Threshold',xtl{k}));

hp2{k} = plot(x(~L,k), y(~L,k), 's', 'Color',cgttt, 'MarkerSize', 10, 'MarkerFaceColor',cgttt, 'DisplayName','> Threshold');

end

xlim([0 4])

set(gca, 'XTick',1:3, 'XTickLabel',xtl)

hold off

yl = yline(1, '--r', 'Threshold', 'LineWidth',2, 'DisplayName','Threshold');

legend([hp1{:} hp2{1} yl], 'Location','eastoutside')

The legend you want appears to me to be conflicting. I did my best with it.

.

MattC 2023-3-5

编辑：MattC 2023-3-5

在 MATLAB Online 中打开

Thank you @Star Strider. I do have a follow up question. So, the data I have is per day basis and I am trying to incorporate that into the plots.

So, let's say

%% Day 1
A = [10,30,50,75]; 
B = [30,50,75,95]; 
C = [15,35,50,75];
%% Day 2
A = [20,40,60,65]; 
B = [40,70,75,95]; 
C = [10,30,45,75];

The threholds remain the same across each day for A,B,C. There is no change in that.

I am not sure how to make the variables A,B,C in such a way that it retains values from previous day while still adding the new day's value. I want to keep the scale dynamic for x. If there is data for day1, day2 then x axis would show that for now but when data for day 3 comes it would automatically be added to the plot
I am trying to incorporate this in the plots for each day. So, when we have this A,B,C in the x axis that would create an issue because I cannot show this for other day.

Example of what I am trying to do is something like this: (Day 1 having A,B,C scaled values all lined up vertically instead of them spread across x axis)

Star Strider 2023-3-5

在 MATLAB Online 中打开

Adapting my code to multiple days was a challenge. This works for two days and three data sets. To adapt it to more days will require unique markers for the additional days.

Taking a guess, since I’m not certain what you are asking —

Scaled = @(data,Th) [data(:) ones(size(data(:)))] * ([Th 1; 0 1] \ [1; 0]); % Scale Data By Threshold

Day{1} = [10,30,50,75;

30,50,75,95;

15,35,50,75];

Day{2} = [20,40,60,65;

40,70,75,95;

10,30,45,75];

A_Threshold = 70;

B_Threshold = 80;

C_Threshold = 85;

Thrshld = [A_Threshold; B_Threshold; C_Threshold];

for k1 = 1:numel(Day)

for k2 = 1:size(Thrshld,1)

DayScaled{k1}(k2,:) = Scaled(Day{k1}(k2,:),Thrshld(k2));

end

Day1 = array2table([DayScaled{1}.'], 'VariableNames',{'A','B','C'});

Day2 = array2table([DayScaled{2}.'], 'VariableNames',{'A','B','C'});

ScaledResult = table(Day1,Day2, 'VariableNames',{'Day 1','Day 2'})

ScaledResult = 4×2 table

Day 1 Day 2 A B C A B C ____________________________ ____________________________ 0.14286 0.375 0.17647 0.28571 0.5 0.11765 0.42857 0.625 0.41176 0.57143 0.875 0.35294 0.71429 0.9375 0.58824 0.85714 0.9375 0.52941 1.0714 1.1875 0.88235 0.92857 1.1875 0.88235

x = ones(size([Day{1}],2),1)*(1:size([Day{1}],1));

rgb = 'bgr';

cmy = 'cmy';

xtl = {'A','B','C'};

mk = {'s','d'};

cgttt = [0.9 0.5 0.3];

figure

hold on

for k1 = 1:numel(Day)

for k2 = 1:size(Thrshld,1)

y = DayScaled{k1}(k2,:);

L = y<=1;

hp1{k1,k2} = plot(x(L,k2), y(L), mk{k1}, 'Color',rgb(k2), 'MarkerSize', 10, 'MarkerFaceColor',rgb(k2), 'DisplayName',sprintf('%s \\leq Threshold, Day %d',xtl{k2},k1));

hp2{k1,k2} = plot(x(~L,k2), y(~L), mk{k1}, 'Color',cgttt, 'MarkerSize', 10, 'MarkerFaceColor',cgttt, 'DisplayName',sprintf('> Threshold Day %d',k1));

end

xlim([0 4])

set(gca, 'XTick',1:3, 'XTickLabel',xtl)

hold off

yl = yline(1, '--r', 'Threshold', 'LineWidth',2, 'DisplayName','Threshold');

legend([hp1{:} hp2{:,[2 3]} yl], 'Location','eastoutside')

This is the best I can do.

.

Star Strider 2023-3-5

在 MATLAB Online 中打开

How would you plot the day count and keep the ‘A’-‘C’ as well? If you do not want to keep tha ‘A’-‘C’ designations, then plotting as a function of the days is straigtforward. Complications arise in plotting both of them, although that is a possibility as well —

Scaled = @(data,Th) [data(:) ones(size(data(:)))] * ([Th 1; 0 1] \ [1; 0]); % Scale Data By Threshold

Day{1} = [10,30,50,75;

30,50,75,95;

15,35,50,75];

Day{2} = [20,40,60,65;

40,70,75,95;

10,30,45,75];

A_Threshold = 70;

B_Threshold = 80;

C_Threshold = 85;

Thrshld = [A_Threshold; B_Threshold; C_Threshold];

for k1 = 1:numel(Day)

for k2 = 1:size(Thrshld,1)

DayScaled{k1}(k2,:) = Scaled(Day{k1}(k2,:),Thrshld(k2));

end

Day1 = array2table([DayScaled{1}.'], 'VariableNames',{'A','B','C'});

Day2 = array2table([DayScaled{2}.'], 'VariableNames',{'A','B','C'});

ScaledResult = table(Day1,Day2, 'VariableNames',{'Day 1','Day 2'})

ScaledResult = 4×2 table

Day 1 Day 2 A B C A B C ____________________________ ____________________________ 0.14286 0.375 0.17647 0.28571 0.5 0.11765 0.42857 0.625 0.41176 0.57143 0.875 0.35294 0.71429 0.9375 0.58824 0.85714 0.9375 0.52941 1.0714 1.1875 0.88235 0.92857 1.1875 0.88235

x = ones(size([Day{1}],2),1)*(1:size([Day{1}],1));

rgb = 'bgr';

cmy = 'cmy';

xtl = {'A','B','C'};

mk = {'s','d'};

cgttt = [0.9 0.5 0.3];

figure

hold on

for k1 = 1:numel(Day)

for k2 = 1:size(Thrshld,1)

y = DayScaled{k1}(k2,:);

L = y<=1;

hp1{k1,k2} = plot(x(L,k2)+3*(k1-1), y(L), mk{k1}, 'Color',rgb(k2), 'MarkerSize', 10, 'MarkerFaceColor',rgb(k2), 'DisplayName',sprintf('%s \\leq Threshold, Day %d',xtl{k2},k1));

hp2{k1,k2} = plot(x(~L,k2)+3*(k1-1), y(~L), mk{k1}, 'Color',cgttt, 'MarkerSize', 10, 'MarkerFaceColor',cgttt, 'DisplayName',sprintf('> Threshold Day %d',k1));

end

xlim([0 6])

set(gca, 'XTick',1:6, 'XTickLabel',xtl)

text(2:3:6, ones(1,2)*min(ylim)-0.075*diff(ylim), compose('Day %d',1:2), 'Horiz','center', 'Vert','top')

hold off

yl = yline(1, '--r', 'Threshold', 'LineWidth',2, 'DisplayName','Threshold');

legend([hp1{:} hp2{:,[2 3]} yl], 'Location','northoutside', 'NumColumns',3, 'FontSize',8)

I encourage you to experiment with this approach. I am noit certaion how easily it scales to more days, however I will defer to you for that.

.

Star Strider 2023-3-5

在 MATLAB Online 中打开

‘How do we plot these 3 values vertically against the threshold value 1 instead of horizontally which we have now?’

I am not certain that I understand what you want to do.

Perhaps this —

Scaled = @(data,Th) [data(:) ones(size(data(:)))] * ([Th 1; 0 1] \ [1; 0]); % Scale Data By Threshold

Day{1} = [10,30,50,75;

30,50,75,95;

15,35,50,75];

Day{2} = [20,40,60,65;

40,70,75,95;

10,30,45,75];

A_Threshold = 70;

B_Threshold = 80;

C_Threshold = 85;

Thrshld = [A_Threshold; B_Threshold; C_Threshold];

for k1 = 1:numel(Day)

for k2 = 1:size(Thrshld,1)

DayScaled{k1}(k2,:) = Scaled(Day{k1}(k2,:),Thrshld(k2));

end

Day1 = array2table([DayScaled{1}.'], 'VariableNames',{'A','B','C'});

Day2 = array2table([DayScaled{2}.'], 'VariableNames',{'A','B','C'});

ScaledResult = table(Day1,Day2, 'VariableNames',{'Day 1','Day 2'})

ScaledResult = 4×2 table

Day 1 Day 2 A B C A B C ____________________________ ____________________________ 0.14286 0.375 0.17647 0.28571 0.5 0.11765 0.42857 0.625 0.41176 0.57143 0.875 0.35294 0.71429 0.9375 0.58824 0.85714 0.9375 0.52941 1.0714 1.1875 0.88235 0.92857 1.1875 0.88235

x = ones(size([Day{1}],2),1)*(1:size([Day{1}],1));

rgb = 'bgr';

cmy = 'cmy';

xtl = {'A','B','C'};

mk = {'s','d'};

cgttt = [0.9 0.5 0.3];

figure

hold on

for k1 = 1:numel(Day)

for k2 = 1:size(Thrshld,1)

y = DayScaled{k1}(k2,:);

L = y<=1;

hp1{k1,k2} = plot(x(L,k1), y(L), mk{k1}, 'Color',rgb(k2), 'MarkerSize', 10, 'MarkerFaceColor',rgb(k2), 'DisplayName',sprintf('%s \\leq Threshold, Day %d',xtl{k2},k1));

hp2{k1,k2} = plot(x(~L,k1), y(~L), mk{k1}, 'Color',cgttt, 'MarkerSize', 10, 'MarkerFaceColor',cgttt, 'DisplayName',sprintf('> Threshold Day %d',k1));

end

xlim([0 3])

% set(gca, 'XTick',1:6, 'XTickLabel',xtl)

set(gca, 'XTick',1:2, 'XTickLabel',compose('Day %d',1:2))

% text(2:3:6, ones(1,2)*min(ylim)-0.075*diff(ylim), compose('Day %d',1:2), 'Horiz','center', 'Vert','top')

hold off

yl = yline(1, '--r', 'Threshold', 'LineWidth',2, 'DisplayName','Threshold');

legend([hp1{:} hp2{:,[2 3]} yl], 'Location','northoutside', 'NumColumns',3, 'FontSize',8)

I’ll keep posting new approaches until I’ve exhausted all the possibilities. The ‘A’-‘C’ designations still appear in the legend. With many more days, though, that could quickly get cumbersome.

.

Star Strider 2023-3-5

在 MATLAB Online 中打开

O.K., so we¹re now scaling single values?

Scaling single values is straightforward. I don’t even have to change the code, other than to use only one marker for all the points.

Perhaps this —

Scaled = @(data,Th) [data(:) ones(size(data(:)))] * ([Th 1; 0 1] \ [1; 0]); % Scale Data By Threshold

Day{1} = [10;

30;

15];

Day{2} = [20;

40;

10];

A_Threshold = 70;

B_Threshold = 80;

C_Threshold = 85;

Thrshld = [A_Threshold; B_Threshold; C_Threshold];

for k1 = 1:numel(Day)

for k2 = 1:size(Thrshld,1)

DayScaled{k1}(k2,:) = Scaled(Day{k1}(k2,:),Thrshld(k2));

end

Day1 = array2table([DayScaled{1}.'], 'VariableNames',{'A','B','C'});

Day2 = array2table([DayScaled{2}.'], 'VariableNames',{'A','B','C'});

ScaledResult = table(Day1,Day2, 'VariableNames',{'Day 1','Day 2'})

ScaledResult = 1×2 table

Day 1 Day 2 A B C A B C ___________________________ _________________________ 0.14286 0.375 0.17647 0.28571 0.5 0.11765

x = ones(size([Day{1}],2),1)*(1:size([Day{1}],1));

rgb = 'bgr';

cmy = 'cmy';

xtl = {'A','B','C'};

mk = {'s','d'};

cgttt = [0.9 0.5 0.3];

figure

hold on

for k1 = 1:numel(Day)

for k2 = 1:size(Thrshld,1)

y = DayScaled{k1}(k2,:);

L = y<=1;

hp1{k1,k2} = plot(x(L,k1), y(L), mk{1}, 'Color',rgb(k2), 'MarkerSize', 10, 'MarkerFaceColor',rgb(k2), 'DisplayName',sprintf('%s \\leq Threshold, Day %d',xtl{k2},k1));

hp2{k1,k2} = plot(x(~L,k1), y(~L), mk{1}, 'Color',cgttt, 'MarkerSize', 10, 'MarkerFaceColor',cgttt, 'DisplayName',sprintf('> Threshold Day %d',k1));

end

xlim([0 3])

% set(gca, 'XTick',1:6, 'XTickLabel',xtl)

set(gca, 'XTick',1:2, 'XTickLabel',compose('Day %d',1:2))

% text(2:3:6, ones(1,2)*min(ylim)-0.075*diff(ylim), compose('Day %d',1:2), 'Horiz','center', 'Vert','top')

hold off

yl = yline(1, '--r', 'Threshold', 'LineWidth',2, 'DisplayName','Threshold');

legend([hp1{:} hp2{:,[2 3]} yl], 'Location','northoutside', 'NumColumns',3, 'FontSize',8)

.

Star Strider 2023-3-5

在 MATLAB Online 中打开

Perhaps this —

Scaled = @(data,Th) [data(:) ones(size(data(:)))] * ([Th 1; 0 1] \ [1; 0]); % Scale Data By Threshold

Day{1} = [10;

30;

15];

Day{2} = [20;

40;

10];

A_Threshold = 70;

B_Threshold = 80;

C_Threshold = 85;

Thrshld = [A_Threshold; B_Threshold; C_Threshold];

for k1 = 1:numel(Day)

for k2 = 1:size(Thrshld,1)

DayScaled{k1}(k2,:) = Scaled(Day{k1}(k2,:),Thrshld(k2));

end

Day1 = array2table([DayScaled{1}.'], 'VariableNames',{'A','B','C'});

Day2 = array2table([DayScaled{2}.'], 'VariableNames',{'A','B','C'});

ScaledResult = table(Day1,Day2, 'VariableNames',{'Day 1','Day 2'})

ScaledResult = 1×2 table

Day 1 Day 2 A B C A B C ___________________________ _________________________ 0.14286 0.375 0.17647 0.28571 0.5 0.11765

x = ones(size([Day{1}],2),1)*(1:size([Day{1}],1));

rgb = 'grb';

cmy = 'cmy';

xtl = {'A','B','C'};

mk = {'s','d'};

cgttt = [0.9 0.5 0.3];

figure

hold on

for k1 = 1:numel(Day)

for k2 = 1:size(Thrshld,1)

y = DayScaled{k1}(k2,:);

L = y<=1;

hp1{k1,k2} = plot(x(L,k1), y(L), mk{1}, 'Color',rgb(k2), 'MarkerSize', 10, 'MarkerFaceColor',rgb(k2), 'DisplayName',sprintf('%s \\leq Threshold',xtl{k2}));

% hp2{k1,k2} = plot(x(~L,k1), y(~L), mk{1}, 'Color',cgttt, 'MarkerSize', 10, 'MarkerFaceColor',cgttt, 'DisplayName',sprintf('> Threshold Day %d',k1));

end

xlim([0 3])

% set(gca, 'XTick',1:6, 'XTickLabel',xtl)

set(gca, 'XTick',1:2, 'XTickLabel',compose('Day %d',1:2))

% text(2:3:6, ones(1,2)*min(ylim)-0.075*diff(ylim), compose('Day %d',1:2), 'Horiz','center', 'Vert','top')

hold off

yl = yline(1, '--r', 'Threshold', 'LineWidth',2, 'DisplayName','Threshold');

% legend([hp1{:} hp2{:,[2 3]} yl], 'Location','northoutside', 'NumColumns',3, 'FontSize',8)

legend([hp1{1,:} yl], 'Location','northoutside', 'NumColumns',4, 'FontSize',8)

That should scale to more days without much difficulty.

.

Star Strider 2023-3-5

编辑：Star Strider 2023-3-5

在 MATLAB Online 中打开

Try this —

Scaled = @(data,Th) [data(:) ones(size(data(:)))] * ([Th 1; 0 1] \ [1; 0]); % Scale Data By Threshold

Day{1} = [90;

30;

15];

Day{2} = [20;

40;

90];

A_Threshold = 70;

B_Threshold = 80;

C_Threshold = 85;

Thrshld = [A_Threshold; B_Threshold; C_Threshold];

for k1 = 1:numel(Day)

for k2 = 1:size(Thrshld,1)

DayScaled{k1}(k2,:) = Scaled(Day{k1}(k2,:),Thrshld(k2));

end

% Day1 = array2table([DayScaled{1}].', 'VariableNames',{'A','B','C'});

% Day2 = array2table([DayScaled{2}].', 'VariableNames',{'A','B','C'});

% ScaledResult = table(Day1,Day2, 'VariableNames',{'Day 1','Day 2'})

ScaledResult = array2table(cell2mat(DayScaled), 'VariableNames',compose('Day %3d',1:numel(DayScaled)), 'RowNames',{'A','B','C'})

ScaledResult = 3×2 table

Day 1 Day 2 _______ _______ A 1.2857 0.28571 B 0.375 0.5 C 0.17647 1.0588

x = ones(size([Day{1}],2),1)*(1:size([Day{1}],1));

rgb = 'grb';

cmy = 'cmy';

xtl = {'A','B','C'};

mk = {'s','d'};

cgttt = [0.9 0.5 0.3];

figure

hold on

for k1 = 1:numel(Day)

for k2 = 1:size(Thrshld,1)

hp0{k2} = plot(NaN, 1, mk{1}, 'Color',rgb(k2), 'MarkerSize', 10, 'MarkerFaceColor',rgb(k2), 'DisplayName',sprintf('%s',xtl{k2}));

y = DayScaled{k1}(k2,:);

L = y<=1;

hp1{k1,k2} = plot(x(L,k1), y(L), mk{1}, 'Color',rgb(k2), 'MarkerSize', 10, 'MarkerFaceColor',rgb(k2), 'DisplayName',sprintf('%s',xtl{k2}));

% hp1{k1,k2} = plot(x(L,k1), y(L), mk{1}, 'Color',rgb(k2), 'MarkerSize', 10, 'MarkerFaceColor',rgb(k2), 'DisplayName',sprintf('%s \\leq Threshold',xtl{k2}));

hp2{k1,k2} = plot(x(~L,k1), y(~L), mk{1}, 'Color',rgb(k2), 'MarkerSize', 10, 'MarkerFaceColor',rgb(k2), 'DisplayName',sprintf('> Threshold'));

end

xlim([0 3])

% set(gca, 'XTick',1:6, 'XTickLabel',xtl)

set(gca, 'XTick',1:2, 'XTickLabel',compose('Day %d',1:2))

% text(2:3:6, ones(1,2)*min(ylim)-0.075*diff(ylim), compose('Day %d',1:2), 'Horiz','center', 'Vert','top')

hold off

yl = yline(1, '--r', 'Threshold', 'LineWidth',2, 'DisplayName','Threshold');

% legend([hp1{:} hp2{:,[2 3]} yl], 'Location','northoutside', 'NumColumns',3, 'FontSize',8)

% legend([hp1{1,:} hp2{1,:} yl], 'Location','northoutside', 'NumColumns',4, 'FontSize',8)

legend([hp0{:} yl], 'Location','northoutside', 'NumColumns',4, 'FontSize',8)

I changed the ‘Day’ cell arrays slightly to test this. This plots all of them, and any above the threshold, giving them their appropriate groups, regardless.

EDIT — (5 Mar 2023 at 23:44)

Changed the format of the ‘ScaledResult’ table. Code otherwise unchanged.

.

MattC 2023-3-5

Thank you for all the help @Star Strider. If there a way I could put in a recognition for helping me solve this problem I would :)

Star Strider 2023-3-5

As always, my pleasure!

Accepting and voting for it is enough. Reputation points are how recognition is provided.

This will also be for other people with similar problems. For my part, I wrote the ‘Scaled’ anonymous function that I expanded to an anonymous function that can do a bivariate linear regression in a single line.

I also edited the code in my previous Comment so that the ‘ScaledResult’ table now has row names for ‘A’-‘C’, with the variable names being the days. I originally set it up for the earlier data, and now with different single data it needs to have a different format. The code in that Comment is otherwise unchanged.

.

请先登录，再进行评论。