How to recognize 6 colors of a face rubik's cube at the same time?

Question

Sang 2022-10-15

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1827523-how-to-recognize-6-colors-of-a-face-rubik-s-cube-at-the-same-time

评论： DGM 2022-10-28

2 个评论
显示无隐藏无

DGM 2022-10-15

Will all the images have the object consistently positioned within the frame?

Will it always be the same cube and lighting?

You might be able to try sampling regions within each square, finding their average colors, and then trying to find the closest match between the observed colors and a list of known colors.

If the object position or lighting changes a lot, things might get complicated.

Sang 2022-10-15

I understand your problem but I don't know what the command is

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

DGM 2022-10-15

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1827523-how-to-recognize-6-colors-of-a-face-rubik-s-cube-at-the-same-time#answer_1075998

编辑：DGM 2022-10-15

在 MATLAB Online 中打开

Let's start with an example that doesn't solve the problem. Herein, I extract the mean color in each block. Much of this example (the figures) is purely for illustration and wouldn't be necessary in practice. All you need is a correctly ordered color table.

% start with the image

Argb = imread('https://www.mathworks.com/matlabcentral/answers/uploaded_files/1157503/image.jpeg');

% create a mask that samples the centers of the blocks

% stdfilt() is used to find regions of contrast (boundaries of blocks)

% this image is thresholded to create a rough low-contrast region mask

% the image is trimmed and padded with zero to detatch blobs from the border

se = ones(100,1);

blockmask = max(stdfilt(Argb,ones(21)),[],3) < 10;

blockmask = padarray(blockmask(2:end-1,2:end-1),[1 1],0,'both');

% imopen() is used with a long strel to cut the blocks apart

% doing it this way is sensitive to image rotation

% erosion shrinks the mask to exclude regions near block edges

% bwareafilt() gets rid of any remaining small blobs

blockmask = imopen(imopen(blockmask,se),se');

blockmask = imerode(blockmask,ones(25));

blockmask = bwareafilt(blockmask,9);

% this is the mask

imshow(blockmask)

% segment the mask and the source image

% CT is a color table containing the mean color in each mask blob

CC = bwconncomp(blockmask);

nblocks = CC.NumObjects;

CT = zeros(nblocks,3);

Act = reshape(Argb,[],3);

for b = 1:nblocks

pixidx = CC.PixelIdxList{b};

sampledpixels = Act(pixidx,:);

CT(b,:) = mean(sampledpixels,1)/255;

end

% get the centroids of all the blobs

S = regionprops(CC,'centroid');

C = vertcat(S(:).Centroid);

% display indicators on the mask

% note that blobs aren't ordered as expected

clf

imshow(blockmask)

for b = 1:nblocks

text(C(b,1),C(b,2),sprintf('%d',b),'color','k','fontsize',20)

end

% try to snap the centers to a grid

% again, this assumes the image isn't severely rotated/distorted

climits = [min(C,[],1); max(C,[],1)];

Cnom = round((C-climits(1,:))./range(climits,1)*2 + 1);

% reorder color samples and centroids

idx = sub2ind([3 3],Cnom(:,2),Cnom(:,1));

CT(idx,:) = CT;

C(idx,:) = C;

% display indicators on the mask after sorting

% now the block information is in column-major order

clf

imshow(blockmask)

for b = 1:nblocks

text(C(b,1),C(b,2),sprintf('%d',b),'color','k','fontsize',20)

end

% display colored patches on the original image

% this shows that the sample colors correspond to the blocks

clf

imshow(Argb); hold on

hs = scatter(C(:,1),C(:,2),5000,CT,'filled');

hs.MarkerEdgeColor = 'k';

% so this is the color table for this image

CT

CT = 9×3

0.0490 0.7965 0.1848 0.1900 0.4245 0.7063 0.8080 0.7477 0.1320 0.9248 0.3623 0.4214 0.8225 0.8080 0.8325 0.9897 0.5775 0.3256 0.9105 0.3310 0.3839 0.0045 0.7953 0.1136 0.9107 0.8509 0.1033

Okay, so now you have a color table. Now what? At this point, you'd have to try to classify the colors. Since I only have one image, I really don't have anything to test against. You might take a few pictures and build a reference table from the average of similar-class samples. Then you can do distance minimization to find which reference color matches a given image under test.

This answer includes a similar process. Similarly, rgb2ind() could be used as well.

If you can provide other images, i might be able to expand on this, otherwise it's up to you to work with the images you're receiving.

24 个评论
显示 22更早的评论隐藏 22更早的评论

DGM 2022-10-15

在 MATLAB Online 中打开

Here's one example based on the linked thread:

% let's pretend we have a new image to test

% i'm going to reuse this same image and modify it

Atest = imread('https://www.mathworks.com/matlabcentral/answers/uploaded_files/1157503/image.jpeg');

Atest = rot90(imnoise(Atest,'gaussian',0,0.001),1);

imshow(Atest)

% ------THIS PART IS BASICALLY THE SAME--------

% create a mask that samples the centers of the blocks

% stdfilt() is used to find regions of contrast (boundaries of blocks)

% this image is thresholded to create a rough low-contrast region mask

% the image is trimmed and padded with zero to detatch blobs from the border

se = ones(100,1);

blockmask = max(stdfilt(Atest,ones(21)),[],3) < 10;

blockmask = padarray(blockmask(2:end-1,2:end-1),[1 1],0,'both');

% imopen() is used with a long strel to cut the blocks apart

% erosion shrinks the mask to exclude regions near block edges

% bwareafilt() gets rid of any remaining small blobs

blockmask = imopen(imopen(blockmask,se),se');

blockmask = imerode(blockmask,ones(25));

blockmask = bwareafilt(blockmask,9);

% segment the mask and the source image

CC = bwconncomp(blockmask);

nblocks = CC.NumObjects;

CT = zeros(nblocks,3);

Act = reshape(Atest,[],3);

for b = 1:nblocks

pixidx = CC.PixelIdxList{b};

sampledpixels = Act(pixidx,:);

CT(b,:) = mean(sampledpixels,1)/255;

end

% get the centroids of all the blobs

S = regionprops(CC,'centroid');

C = vertcat(S(:).Centroid);

% try to snap the centers to a grid

% again, this assumes the image isn't severely rotated/distorted

climits = [min(C,[],1); max(C,[],1)];

Cnom = round((C-climits(1,:))./range(climits,1)*2 + 1);

% reorder color samples and centroids

idx = sub2ind([3 3],Cnom(:,2),Cnom(:,1));

CT(idx,:) = CT;

C(idx,:) = C;

% ------THIS PART IS NEW--------

% specify a correlated list of colors and color names

% these are drawn from the reference image(s)

colornames = {'w','r','g','b','y','o'};

colorrefs = [0.8245 0.8105 0.8363; 0.9179 0.3463 0.4028; ...

0.02596 0.7964 0.148; 0.1896 0.4244 0.7066; ...

0.8615 0.8011 0.1177; 0.9916 0.58 0.3277];

% find color distances in RGB

D = CT - permute(colorrefs,[3 2 1]);

D = squeeze(sum(D.^2,2)) ;

% find index of closest match for each block

[~,idx] = min(D,[],2);

% look up color names

blocknames = reshape(colornames(idx),3,3)

blocknames = 3×3 cell array

{'r'} {'g'} {'y'} {'r'} {'w'} {'o'} {'g'} {'b'} {'y'}

Bear in mind that things like thresholds and whatnot may need to be adjusted. I only made this work with the one image.

DGM 2022-10-16

在 MATLAB Online 中打开

Now that there are two examples, I'm going to modify the reference color table by finding the average of the color samples across both images. This isn't something that needs to be done every time, but the reference table should be built from more than one image.

% so this is the CT for the first image
CT = [0.04898 0.7965 0.1848;0.19 0.4245 0.7063;0.808 0.7477 0.132;0.9248 0.3623 0.4214;0.8225 0.808 0.8325;0.9897 0.5775 0.3256;0.9105 0.331 0.3839;0.004511 0.7953 0.1136;0.9107 0.8509 0.1033];
% this is the CT for the second image
CT2 = [0.854 0.4329 0.2038;0.6642 0.6453 0.6144;0.7724 0.25 0.2505;0.1537 0.348 0.5887;0.8225 0.2716 0.2594;0.003548 0.7385 0.002046;0.7865 0.7929 0.731;0.882 0.8471 0.001494;0.8792 0.8484 0.001481];
% find averages to build reference table from CT1,CT2
%   CT1		CT2
% w 5		[2 7]
% r [4 7]	[3 5]
% g [1 8]	6
% b 2		4
% y [3 9]	[8 9]
% o 6		1
% in order: 'w','r','g','b','y','o';
colorrefs = [mean([CT(5,:); CT2([2 7],:)],1);
             mean([CT([4 7],:); CT2([3 5],:)],1); 
             mean([CT([1 8],:); CT2(6,:)],1);
             mean([CT(2,:); CT2(4,:)],1);
             mean([CT([3 9],:); CT2([8 9],:)],1); 
             mean([CT(6,:); CT2(1,:)],1)];
% this is the table to be used later as a reference
mat2str(colorrefs,4)
ans = '[0.7577 0.7487 0.726;0.8576 0.3037 0.3288;0.01901 0.7768 0.1001;0.1719 0.3862 0.6475;0.87 0.8235 0.05957;0.9219 0.5052 0.2647]'

With the new reference table, the prior code should work to identify the colors:

% the new test image
Atest = imread('https://www.mathworks.com/matlabcentral/answers/uploaded_files/1158338/image.jpeg');
% create a mask that samples the centers of the blocks
% stdfilt() is used to find regions of contrast (boundaries of blocks)
% this image is thresholded to create a rough low-contrast region mask
% the image is trimmed and padded with zero to detatch blobs from the border
se = ones(100,1);
blockmask = max(stdfilt(Atest,ones(21)),[],3) < 10;
blockmask = padarray(blockmask(2:end-1,2:end-1),[1 1],0,'both');
% imopen() is used with a long strel to cut the blocks apart
% erosion shrinks the mask to exclude regions near block edges
% bwareafilt() gets rid of any remaining small blobs
blockmask = imopen(imopen(blockmask,se),se');
blockmask = imerode(blockmask,ones(25));
blockmask = bwareafilt(blockmask,9);
% segment the mask and the source image
CC = bwconncomp(blockmask);
nblocks = CC.NumObjects;
CT = zeros(nblocks,3);
Act = reshape(Atest,[],3);
for b = 1:nblocks
    pixidx = CC.PixelIdxList{b};
    sampledpixels = Act(pixidx,:);
    CT(b,:) = mean(sampledpixels,1)/255;
end
% get the centroids of all the blobs
S = regionprops(CC,'centroid');
C = vertcat(S(:).Centroid);
% try to snap the centers to a grid
% again, this assumes the image isn't severely rotated/distorted
climits = [min(C,[],1); max(C,[],1)];
Cnom = round((C-climits(1,:))./range(climits,1)*2 + 1);
% reorder color samples and centroids
idx = sub2ind([3 3],Cnom(:,2),Cnom(:,1));
CT(idx,:) = CT;
C(idx,:) = C;
% specify a correlated list of colors and color names
% these are drawn from the reference image(s)
colornames = {'w','r','g','b','y','o'};
% this is the old reference from image 1 alone
% colorrefs = [0.8245 0.8105 0.8363; 0.9179 0.3463 0.4028; ...
% 			 0.02596 0.7964 0.148; 0.1896 0.4244 0.7066; ...
% 			 0.8615 0.8011 0.1177; 0.9916 0.58 0.3277];
% this is the updated reference from both images (calculated above)
colorrefs = [0.7577 0.7487 0.726;0.8576 0.3037 0.3288; ...
             0.01901 0.7768 0.1001;0.1719 0.3862 0.6475;
             0.87 0.8235 0.05957;0.9219 0.5052 0.2647];
% find color distances in RGB
D = CT - permute(colorrefs,[3 2 1]);
D = squeeze(sum(D.^2,2)) ;
% find index of closest match for each block
[~,idx] = min(D,[],2);
% look up color names
blocknames = reshape(colornames(idx),3,3)
blocknames = 3×3 cell array
    {'o'}    {'b'}    {'w'}
    {'w'}    {'r'}    {'y'}
    {'r'}    {'g'}    {'y'}

As the comments note, the method I used for blob isolation and ordering is sensitive to how well the cube is framed. If the lighting becomes inconsistent, it may also make it difficult to use an approach based on comparison to fixed reference colors like this.

DGM 2022-10-17

编辑：DGM 2022-10-17

在 MATLAB Online 中打开

As I mentioned, if the position/framing changes significantly, changes will need to be made. The uncropped photo is a significant change. Likewise, the part of the code that's throwing an error is attempting to re-grid the blobs based on the assumption that the object is close-cropped, centered, and aligned with the frame. The reason it's throwing the error is probably not due to the image being uncentered, but rather because C (the list of centroids) is empty because the segmentation failed for some reason. Since I don't know what PNG file you're testing, I don't know why.

As to what background would work best, choose something that contrasts with everything on the target object. Black would probably be best, since part of the target is nominally white.

Even if it's easy to distinguish the object from the background based on color, it's not simple to universally deal with the perspective. The problem is trying to exclude the sides of the cube and only deal with the top face. Trying to rely on the shadow would make the cropping routine heavily dependent on orientation and lighting.

Unless you need to find and process the cube in any presentation, simplify the task by making the presentations consistent and favorable for the processing task.

This is a naive attempt to crop the top cube face from the given image based on color alone. This will be sensitive to changes in the image, and should be considered not as an example of a robust segmentation, but rather as an example of how easy it is to make robust segmentation difficult.

Auncropped = imread('https://www.mathworks.com/matlabcentral/answers/uploaded_files/1158898/IMG_20221013_201913.jpg');

% assume we can find the cube based on color alone

[H S V] = imsplit(rgb2hsv(Auncropped));

objmask = V.*S > 0.5; % threshold (dependent on lighting/background)

objmask = imclose(objmask,ones(30)); % try to make sure blocks are connected

objmask = bwareaopen(objmask,70E3); % get rid of small blobs (dependent on image size)

% assuming the only thing left is the cube, crop it

% this will break if the cube hasn't been isolated

mkrows = any(objmask,2);

mkcols = any(objmask,1);

Acropped = Auncropped(mkrows,mkcols,:);

% this is the cropped result

imshow(Acropped)

At this point, the prior code should work.

Sang 2022-10-18

在 MATLAB Online 中打开

Can you help me combine these 2 programs to make it work in common?

1.image cropping program

Auncropped = imread('https://www.mathworks.com/matlabcentral/answers/uploaded_files/1158898/IMG_20221013_201913.jpg');
% assume we can find the cube based on color alone
[H S V] = imsplit(rgb2hsv(Auncropped));
objmask = V.*S > 0.5; % threshold (dependent on lighting/background)
objmask = imclose(objmask,ones(30)); % try to make sure blocks are connected
objmask = bwareaopen(objmask,70E3); % get rid of small blobs (dependent on image size)
% assuming the only thing left is the cube, crop it
% this will break if the cube hasn't been isolated
mkrows = any(objmask,2);
mkcols = any(objmask,1);
Acropped = Auncropped(mkrows,mkcols,:);
% this is the cropped result
imshow(Acropped)

2. color recognition program

% the new test image
Atest = imread('rubik8.jpg');
imshow(Atest)
% create a mask that samples the centers of the blocks
% stdfilt() is used to find regions of contrast (boundaries of blocks)
% this image is thresholded to create a rough low-contrast region mask
% the image is trimmed and padded with zero to detatch blobs from the border
se = ones(100,1);
blockmask = max(stdfilt(Atest,ones(21)),[],3) < 10;
blockmask = padarray(blockmask(2:end-1,2:end-1),[1 1],0,'both');
% imopen() is used with a long strel to cut the blocks apart
% erosion shrinks the mask to exclude regions near block edges
% bwareafilt() gets rid of any remaining small blobs
blockmask = imopen(imopen(blockmask,se),se');
blockmask = imerode(blockmask,ones(25));
blockmask = bwareafilt(blockmask,9);
% segment the mask and the source image
CC = bwconncomp(blockmask);
nblocks = CC.NumObjects;
CT = zeros(nblocks,3);
Act = reshape(Atest,[],3);
for b = 1:nblocks
    pixidx = CC.PixelIdxList{b};
    sampledpixels = Act(pixidx,:);
    CT(b,:) = mean(sampledpixels,1)/255;
end
% get the centroids of all the blobs
S = regionprops(CC,'centroid');
C = vertcat(S(:).Centroid);
% try to snap the centers to a grid
% again, this assumes the image isn't severely rotated/distorted
climits = [min(C,[],1); max(C,[],1)];
Cnom = round((C-climits(1,:))./range(climits,1)*2 + 1);
% reorder color samples and centroids
idx = sub2ind([3 3],Cnom(:,2),Cnom(:,1));
CT(idx,:) = CT;
C(idx,:) = C;
% specify a correlated list of colors and color names
% these are drawn from the reference image(s)
colornames = {'w','r','g','b','y','o'};
% this is the old reference from image 1 alone
% colorrefs = [0.8245 0.8105 0.8363; 0.9179 0.3463 0.4028; ...
% 			 0.02596 0.7964 0.148; 0.1896 0.4244 0.7066; ...
% 			 0.8615 0.8011 0.1177; 0.9916 0.58 0.3277];
% this is the updated reference from both images (calculated above)
colorrefs = [0.7577 0.7487 0.726;0.8576 0.3037 0.3288; ...
             0.01901 0.7768 0.1001;0.1719 0.3862 0.6475;
             0.87 0.8235 0.05957;0.9219 0.5052 0.2647];
% find color distances in RGB
D = CT - permute(colorrefs,[3 2 1]);
D = squeeze(sum(D.^2,2)) ;
% find index of closest match for each block
[~,idx] = min(D,[],2);
% look up color names
blocknames = reshape(colornames(idx),3,3)

with the following image as input

DGM 2022-10-19

在 MATLAB Online 中打开

The code I gave for cropping and processing should still work on that image. Take Acropped and process it with the second script. In this case, I'm just going to rename Atest to Acropped. Nothing else has been changed.

Auncropped = imread('https://www.mathworks.com/matlabcentral/answers/uploaded_files/1160693/image.jpeg');

% assume we can find the cube based on color

[H S V] = imsplit(rgb2hsv(Auncropped));

objmask = V.*S > 0.5; % threshold (dependent on lighting/background)

objmask = imclose(objmask,ones(30)); % try to make sure blocks are connected

objmask = bwareaopen(objmask,70E3); % get rid of small blobs (dependent on image size)

% assuming the only thing left is the cube, crop it

% this will break if the cube hasn't been isolated

mkrows = any(objmask,2);

mkcols = any(objmask,1);

Acropped = Auncropped(mkrows,mkcols,:);

% this is the cropped result

imshow(Acropped)

% create a mask that samples the centers of the blocks

% stdfilt() is used to find regions of contrast (boundaries of blocks)

% this image is thresholded to create a rough low-contrast region mask

% the image is trimmed and padded with zero to detatch blobs from the border

se = ones(100,1);

blockmask = max(stdfilt(Acropped,ones(21)),[],3) < 10;

blockmask = padarray(blockmask(2:end-1,2:end-1),[1 1],0,'both');

% imopen() is used with a long strel to cut the blocks apart

% erosion shrinks the mask to exclude regions near block edges

% bwareafilt() gets rid of any remaining small blobs

blockmask = imopen(imopen(blockmask,se),se');

blockmask = imerode(blockmask,ones(25));

blockmask = bwareafilt(blockmask,9);

% segment the mask and the source image

CC = bwconncomp(blockmask);

nblocks = CC.NumObjects;

CT = zeros(nblocks,3);

Act = reshape(Acropped,[],3);

for b = 1:nblocks

pixidx = CC.PixelIdxList{b};

sampledpixels = Act(pixidx,:);

CT(b,:) = mean(sampledpixels,1)/255;

end

% get the centroids of all the blobs

S = regionprops(CC,'centroid');

C = vertcat(S(:).Centroid);

% try to snap the centers to a grid

% again, this assumes the image isn't severely rotated/distorted

climits = [min(C,[],1); max(C,[],1)];

Cnom = round((C-climits(1,:))./range(climits,1)*2 + 1);

% reorder color samples and centroids

idx = sub2ind([3 3],Cnom(:,2),Cnom(:,1));

CT(idx,:) = CT;

C(idx,:) = C;

% specify a correlated list of colors and color names

% these are drawn from the reference image(s)

colornames = {'w','r','g','b','y','o'};

colorrefs = [0.7577 0.7487 0.726;0.8576 0.3037 0.3288; ...

0.01901 0.7768 0.1001;0.1719 0.3862 0.6475;

0.87 0.8235 0.05957;0.9219 0.5052 0.2647];

% find color distances in RGB

D = CT - permute(colorrefs,[3 2 1]);

D = squeeze(sum(D.^2,2)) ;

% find index of closest match for each block

[~,idx] = min(D,[],2);

% look up color names

blocknames = reshape(colornames(idx),3,3)

blocknames = 3×3 cell array

{'g'} {'b'} {'o'} {'y'} {'o'} {'o'} {'r'} {'y'} {'w'}

Bear in mind if the cropping procedure needs to become more complicated, it might be convenient to modularize things and break this down into two separate functions or something.

Image Analyst 2022-10-28

在 MATLAB Online 中打开

You must have a version older than r2018b, when imsplit was introduced. So you can do

hsvImage = rgb2hsv(Auncropped);
H = hsvImage(:, :, 1);
S = hsvImage(:, :, 2);
V = hsvImage(:, :, 3);

DGM 2022-10-28

splitchans.m

Do you have an older version (R2018a or older), or do you not have IPT? If you don't have IPT, there will be other issues. If you're running something significantly older than R2018x, there may be other version-dependent issues that may show up.

If the only issue is the lack of imsplit(), you can use the attached file to replace it. MIMT splitchans() can do everything IPT imsplit() does, and should not be dependent on anything. You'll obviously have to change that line to call splitchans() instead of imsplit().

请先登录，再进行评论。

Answer 2

Image Analyst 2022-10-15

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1827523-how-to-recognize-6-colors-of-a-face-rubik-s-cube-at-the-same-time#answer_1076028

There are lots of ways to "solve" this but it depends on what you want to know? Do you want to know only how many colors there are? Do you want to know the predicted reference color for every pixel in the image? Do you want to find each cube face and find the dominant color for that cube face?

What I'd to is to classify every pixel in the image into one of your six known reference colors. For example you could use the attached demo to use discriminant analysis (see attached demo) or K nearest neighbors (see attached demo), or you could convert to LAB color space and find the lowest delta E color difference like I did in this link https://www.mathworks.com/matlabcentral/answers/1819405-how-to-change-the-color-of-each-pixel-in-a-masked-image#comment_2410068 for a poster who was comparing a melanoma spot to 6 reference colors.