GAN intermediate layer for image multiplication

Question

Glacial Claw 2023-2-23

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1917400-gan-intermediate-layer-for-image-multiplication

编辑： Rahul 2023-3-1

So I have created a GAN that uses a regular noise input for image synthesis. However, I want to multiply or get the product of the generated image with an outside image to speed up training/ training convergence. Here is the basic idea:

%This is the layer setup I use for my GAN
numFilters = 128;
numLatentInputs = 256;
inputSize = [128 128 3];
projectionSize = [4 4 3];
layersGenerator = [
    featureInputLayer(numLatentInputs)
    projectAndReshapeLayer(projectionSize)
    resize2dLayer('OutputSize',[4 4],'Method','bilinear')
    batchNormalizationLayer
    leakyReluLayer
    
    resize2dLayer('OutputSize',[16 16],'Method','bilinear')
    batchNormalizationLayer
    leakyReluLayer
    resize2dLayer('OutputSize',[32 32],'Method','bilinear')
    batchNormalizationLayer
    leakyReluLayer
    
    resize2dLayer('OutputSize',[64 64],'Method','bilinear')
    batchNormalizationLayer
    leakyReluLayer
   
    resize2dLayer('OutputSize',[128 128],'Method','bilinear')
    batchNormalizationLayer
    tanhLayer];
layersDiscriminator = [
    imageInputLayer(inputSize,Normalization="none")
    dropoutLayer(dropoutProb)
    resize2dLayer('OutputSize',[128 128],'Method','bilinear')
    convolution2dLayer(filterSize,numFilters,Stride=2,Padding="same")
    leakyReluLayer(scale)
    resize2dLayer('OutputSize',[64 64],'Method','bilinear')
    convolution2dLayer(filterSize,2*numFilters,Stride=2,Padding="same")
    batchNormalizationLayer
    leakyReluLayer(scale)
    resize2dLayer('OutputSize',[32 32],'Method','bilinear')
    convolution2dLayer(filterSize,4*numFilters,Stride=2,Padding="same")
    batchNormalizationLayer
    leakyReluLayer(scale)
    resize2dLayer('OutputSize',[16 16],'Method','bilinear')
    convolution2dLayer(filterSize,8*numFilters,Stride=2,Padding="same")
    batchNormalizationLayer
    leakyReluLayer(scale)
    resize2dLayer('OutputSize',[8 8],'Method','bilinear')
    convolution2dLayer(8,1)
    sigmoidLayer];
numEpochs = 2000;
miniBatchSize = 16;
learnRate = 0.0003;
gradientDecayFactor = 0.5;
squaredGradientDecayFactor = 0.999;
flipProb = 0.75;
validationFrequency = 5;
%The custom layer that I have already made:
classdef imageProduct < nnet.layer.Layer 
    properties (Constant)
        ImageinputT = imread("image.jpg");
    end 
    methods
        function layer = imageProduct()
            layer.Type = "Image Product";
            layer.Description = "Combine Images";
            layer.OutputNames = "out";
        end
        function [Z] = predict(~,X)
            %gets the image size so the input image can be resized. Basically the target scale
            [WidthX, HeightX] = size(X); 
            %resizes the imported image
            GinputIm_scaled = imresize(crossAttention.ImageinputT, [WidthX HeightX]); 
            %converts images to double
            GinputIm_scaled_single = single(GinputIm_scaled); 
            Xsingle = single(X);
            %Gets the tensor product of the two images
            Xnew = tensorprod(GinputIm_scaled_single, Xsingle); 
            Z = Xnew; %output image
        end
    end
end