Using pointCloudInputLayer with convolution3D

Question

J Schreiber 2024-11-19，0:29

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/2167098-using-pointcloudinputlayer-with-convolution3d

回答： Gayathri 2024-11-19，6:36

I have a [10000, 3] point cloud in R^3 and I want to pass it into a network that uses convolution3d, but I keep getting an error about the number of spatial dimensions being passed to the conv3d layer. This is the code I have so far:

X = randn(10000,3);
dsTrain = arrayDatastore(X,IterationDimension=1);
numOutputs = 1;
miniBatchSize = 128;
mbq = minibatchqueue(dsTrain,numOutputs, ...
    MiniBatchSize = miniBatchSize, ...
    MiniBatchFcn=@preprocessMiniBatch, ...
    MiniBatchFormat="SSSCB", ...
    PartialMiniBatch="discard");
numLatentChannels = 5;
featureSize = [10000 3];
%encoding layers
layersE = [
    pointCloudInputLayer(featureSize)  %image3dInputLayer(gridSize, 'Name', 'input')
    convolution3dLayer(3,32,Padding="same",Stride=2)
    reluLayer
    convolution3dLayer(3,64,Padding="same",Stride=2)
    reluLayer
    fullyConnectedLayer(numLatentChannels)];
projectionSize = [7 7 64];
numInputChannels = 1;
%decoding layers
layersD = [
    featureInputLayer(numLatentChannels)
    projectAndReshapeLayer(projectionSize)
    transposedConv3dLayer(3,64,Cropping="same",Stride=2)
    reluLayer
    transposedConv3dLayer(3,32,Cropping="same",Stride=2)
    reluLayer
    transposedConv3dLayer(3,numInputChannels,Cropping="same")
    sigmoidLayer];
netE = dlnetwork(layersE);
netD = dlnetwork(layersD);
%initialize parameters for ADAM solver
trailingAvgE = []; trailingAvgSqE = [];
trailingAvgD = []; trailingAvgSqD = [];
numObservationsTrain = 10000;
numIterationsPerEpoch = ceil(numObservationsTrain / miniBatchSize);
numIterations = numEpochs * numIterationsPerEpoch;
epoch = 0;
iteration = 0;
% Loop over epochs
while epoch < 10
    epoch = epoch + 1;
    % Shuffle data.
    shuffle(mbq);
    % Loop over mini-batches.
    while hasdata(mbq) && ~monitor.Stop
        iteration = iteration + 1;
        % Read mini-batch of data.
        X = next(mbq);
        % Evaluate loss and gradients.
        [loss,gradientsE,gradientsD] =...
            dlfeval(@ModelLoss,netE,netD,X);
        % Update learnable parameters.
        [netE,trailingAvgE,trailingAvgSqE] = adamupdate(netE, ...
            gradientsE,trailingAvgE,trailingAvgSqE,iteration,learnRate);
        
        [netD, trailingAvgD, trailingAvgSqD] = adamupdate(netD, ...
            gradientsD,trailingAvgD,trailingAvgSqD,iteration,learnRate);
    end
end

Where ModelLoss is defined as

function [Loss,gradientsE,gradientsD] = ModelLoss(netE,netD)
Z = forward(netE,X);
Y = forward(netD,Z);
Loss = mse(Y,X)
[gradientsE,gradientsD] = dlgradient(loss,netE.Learnables,netD.Learnables);
end

and projectAndReshapeLayer is defined as

classdef projectAndReshapeLayer < nnet.layer.Layer ...
        & nnet.layer.Formattable ...
        & nnet.layer.Acceleratable
    properties
        % Layer properties.
        OutputSize
    end
    properties (Learnable)
        % Layer learnable parameters.
        Weights
        Bias
    end
    methods
        function layer = projectAndReshapeLayer(outputSize,NameValueArgs)
            % layer = projectAndReshapeLayer(outputSize)
            % creates a projectAndReshapeLayer object that projects and
            % reshapes the input to the specified output size.
            %
            % layer = projectAndReshapeLayer(outputSize,Name=name)
            % also specifies the layer name.
            % Parse input arguments.
            arguments
                outputSize
                NameValueArgs.Name = "";
            end
            % Set layer name.
            name = NameValueArgs.Name;
            layer.Name = name;
            % Set layer description.
            layer.Description = "Project and reshape to size " + ...
                join(string(outputSize));
            % Set layer type.
            layer.Type = "Project and Reshape";
            % Set output size.
            layer.OutputSize = outputSize;
        end
        function layer = initialize(layer,layout)
            % layer = initialize(layer,layout) initializes the layer
            % learnable parameters.
            %
            % Inputs:
            %         layer  - Layer to initialize
            %         layout - Data layout, specified as a 
            %                  networkDataLayout object
            %
            % Outputs:
            %         layer - Initialized layer
            % Layer output size.
            outputSize = layer.OutputSize;
            % Initialize fully connect weights.
            if isempty(layer.Weights)
                % Find number of channels.
                idx = finddim(layout,"C");
                numChannels = layout.Size(idx);
                % Initialize using Glorot.
                sz = [prod(outputSize) numChannels];
                numOut = prod(outputSize);
                numIn = numChannels;
                layer.Weights = initializeGlorot(sz,numOut,numIn);
            end
            % Initialize fully connect bias.
            if isempty(layer.Bias)
                % Initialize with zeros.
                layer.Bias = initializeZeros([prod(outputSize) 1]);
            end
        end
        function Z = predict(layer, X)
            % Forward input data through the layer at prediction time and
            % output the result.
            %
            % Inputs:
            %         layer - Layer to forward propagate through
            %         X     - Input data, specified as a formatted dlarray
            %                 with a "C" and optionally a "B" dimension.
            % Outputs:
            %         Z     - Output of layer forward function returned as
            %                 a formatted dlarray with format "SSCB".
            % Fully connect.
            weights = layer.Weights;
            bias = layer.Bias;
            X = fullyconnect(X,weights,bias);
            % Reshape.
            outputSize = layer.OutputSize;
            Z = reshape(X,outputSize(1),outputSize(2),outputSize(3),[]);
            Z = dlarray(Z,"SSCB");
        end
    end
end

And preprocessMiniBatch is defined as

function X = preprocessMiniBatch(dataX)
% Concatenate.
X = cat(1,dataX{:});
end

I keep getting the error:

Layer 'conv3d_1': Input data must have three spatial dimensions only, two spatial dimensions and one temporal dimension, or three spatial dimensions and one temporal dimension. Instead, it has 1 spatial dimensions and 0 temporal dimensions.

It was my understanding that pointCloudInputLayer could take raw point cloud data. I also tried to voxelize the point cloud (following the example from Encode Point Cloud Data For Deep Learning) to use with image3dInputLayer instead, but wasn't sure how to follow since my data does not have any labels.

Any help is appreciated.

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

Matt J 2024-11-19，2:34

编辑：Matt J 2024-11-19，2:38

It was my understanding that pointCloudInputLayer could take raw point cloud data.

Yes, but the error message is talking about the input to the convolution3dLayer, not the input to the network as a whole. It is not clear what you would like the 3D convolutional layer to do with point cloud data. Point cloud coordinates are not the same kind of data as the voxel values of a 3D image.

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Gayathri 2024-11-19，6:36

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/2167098-using-pointcloudinputlayer-with-convolution3d#answer_1546618

在 MATLAB Online 中打开

Hello @J Schreiber,

I understand that you are facing error while using “convolution3dLayer” along with “pointCloudInputLayer”. You've also attempted to use “image3dInputLayer” by voxelizing the input data.

The error occurs because the input to the “convolution3dLayer” does not have the expected number of dimensions. To use 3D convolutions, your input data should be in a 4D format that represents a 3D volume. One alternative is to replace the “convolution3dLayer” with a “fullyConnectedLayer”.

To continue using the “convolution3dLayer”, we will have to convert the point cloud to voxel grid representation. Also, we will have to use “image3dInputLayer” instead of the “pointCloudInputLayer”, as mentioned in the question.

% Example point cloud data
X = randn(10000, 3); % 10000 points with (x, y, z) coordinates
% Parameters for voxelization
voxelGridSize = [32, 32, 32]; % Define the size of the voxel grid
% Convert point cloud to voxel grid
voxelGrid = pointCloudToVoxelGrid(X, voxelGridSize);
% Create an arrayDatastore with voxelized data
dsTrain = arrayDatastore(voxelGrid, IterationDimension=4);
numLatentChannels = 1;
% Encoding layers using 3D convolutions
layersE = [
    image3dInputLayer(voxelGridSize, 'Name', 'input')
    convolution3dLayer(3, 32, Padding="same", Stride=2)
    reluLayer
    convolution3dLayer(3, 64, Padding="same", Stride=2)
    reluLayer
    fullyConnectedLayer(numLatentChannels)];
netE = dlnetwork(layersE);
% Function to convert point cloud to voxel grid
function voxelGrid = pointCloudToVoxelGrid(points, gridSize)
    % Initialize a zero grid
    voxelGrid = zeros([gridSize, 1]);
    
    % Normalize and scale points to fit within the grid
    minVals = min(points);
    maxVals = max(points);
    scaledPoints = (points - minVals) ./ (maxVals - minVals);
    voxelIndices = floor(scaledPoints .* (gridSize - 1)) + 1;
    
    % Populate the voxel grid
    for i = 1:size(points, 1)
        voxelGrid(voxelIndices(i, 1), voxelIndices(i, 2), voxelIndices(i, 3)) = 1;
    end
end

This code will help resolve the error pertaining to the input dimensions of the “convolution3dLayer”.

For more information on “image3dInputLayer” function, please refer to the following documentation link.

https://www.mathworks.com/help/releases/R2024a/deeplearning/ref/nnet.cnn.layer.image3dinputlayer.html

Hope you find this information helpful.

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

Using pointCloudInputLayer with convolution3D

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

回答（1 个）

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

另请参阅

类别

标签

产品

版本

Community Treasure Hunt

Using pointCloudInputLayer with convolution3D

1 个评论 显示 -1更早的评论隐藏 -1更早的评论

回答（1 个）

0 个评论 显示 -2更早的评论隐藏 -2更早的评论

另请参阅

类别

标签

产品

版本

Community Treasure Hunt

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

0 个评论
显示 -2更早的评论隐藏 -2更早的评论