My training data exceeds the maximum array size preference.

11 次查看（过去 30 天）
Yasmin Ben Azouz
Yasmin Ben Azouz 2022-7-17
回答： Siraj 2023-9-14
Hi everyone,
I am currently trying to train a CNN using the trainNetwork function. The data I am using is 7200000x1 double sequence data, the labels are simply one or zero, also in a 7200000x1 double. I have this data for 16 sequences. Everytime I try to use the code I get the following error:
Error using trainNetwork (line 184)
Layer 2: Invalid initializer. Requested 480x7200000 (25.7GB) array exceeds maximum array size preference (8.0GB). This might cause MATLAB to become unresponsive.
Error in YASTrainNetwork (line 94)
net = trainNetwork(Xtrain, Ytrain, layers, options);
Caused by:
Error using nnet.internal.cnn.assembler.InitializeMixin/initializeLearnableParameters (line 21)
Layer 2: Invalid initializer. Requested 480x7200000 (25.7GB) array exceeds maximum array size preference (8.0GB). This might cause MATLAB to become unresponsive.
Related documentation
After dividing and normalizing the data, these are the net options I am currently using:
options = trainingOptions('adam', ...
'MaxEpochs',MaxEpochs, ...
'MiniBatchSize',MiniBatchSize, ...
'InitialLearnRate',InitialLearnRate, ...
'LearnRateDropPeriod',3, ...
'LearnRateSchedule','piecewise', ...
'GradientThreshold',1, ...
'Plots','training-progress',...
'shuffle','every-epoch',...
'SequenceLength',500,...
'Verbose',0,...
'ValidationData',{Xval,Yval},...
'ValidationFrequency',50,...
'ValidationPatience',inf,...
'DispatchInBackground',true );
I understand I need to divide the data in some way, but I can't seem to find out how exactly. Can someone explain this to me?
  2 个评论
Ben
Ben 2022-7-20
What are layers, specifically layers(2)? And what shape is Xtrain?
I wouldn't expect most sequence layers to initialize weights the same size as the sequence length. For 1D convolutions you would typically choose a smaller filter size than the full sequence length (particularly if your sequences are this long).
Yasmin Ben Azouz
Yasmin Ben Azouz 2022-7-28
Hi Ben, These are my layers. Thank you for the answer!
layers = [...
sequenceInputLayer(numSequence,'Normalization', 'zscore')
bilstmLayer(numHiddenUnits,'OutputMode','sequence')
fullyConnectedLayer(2)
softmaxLayer
classificationLayer];

请先登录，再进行评论。

请先登录，再回答此问题。

回答（1 个）

Siraj
Siraj 2023-9-14
Hi! It is my understanding that you are getting the error because, due to very large input size, the weights matrix for layer 2 is of such dimensions that it requires more memory than permitted.
Here are some potential solutions to address the issue:
1. Apply dimensionality reduction techniques or feature extraction methods to reduce the input size. These techniques aim to reduce the number of features or dimensions in your dataset while retaining important information. For more detailed information and examples, you can refer to the following link:
https://www.mathworks.com/help/stats/dimensionality-reduction.html
2. Next you can go to MATLAB > Preferences > Workspace and ensure the maximum array size limit is set to 100%. Then execute 'memory' command in the Command Window and send the output. Ensure that the maximum possible array size is larger than the memory required by the data used in neural network. To learn more about the “memory” command refer to the following link.
https://www.mathworks.com/help/matlab/ref/memory.html
3. Also, check that the Java Heap Memory is not set to a very large value because that might restrict the amount of memory available to perform computations. Refer to the following link to adjust the amount of memory that MATLAB allocates for Java objects.
https://www.mathworks.com/help/matlab/matlab_external/java-heap-memory-preferences.html
Hope this helps.

类别

在 Help Center 和 File Exchange 中查找有关 Image Data Workflows 的更多信息

标签

产品


版本

R2021b

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by