My training data exceeds the maximum array size preference.
4 次查看(过去 30 天)
显示 更早的评论
Hi everyone,
I am currently trying to train a CNN using the trainNetwork function. The data I am using is 7200000x1 double sequence data, the labels are simply one or zero, also in a 7200000x1 double. I have this data for 16 sequences. Everytime I try to use the code I get the following error:
Error using trainNetwork (line 184)
Layer 2: Invalid initializer. Requested 480x7200000 (25.7GB) array exceeds maximum array size preference (8.0GB). This might cause MATLAB to become unresponsive.
Error in YASTrainNetwork (line 94)
net = trainNetwork(Xtrain, Ytrain, layers, options);
Caused by:
Error using nnet.internal.cnn.assembler.InitializeMixin/initializeLearnableParameters (line 21)
Layer 2: Invalid initializer. Requested 480x7200000 (25.7GB) array exceeds maximum array size preference (8.0GB). This might cause MATLAB to become unresponsive.
Related documentation
After dividing and normalizing the data, these are the net options I am currently using:
options = trainingOptions('adam', ...
'MaxEpochs',MaxEpochs, ...
'MiniBatchSize',MiniBatchSize, ...
'InitialLearnRate',InitialLearnRate, ...
'LearnRateDropPeriod',3, ...
'LearnRateSchedule','piecewise', ...
'GradientThreshold',1, ...
'Plots','training-progress',...
'shuffle','every-epoch',...
'SequenceLength',500,...
'Verbose',0,...
'ValidationData',{Xval,Yval},...
'ValidationFrequency',50,...
'ValidationPatience',inf,...
'DispatchInBackground',true );
I understand I need to divide the data in some way, but I can't seem to find out how exactly. Can someone explain this to me?
2 个评论
Ben
2022-7-20
What are layers, specifically layers(2)? And what shape is Xtrain?
I wouldn't expect most sequence layers to initialize weights the same size as the sequence length. For 1D convolutions you would typically choose a smaller filter size than the full sequence length (particularly if your sequences are this long).
回答(1 个)
Siraj
2023-9-14
Hi! It is my understanding that you are getting the error because, due to very large input size, the weights matrix for layer 2 is of such dimensions that it requires more memory than permitted.
Here are some potential solutions to address the issue:
1. Apply dimensionality reduction techniques or feature extraction methods to reduce the input size. These techniques aim to reduce the number of features or dimensions in your dataset while retaining important information. For more detailed information and examples, you can refer to the following link:
2. Next you can go to MATLAB > Preferences > Workspace and ensure the maximum array size limit is set to 100%. Then execute 'memory' command in the Command Window and send the output. Ensure that the maximum possible array size is larger than the memory required by the data used in neural network. To learn more about the “memory” command refer to the following link.
3. Also, check that the Java Heap Memory is not set to a very large value because that might restrict the amount of memory available to perform computations. Refer to the following link to adjust the amount of memory that MATLAB allocates for Java objects.
Hope this helps.
0 个评论
另请参阅
类别
在 Help Center 和 File Exchange 中查找有关 Image Data Workflows 的更多信息
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!