- Datastore with custom file reader - MATLAB - MathWorks India
- Save and Load Parts of Variables in MAT-Files - MATLAB & Simulink - MathWorks India
- Datastores for Deep Learning - MATLAB & Simulink - MathWorks India
- Large Files and Big Data - MATLAB & Simulink - MathWorks India
Using Sparse Matrix in Series Network
7 次查看(过去 30 天)
显示 更早的评论
I'm attempting to use a Series Network for a classification problem. However, I'm finding that the data I'm using currently is far too large. To deal with this I've been attempting to use the sparse function to turn them into sparse matrices, but I keep getting the error message "Invalid input data type". I've written a smaller script with a simpler data file to test it out, and even switched to a regression problem to see if the issue was the way I was changing the classification problem. However, even if I can get this simple problem to run, trying to change the input to sparse and input into the training function causes errors. Is it possible to use sparse matrices in series networks? Or am I out of luck?
I've attached a test file that contains the very simple script for an example. The dataset being used is the "accident.mat" file from MATLAB.
0 个评论
回答(1 个)
Shivam Malviya
2022-10-11
Hi Rachel,
I understand that you want to work with a large dataset. To reduce the size of the dataset, you are converting it into a sparse matrix.
Is it possible to use sparse matrices in series networks?
No, trainNetwork doesn't support sparse matrix. That said, I have informed the concerned team about this.
To handle large datasets with neural networks, you may use Datastores. I have attached a sample script which loads one row at a time from the hwydata variable in accidents.mat file.
Please refer to the following links for more information;
I hope the above information helps.
2 个评论
Shivam Malviya
2022-10-12
Hi Rachel,
"On a much larger problem, it seems to take a long time to run."
- We can improve the performance by making the batch size and the read size equal. I have updated the script. Please find the attached script.
I could not execute the attached script because I don't have "preTexas.mat". Could you please share that file?
Also, is it using less memory now?
- If not, make sure that your MAT file version is 7.3. As other versions do not support partial loading.
- You may use the following workflow to check the version
- Execute the following command
type <MAT-File>.mat
- Check the comment at the beginning of the file.
- Make sure it is like the one below.
MATLAB 7.3 MAT-file
- If it is not, do as follows;
load('<MAT-File>.mat');
save('mycopy.mat','-v7.3');
Please find the attached script.
I hope this helps!
另请参阅
类别
在 Help Center 和 File Exchange 中查找有关 Sequence and Numeric Feature Data Workflows 的更多信息
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!