Memory issue to create 4D array

2019 4 7
1 个回答
更新时间：2019 4 8
3 次查看（30 天）

0 个投票

I have 35000 funds image data-set i want to create 4D array of this data-set but when i ran the code and read images one by one than error occurred memory limitation . plz help me to resolve issue.
my code is here.
close all
clear all
Directory= 'C:\Users\Shehzad\Desktop\sample\*.jpeg';
Imgs = dir(Directory);
for j=1:length(Imgs)
thisname = Imgs(j).name;
D=Imgs(j).folder;
thisfile = fullfile(D, thisname);
Img = imread(thisfile); % try to read image
data(:,:,:,j)=Img;
clear Img;
end
save Data data
save Imgs Imgs
Output is:
Requested 2500x2700x3x630 (11.9GB) array exceeds maximum array size preference. Creation of arrays greater than this limit
may take a long time and cause MATLAB to become unresponsive. See array size limit or preference panel for more
information.
Error in Project (line 10)
data(:,:,:,j)=Img;

请先登录，再回答此问题。

回答（1 个）

Rik
Rik 2019-4-7

0 个投票

The amount of data is probably huge, is not pre-allocated and will probably not fit in memory. Have you calculated an estimated size of the needed array?
If you can't add fysical RAM to your system, there are some tools that you can use to access data still on your disk and only load them to RAM when you need it, like datastore and imageDatastore.

6 个评论
显示 4更早的评论 隐藏 4更早的评论

Shehzad Ali
Shehzad Ali 2019-4-8
First of all thank you.
yes I have done it. its 300GB approximately.
but I divide my dataset into 3 parts, 11706 images in each part. We need appx. 115GB memory required for each part. How can I use datastore and imageDatastore plz tell me about it.
Walter Roberson
Walter Roberson 2019-4-8
What would you do with a 115 gigabytes dataset in memory if you were able to construct it?
How much memory do you have?
Shehzad Ali
Shehzad Ali 2019-4-8
I want to make 4D array of that dataset and then save as data.mat file.
I have 12GB RAM.
Walter Roberson
Walter Roberson 2019-4-8
If you use matfile and a -v7.3 .mat then you might be able to extend it one slice at a time without storing the entire array in memory.
What are you going to do with the 140 or so gigabyte .mat file that you get as output? You will never be able to load it on your computer.
Shehzad Ali
Shehzad Ali 2019-4-8
I am working on Dibatic retinopathy dectection using convolutional neural network. once i made it then I load one by one image through imageDatastore.
Walter Roberson
Walter Roberson 2019-4-8
Here it sounds like the "one by one image" that you intend to load with imageDatastore would be each of the three 4D arrays that are 115 gigabytes each ??
I thought Diabetic Retinopathy involved individual RGB images (so 3D), rather than volume scans or time series ?

请先登录，再进行评论。

类别

在 帮助中心 和 File Exchange 中查找有关 Image Arithmetic 的更多信息

产品

标签

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by