ImageDataStore and tall array, How to use to save Labels and 4D Matrices in for loop?

7 次查看（过去 30 天）

M 2023-11-7

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/2043962-imagedatastore-and-tall-array-how-to-use-to-save-labels-and-4d-matrices-in-for-loop

评论： Matt J 2023-11-13

I generate 4D matrices of images (Double) and their respective labels (Double) with huge sizes in each iteration,

for example in one iteration I get images with size 120*120*1*6000 where 6000 the number of images, 1 is the number of channels, and their respective lables 1*6000

How to use ImageDataStore and tall array to store Labels and 4D Matrices in for loop in each iteration to use it then in further analysis in machine learning?

Also, Is there any other efficient solution to deal with huge data?

Note that the number of images and their labels vary in each iteration

4 个评论
显示 2更早的评论隐藏 2更早的评论

M 2023-11-13

编辑：M 2023-11-13

@Walter Robersonany hint how to pass in each iteration the 4D double matrices and double lables to imagedatastore and tall arrays please?

Because in the documentation there is no double format supported!

SupportedOutputFormats: ["png" "jpg" "jpeg" "tif" "tiff"]

M 2023-11-13

Hi @Divya Gaddipati , Do you have any idea please? thanks

请先登录，再进行评论。

请先登录，再回答此问题。

回答（3 个）

Jorrit Montijn 2023-11-7

1
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/2043962-imagedatastore-and-tall-array-how-to-use-to-save-labels-and-4d-matrices-in-for-loop#answer_1347912

I'm not sure what your workflow exactly looks like, but if you transform your images from double-precision floating numbers (64 bits per pixel) to uint8 (8 bits per pixel) you'll only need 1/8th of the memory capacity.

In most cases this won't even cause a loss of data, as images are often encoded in 8-bit depth anyway. In each iteration, you can then transform the (set of) image(s) you're working on back to double-precision floats so you don't have to change your workflow.

Transform the whole stack:

Ims_uint8 = cast(Ims_double*255,'uint8');

Then in each iteration:

Ims_double = cast(Ims_uint8,'double')/255;

2 个评论
显示无隐藏无

Walter Roberson 2023-11-7

im2uint8 and im2double potentially

M 2023-11-12

@Walter Roberson @Jorrit Montijn extremely slow, not efficient solution

请先登录，再进行评论。

Matt J 2023-11-7

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/2043962-imagedatastore-and-tall-array-how-to-use-to-save-labels-and-4d-matrices-in-for-loop#answer_1347992

imageDatastores are provided for situations like that.

8 个评论
显示 6更早的评论隐藏 6更早的评论

M 2023-11-13

编辑：M 2023-11-13

@Matt J still I am not getting the process, what do you mean by Each image should be kept in its own file, which the data store would read in one-by-one as you loop

do you mean that I have to save all the files first then put them in the imds? What is the benefits of doing that??

If your images must be doubles (though I wonder why),

to not change the work flow

you can put them in individual .mat files.

how to do that? by them in the desk??

I cant find any simple example, the process is complex!

Matt J 2023-11-13

do you mean that I have to save all the files first then put them in the imds? What is the benefits of doing that??

Yes. That way, the images are stored on disk, rather than consuming RAM.

请先登录，再进行评论。

Image Analyst 2023-11-7

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/2043962-imagedatastore-and-tall-array-how-to-use-to-save-labels-and-4d-matrices-in-for-loop#answer_1348027

Cell arrays are very inefficient compared to regular numerical arrays. They use a lot more memory. You could even use single instead of double to preallocate using half the memory.

Not sure what you're doing but you may be doing what a lot of beginners do and that is to read ALL the input images into one huge array in a loop, then after the loop process each slice of the huge array as an individual image. This is usually NOT the way to process a sequence of images. You usually (if it's a series of 2-D images you want to analyze and not a volumetric image that you're reading in slice-by-slice) want to have a loop where you read in one image at a time and then analyze it immediately in that same iteration. See the FAQ:

https://matlab.fandom.com/wiki/FAQ#How_can_I_process_a_sequence_of_files?

Another option is an imageDatastore as @Matt J already mentioned.

If you must read all the images into memory at once time, then try to use a single array rather than a cell array.