Training a convolutional neural network with matconvnet using an hdf5 file.

Question

Samuel Spencer 2017-10-25

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/363164-training-a-convolutional-neural-network-with-matconvnet-using-an-hdf5-file

评论： sally 2018-1-14

Hi,

I have a large dataset (~1gb, hopefully with expansion to 100gb) stored hierarchically in an hdf5 file which I'd like to use with neural networks, specifically the MatConvNet package with MexConv3D (and R2016b). Ideally, I'd like to not have to load the entire file into memory, so is there a way of achieving something similar to the 'matfile' command with hdf5? Alternatively, is there a way to do something along the lines of (in pseudocode):

for all images in file:
    image=hdfread('/Path/to/dataentry') #Load in one image
    train_neural_net(image)  #Do bit of training needed just on one image in the datafile.
    deallocate(image) #Wipe image from memory, keeping just the net

Many thanks in advance

2 个评论
显示无隐藏无

per isakson 2017-10-26

"something similar to the 'matfile' command with hdf5" No.
"Alternatively, is there a way to do something along the lines of (in pseudocode):" Yes, that's straightforward.

sally 2018-1-14

an unrelated comment to the answer.. but a question please, have u used the MexConv3D package as a CNN for 3D input images? if so, how to train this algorithm on a large dataset of 3D images? how to update the weights..

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Amy 2017-10-27

1
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/363164-training-a-convolutional-neural-network-with-matconvnet-using-an-hdf5-file#answer_288152

Hi Samuel,

There is not a way to access the data in an hdf5 file without loading it into memory, but there are examples of reading in subsets of data using 'h5read' in its documentation.

It is definitely possibly to train your network in a loop as you describe, loading and unloading the training data for each loop iteration. There are a few tricks you have to use to get this to work:

You have to make sure that the network is configured only to use input data as training data.
You have to use the training algorithm “traingd”.

Note that doing the training this way is very slow (switching off the GUI might help speed it up). You may not want to train for each individual image, but rather for smaller batches of images.