What Is a Datastore?

A datastore is an object for reading a single file or a collection of files or data. The properties of a datastore describe the data and specify how to read the data from the datastore.

The advantages of using datastores to work with deep learning data sets include:

Directly loading your data into MATLAB^® might be appropriate when working with a very small data set, but for larger data sets and more complex networks such as networks with multiple inputs or outputs, use a datastore.

How to Create and Read from Datastores

Choose the data that you want to use for deep learning. This code downloads the Omniglot training data set, which contains images showing handwritten characters from 50 alphabets [1].

downloadFolder = tempdir;
url = "https://github.com/brendenlake/omniglot/raw/master/python/images_background.zip";
filename = fullfile(downloadFolder,"images.zip");
dataFolder = fullfile(downloadFolder,"images_background");

if ~exist(dataFolder,"dir")
    fprintf("Downloading Omniglot training data set (4.5 MB)... ")
    websave(filename,url)
    unzip(filename,downloadFolder)
    fprintf("Done.\n")
end

Create an imageDatastore object that includes all files and subfolders within dataFolder. If you have another type of data, choose another built-in datastore. For more information, see Select Datastore.

imds = imageDatastore(dataFolder,IncludeSubfolders=true);

To verify that the datastore can access the images, read an image from the datastore using the read function, and display it.

I = read(imds);
imshow(I)

Subsequent calls to read continue reading from the endpoint of the previous call. To reset the datastore to the state where no data has been read from it, use the reset function.

reset(imds)

Use Datastores for Training, Validation, and Inference

Datastores are valid inputs for training, validation, and inference.

data = read(ds)

data =

  4×2 cell array

    {224×224×3 double}    {[2]}
    {224×224×3 double}    {[7]}
    {224×224×3 double}    {[9]}
    {224×224×3 double}    {[9]}

trainingOptions("adam", ...
InputDataFormats="CTB");

Transform Datastores

A transformed datastore applies a particular data transformation to an underlying datastore when reading data. To create a transformed datastore, use the transform function, and specify the underlying datastore and the transformation.

For simple transformations that you can express in one line of code, you can specify a handle to an anonymous function as the @fcn argument of transform. For more information, see Anonymous Functions. For example, you can use the transform function to create a transformed datastore that applies the imresize function to resize images when you read them from the datastore.

imageSize = [244 244];
tds = transform(imds,@(I) imresize(I,imageSize))

For more complex transformations involving several preprocessing operations, define the complete set of transformations in your own function. Then, specify a handle to your function as the @fcn argument of transform. For an example showing how to apply a custom preprocessing function using the transform function, see Prepare Datastore for Image-to-Image Regression.

The function handle provided to transform must accept input data in the same format as returned by the read function of the underlying datastore.

Combine Datastores

The combine function associates multiple datastores with each other. Calling the read function of a combined datastore reads one batch of data from all of the N underlying datastores, which must return the same number of observations. Reading from a combined datastore returns the horizontally concatenated results in an N-column cell array that is suitable for training and validation.

For example, if you are training an image-to-image regression network, then you can create the training data set by combining two image datastores. This sample code demonstrates combining two image datastores named imdsX and imdsY. The combined datastore imdsTrain returns data as a two-column cell array.

imdsTrain = combine(imdsX,imdsY);
images = read(imdsTrain)

images =

  1×2 cell array

    {105×105 logical}    {105×105 logical}

For an example showing how to combine datastores using the combine function, see Train Network Using Out-of-Memory Sequence Data.

Train Networks with Multiple Inputs or Outputs Using Datastores

To train a network with multiple input layers or multiple outputs, use the combine and transform functions to create a datastore that outputs a cell array with (numInputs + numOutputs) columns, where numInputs is the number of network inputs and numOutputs is the number of network outputs. The first numInputs columns specify the predictors for each input, and the last numOutputs columns specify the responses. The InputNames and OutputNames properties of the neural network determine the order of the inputs and outputs, respectively.

This table shows example outputs of calling the read function for a datastore, ds.

data = read(ds)

data =

  4×2 cell array

    {224×224×3 double}    {[2]}
    {224×224×3 double}    {[7]}
    {224×224×3 double}    {[9]}
    {224×224×3 double}    {[9]}

data = read(ds)

data =

  4×2 table

        Predictors        Response
    __________________    ________

    {224×224×3 double}       2    
    {224×224×3 double}       7    
    {224×224×3 double}       9    
    {224×224×3 double}       9  

data = read(ds)

data =

  4×4 cell array

    {224×224×3 double}    {128×128×3 double}    {[2]}    {[-42]}
    {224×224×3 double}    {128×128×3 double}    {[2]}    {[-15]}
    {224×224×3 double}    {128×128×3 double}    {[9]}    {[-24]}
    {224×224×3 double}    {128×128×3 double}    {[9]}    {[-44]}

For an example showing how to train a network with multiple inputs using a combined datastore, see Train Network on Image and Feature Data. For more information about networks with multiple inputs and outputs, see Multiple-Input and Multiple-Output Networks.

Select Datastore

For many applications, the easiest approach is to start with a built-in datastore. For more information about the available built-in datastores, see Select Datastore for File Format or Application. However, you can use only some types of built-in datastores as direct input for network training, validation, and inference.

You can use other built-in datastores as input for deep learning, but you must preprocess the data read from these datastores into a format required by a deep learning network. With the built-in datastores and the transform and combine functions, you can use datastores for the majority of your deep learning training and prediction tasks. For more information on the required format of read data, see Layout of Predictors and Targets, and for more information on how to preprocess data read from datastores, see Transform Datastores and Combine Datastores.

For some applications, there may not be a built-in datastore type that fits your data well. In these cases, you can create a custom datastore. For more information, see Develop Custom Datastore. All custom datastores are valid inputs to deep learning interfaces as long as the read function of the custom datastore returns data in the required form.

Quantization supports many of the built-in datastores. For more information, see Prepare Data for Quantizing Networks.