Check if your custom datastore constructor works with the minimal required inputs.

Datastore object of your custom datastore type with the minimal expected properties and methods

Check if your datastore object ds has matlab.io.Datastore as one of its superclasses.

Run this command:

isa(ds,'matlab.io.Datastore') 

1 or true

Call your custom datastore constructor with the required inputs and any supported input arguments and name-value pair arguments.

Datastore object of your custom datastore type with the minimal expected properties and methods

Call the read method on a datastore object ds.

t = read(ds);

Data from the beginning of the datastore

If you specify read size, then the size of the returned data is equivalent to read size.

Call the read method again on the datastore object.

t = read(ds);

Data starting from the end point of the previous read operation

If you specify read size, then the size of the returned data is equivalent to read size.

Continue calling the read method on the datastore object in a while loop.

while(hasdata(ds))
  t = read(ds);
end

No errors

Correct data in the correct format

When data is available to read, check the info output (if any) of the read method.

Call a datastore object ds.

[t,info] = read(ds);

No error

info contains the expected information

t contains the expected data

When no more data is available to read, call read on the datastore object.

Either expected output or an error message based on your custom datastore implementation.

Call the readall method on the datastore object.

All data

Call the readall method on the datastore object, when hasdata(ds) is false.

Read from the datastore until hasdata(ds) is false, and then call the readall method.

while(hasdata(ds))
  t = read(ds);
end

readall(ds)

All data

Call the hasdata method on the datastore object before making any calls to read

true

Call the hasdata method on the datastore object after making a few calls to read, but before all the data is read

true

When more data is available to read, call the readall method, and then call the hasdata method.

true

When no more data is available to read, call the hasdata method.

false

Call the reset method on the datastore object before making any calls to the read method.

Verify that the read method returns the appropriate data after a call to the reset method.

reset(ds);
t = read(ds);

No errors

The read returns data from the beginning of the datastore.

If you specify read size, then the size of the returned data is equivalent to read size.

When more data is available to read, call the reset method after making a few calls to the read method.

Verify that the read method returns the appropriate data after making a call to the reset method.

No errors

The read method returns data from the beginning of the datastore.

If you specify read size, then the size of the returned data is equivalent to read size.

When more data is available to read, call the reset method after making a call to the readall method.

Verify that the read method returns the appropriate data after making a call to the reset method.

No errors

The read method returns data from the beginning of the datastore.

If you specify read size, then the size of the returned data is equivalent to read size.

When no more data is available to read, call the reset method on the datastore object and then call the read method

Verify that read returns the appropriate data after a call to the reset method.

No errors

The read method returns data from the beginning of the datastore.

If you specify read size, then the size of the returned data is equivalent to read size.

Call the progress method on the datastore object before making any calls to the read method.

0 or an expected output based on your custom datastore implementation.

Call the progress method on the datastore object after making a call to readall, but before making any calls to read

readall(ds);
progress(ds)

0 or an expected output based on your custom datastore implementation.

Call the progress method on the datastore object after making a few calls to read and while more data is available to read.

A fraction between 0 and 1 or an expected output based on your custom datastore implementation.

Call the progress method on the datastore object when no more data is available to read.

1 or an expected output based on your custom datastore implementation.

Call preview on the datastore object before making any calls to read.

The preview method returns the expected data from the beginning of the datastore, based on your custom datastore implementation.

Call preview on the datastore object after making a few calls to read and while more data is available to read.

The preview method returns the expected data from the beginning of the datastore, based on your custom datastore implementation.

Call preview on the datastore object after making a call to readall and while more data is available to read.

The preview method returns the expected data from the beginning of the datastore, based on your custom datastore implementation.

Call preview on the datastore object after making a few calls to read and a call to reset.

The preview method returns the expected data from the beginning of the datastore, based on your custom datastore implementation.

Call preview on the datastore object when no more data is available to read.

The preview method returns the expected data from the beginning of the datastore, based on your custom datastore implementation.

Call preview after making a few calls to read method and then call read again.

The read method returns data starting from the end point of the previous read operation.

If you specify read size, then the size of the returned data is equivalent to read size.

Call preview, and then call readall on the datastore.

The readall method returns all the data from the datastore.

While datastore has data available to read, call preview, and then call hasdata.

The hasdata method returns true.

Call partition on the datastore object ds with a valid number of partitions and a valid partition index.

Call read on a partition of the datastore and verify the data.

subds = partition(ds,n,index)
read(subds)

Verify that the partition is valid.

isequal(properties(ds),properties(subds))
isequal(methods(ds),methods(subds))

The partition method partitions the datastore into n partitions and returns the partition corresponding to the specified index.

The returned partition subds must be a datastore object of your custom datastore.

The partitioned datastore subds must have the same methods and properties as the original datastore.

The isequal statement returns true.

Calling read on the partition returns data starting from the beginning of the partition.

If you specify read size, then the size of the returned data is equivalent to read size.

Call partition on the datastore object ds with number of partitions specified as 1 and index of returned partition specified as 1.

Verify the data returned by calling read and preview on a partition of the partitioned datastore.

subds = partition(ds,1,1)
isequal(properties(ds),properties(subds))
isequal(methods(ds),methods(subds))
isequaln(read(subds),read(ds))
isequaln(preview(subds),preview(ds))

The partition subds must be a datastore object of your custom datastore.

The partition subds must have the same methods and properties as the original datastore ds.

The isequal and isequaln statements returns true.

Call partition on the partition subds with a valid number of partitions and a valid partition index.

The repartitioning of a partition of the datastore should work without errors.

Call initializeDatastore on the datastore object ds with a valid info struct.

The info struct contains these fields:

FileName is of data type char and the fields Offset and Size are of the data type double.

For example, initialize the info struct, and then call initializeDatastore on the datastore object ds.

info = struct('FileName','myFileName.ext',...
                       'Offset',0,'Size',500)
initializeDatastore(ds,info)

Verify the initialization by examining the properties of your datastore object.

ds

The initializeDatastore method initializes the custom datastore object ds with the necessary information from the info struct.

Call getLocation on the datastore object.

location = getLocation(ds)

Based on your custom datastore implementation, the location output is either of these:

If location is a matlab.io.datastore.DsFileSet object, then call resolve to verify the files in the location output.

resolve(location)

The getLocation method returns the location of files in Hadoop.

Call isfullfile on the datastore object.

Based on your custom datastore implementation, the isfullfile method returns true or false.

Create a tall array by calling tall on the datastore object ds.

t = tall(ds)

The tall function returns an output that is the same data type as the output of the read method of the datastore.

For this test step, create a datastore object with data that fits in your system memory. Then, create a tall array using this datastore object.

t = tall(ds)

If your data is numeric, then apply an appropriate function like the mean function to both the ds and t, then compare the results.

If your data is of the data type string or categorical, then apply the unique function on a column of ds and a column of t, then compare the results.

Apply gather and verify the result.

For examples, see Big Data Workflow Using Tall Arrays and Datastores (Parallel Computing Toolbox).

No errors

The function returns an output of the correct data type (not of a tall data type).

The function returns the same result whether it is applied to ds or to t.

Call mapreduce on the datastore object ds.

outds = mapreduce(ds,@mapper,@reducer)

To support the use of the mapreduce function, the read method of your custom datastore must return both the info and the data output arguments.

No error

The MapReduce operation returns the expected result