read

Read chunk of data in datastore

collapse all in page

Syntax

data = read(dst)
[data,info] = read(simdst)

Description

data = read(dst) reads a chunk of samples from the datastore dst and updates the read position. Subsequent calls to the read function continue reading from the endpoint of the previous call. Use the read function to incrementally process signals that are too large to fit into memory.

  • When the datastore is a matlab.io.datastore.sdidatastore object, the number of samples read by the read function varies. The returned timetable object data always fits into memory.

  • When the data store is a matlab.io.datastore.SimulationDatastore object, use the ReadSize property of the SimulationDatastore object to specify the amount of data, in samples (time steps), to read at a time. Use the progress function and the NumSamples property to determine the current read position.

example

[data,info] = read(simdst) returns information about the extracted data in info.

Examples

collapse all

Open Live Script

A matlab.io.datastore.sdidatastore object references signal data in the Simulation Data Inspector repository. When the signal is too large to fit into memory, you can use the sdidatastore object to incrementally process the data manually or to create a tall timetable for the signal that handles the incremental processing for you.

Create sdidatastore for Signal

Simulate the sldemo_fuelsys model, which is configured to log several signals, to create data in the Simulation Data Inspector repository.

mdl = "sldemo_fuelsys";
sim(mdl);

Logged signal data is returned in a Simulink.SimulationData.Dataset object named sldemo_fuelsys_output.

sldemo_fuelsys_output
sldemo_fuelsys_output = 
Simulink.SimulationData.Dataset 'sldemo_fuelsys_output' with 10 elements

                         Name            BlockPath                                
                         ______________  ________________________________________ 
    1  [1x1 Signal]      ''              sldemo_fuelsys/EGO Fault Switch         
    2  [1x1 Signal]      air_fuel_ratio  sldemo_fuelsys/Engine Gas Dynamics      
    3  [1x1 Signal]      ''              sldemo_fuelsys/Engine Speed Fault Switch
    4  [1x1 Signal]      speed           sldemo_fuelsys/Engine_Speed_Selector    
    5  [1x1 Signal]      ''              sldemo_fuelsys/MAP Fault Switch         
    6  [1x1 Signal]      map             sldemo_fuelsys/MAP_Selector             
    7  [1x1 Signal]      ego             sldemo_fuelsys/O2_Voltage_Selector      
    8  [1x1 Signal]      ''              ...o_fuelsys/Throttle Angle Fault Switch
    9  [1x1 Signal]      throttle        sldemo_fuelsys/Throttle_Angle_Selector  
   10  [1x1 Signal]      fuel            sldemo_fuelsys/To Plant                 

  - Use braces { } to access, modify, or add elements using index.

Use the Simulation Data Inspector programmatic interface to get the signal ID for the signal named speed.

runCount = Simulink.sdi.getRunCount;
latestRunID = Simulink.sdi.getRunIDByIndex(runCount);
latestRun = Simulink.sdi.getRun(latestRunID);
speedSigID = getSignalIDsByName(latestRun,"speed");

Use the signal ID to create an sdidatastore object for the speed signal.

speedSDIds = matlab.io.datastore.sdidatastore(speedSigID);

Verify Contents of Datastore

Check the Name property of the sdidatastore object to verify that it matches your expectations.

speedSDIds.Name
ans = 
'speed'

You can also use the preview function to verify the first ten samples in the signal.

preview(speedSDIds)
ans=10×1 timetable
         Time          Data 
    ______________    ______

    0 sec             314.16
    0.00056195 sec    314.16
    0.0033717 sec     314.16
    0.01 sec          314.16
    0.02 sec          314.16
    0.03 sec          314.16
    0.04 sec          314.16
    0.043098 sec      314.16
    0.043098 sec      314.16
    0.043098 sec      314.16

Process Signal Data with sdidatastore Object

When your signal is too large to fit into memory, you can use the read function to read chunks of data from the Simulation Data Inspector repository to incrementally process your data. Use the hasdata function as the condition for a while loop to incrementally process the whole signal. For example, find the maximum signal value.

latestMax = [];

while hasdata(speedSDIds)
    speedChunk = read(speedSDIds);
    speedChunkData = speedChunk.Data;
    latestMax = max([speedChunkData; latestMax]);
end

latestMax
latestMax = 
314.1593

On each read operation, the read function updates the read position for the start of the next read operation. After reading some or all of the sdidatastore object, you can reset the read position to start again from the beginning of the signal.

reset(speedSDIds)

Process Signal Data in Memory

When the signal referenced by your sdidatastore object fits into memory, you can use the readall function to read all the signal data into memory for processing rather than reading and processing the data incrementally with the read function. The readall function returns a timetable with all the signal data.

speedTimetable = readall(speedSDIds);
speedMax = max(speedTimetable.Data)
speedMax = 
314.1593
Open Live Script

You can log big data from a simulation and inspect and analyze portions of that data by interacting with a matlab.io.datastore.SimulationDatastore object.

Log Big Data from Model

Open the model sldemo_fuelsys.

mdl = "sldemo_fuelsys";
open_system(mdl)

Select Configuration Parameters > Data Import/Export > Log Dataset data to file to log data to persistent storage. Alternatively, you can log data using Dataset format to a MAT file instead of the workspace programmatically. 

set_param(mdl,"LoggingToFile","on")

Simulate the model.

sim(mdl);

The MAT file out.mat appears in your current folder. Logged signal data is stored in the MAT file with the variable name sldemo_fuelsys_out.

Create a DatasetRef object that refers to the logged signal data.

DSRef = Simulink.SimulationData.DatasetRef("out.mat","sldemo_fuelsys_output");

Preview Big Data

Use curly braces to return a SimulationDatastore representation of the fuel signal, which is the tenth element in the DatasetRef object DSRef. The SimulationDatastore object exists in the Values property of the returned Signal object.

SimDataSig = DSRef{10};
DStore = SimDataSig.Values;

Use the preview function to inspect the first ten samples of logged data for the fuel signal.

preview(DStore)
ans=10×1 timetable
         Time          Data 
    ______________    ______

    0 sec             1.2855
    0.00056195 sec    1.2855
    0.0033717 sec     1.2855
    0.01 sec          1.2398
    0.02 sec           1.199
    0.03 sec          1.1628
    0.04 sec          1.1309
    0.043098 sec      1.1309
    0.043098 sec      1.1309
    0.043098 sec      1.1309

Inspect Specific Sample

Suppose you want to inspect the 603rd sample of logged fuel data. Set the ReadSize property of DStore to a number that, considering memory resources, your computer can tolerate. For example, set ReadSize to 200.

DStore.ReadSize = 200;

Read from the datastore three times. Each read operation advances the reading position by 200 samples.

read(DStore);
read(DStore);
read(DStore);

Now that you are close to the 603rd sample, you can set ReadSize to a smaller number to make the target sample easier to find. For example, set ReadSize to 5.

DStore.ReadSize = 5;

Read from the datastore again. The third sample of read data is the 603rd sample in the datastore.

read(DStore)
ans=5×1 timetable
      Time       Data 
    ________    ______

    5.83 sec    1.6609
    5.84 sec    1.6733
    5.85 sec    1.6831
    5.86 sec     1.691
    5.87 sec    1.6975

Inspect Earlier Sample

You can also inspect an earlier sample. For example, inspect the 403rd sample of logged fuel data. Due to previous read operations, the datastore now reads starting from the 606th sample.

Use the reset function to reset DStore. Then, read from the first sample up to the 403rd sample.

reset(DStore);

Set ReadSize to 200.

DStore.ReadSize = 200;

Read from the datastore twice to advance the read position to the 401st sample.

read(DStore);
read(DStore);

Set ReadSize to 5 and read from the datastore. Now, the third sample of read data is the 403rd sample in the datastore.

DStore.ReadSize = 5;
read(DStore)
ans=5×1 timetable
      Time       Data  
    ________    _______

    3.86 sec     0.9895
    3.87 sec    0.98253
    3.88 sec    0.97559
    3.89 sec    0.96867
    3.9 sec     0.96178

Extract Multiple Samples

You can also use the read function to extract multiple samples. For example, extract samples 1001 through 1020.

Reset the datastore. Then, advance to sample 1001 by setting the ReadSize property to 200 and reading the datastore five times.

reset(DStore)

DStore.ReadSize = 200;
for i = 1:5
    read(DStore);
end

Set the ReadSize to 20 to extract 20 samples from the datastore.

DStore.ReadSize = 20;

Extract samples 1001 through 1020. Store the extracted data in a variable named targetSamples.

targetSamples = read(DStore)
targetSamples=20×1 timetable
      Time       Data 
    ________    ______

    9.77 sec      1.68
    9.78 sec    1.6856
    9.79 sec    1.6905
    9.8 sec     1.6948
    9.81 sec    1.6812
    9.82 sec    1.6714
    9.83 sec    1.6642
    9.84 sec     1.659
    9.85 sec    1.6553
    9.86 sec    1.6527
    9.87 sec    1.6684
    9.88 sec    1.6806
    9.89 sec    1.6904
    9.9 sec     1.6982
    9.91 sec    1.7047
    9.92 sec    1.7101
      ⋮

Find Maximum Value of Data in Datastore

Use the hasdata function as the condition for a while loop to incrementally analyze the data in chunks of 200 samples.

reset(DStore);
DStore.ReadSize = 200;
runningMax = [];
while hasdata(DStore)
    tt = read(DStore);
    rawChunk = tt.Data;
    runningMax = max([rawChunk; runningMax]);
end

The variable runningMax stores the maximum value in the entire datastore.

runningMax
runningMax = 
1.7291

Input Arguments

collapse all

Datastore to read, specified as a matlab.io.datastore.sdidatastore or matlab.io.datastore.SimulationDatastore object.

Datastore to read, specified as a matlab.io.datastore.SimulationDatastore object.

Output Arguments

collapse all

Chunk of data read from datastore, returned as a timetable object. For information about timetable objects, see Timetables.

Information about read data, returned as a structure array. The structure has one field, FileName, which is a fully resolved path containing the path string, the name of the file, and the file extension.

Alternatives

When working with matlab.io.datastore.sdidatastore datastores, you can use your sdidatastore object to create a tall timetable to process signals too large to fit into memory. The tall timetable handles loading and processing the chunks of signal data for you. For more information about working with tall timetables, see Tall Arrays.

Version History

Introduced in R2017b

See Also

Objects

Functions

Topics