Get Started with MDF Datastore

Open Live Script

This example shows you how to use the MDF datastore feature of Vehicle Network Toolbox™ to quickly and efficiently process a data set spread across a collection of multiple MDF files. This workflow is also valuable when there are too much data to fit into available memory.

Access MDF Files in a Datastore

Find the collection of MDF files representing logged information from multiple test sequences. Note that MDF files to be used by MDF datastore as a set must have the same channel group and channel content structure.

dir("File*.mf4")

File01.mf4  File02.mf4  File03.mf4  File04.mf4  File05.mf4

Create an MDF Datastore

You create an MDF datastore by selecting a folder location containing a collection of MDF files. In this case, target all files in the current working directory.

mds = mdfDatastore(pwd)

mds = 
  MDFDatastore with properties:

  Datastore Details
                         Files: {
                                '/tmp/Bdoc24b_2725827_3770987/tpf6b9d6b0/vnt-ex10761765/File01.mf4';
                                '/tmp/Bdoc24b_2725827_3770987/tpf6b9d6b0/vnt-ex10761765/File02.mf4';
                                '/tmp/Bdoc24b_2725827_3770987/tpf6b9d6b0/vnt-ex10761765/File03.mf4'
                                 ... and 2 more
                                }
                 ChannelGroups: 
                                  GroupNumber    AcquisitionName       Comment           ... and 10 more columns    
                                  ___________    _______________    _____________                

                                       1           <undefined>      Integer Types    {0x0 double}
                                       2           <undefined>      Float Types      {0x0 double}


                      Channels: 
                                        Name           GroupNumber    DisplayName        ... and 17 more columns    
                                  _________________    ___________    ___________                

                                  "Float_32_LE"             2             ""         {0x0 double}
                                  "Float_64_LE"             2             ""         {0x0 double}
                                  "Signed_Int16_LE"         1             ""         {0x0 double}

                                ... and 3 more rows

  Options
          SelectedChannelNames: {
                                'Signed_Int16_LE';
                                'Unsigned_UInt32_LE';
                                'time'
                                }
    SelectedChannelGroupNumber: 1
                      ReadSize: "file"
                       ReadRaw: 0
                  ValidityRule: "include"
                 IncludeEvents: 1

Configure MDF Datastore

Multiple options allow control of what data are read from the MDF files and how the reads are performed. In this case, the first channel group is used by default. Note that only one channel group may be selected by the datastore at a time. You can also specify certain channels within the selected channel group to read. In this case, all channels are read by default.

mds.SelectedChannelGroupNumber

ans = 
1

mds.SelectedChannelNames

ans = 3x1 string
    "Signed_Int16_LE"
    "Unsigned_UInt32_LE"
    "time"

Preview MDF Datastore

Using the preview function, you can obtain a quick view of the data available in the file set. Preview always returns up to eight data points from the first file in the datastore.

preview(mds)

ans=8×2 timetable
        time     Signed_Int16_LE    Unsigned_UInt32_LE
        _____    _______________    __________________

        0 sec           0                   0         
        1 sec           1                   1         
        2 sec           2                   2         
        3 sec           3                   3         
        4 sec           4                   4         
        5 sec           5                   5         
        6 sec           6                   6         
        7 sec           7                   7

Read All Data in MDF Datastore

You can use the readall function to read the entire data in a single call. This is an efficient way to read from many files when the data set fits into available memory. After running readall, the datastore resets to the beginning of the data set.

data = readall(mds);
data(1:5,:)

ans=5×2 timetable
        time     Signed_Int16_LE    Unsigned_UInt32_LE
        _____    _______________    __________________

        0 sec           0                   0         
        1 sec           1                   1         
        2 sec           2                   2         
        3 sec           3                   3         
        4 sec           4                   4

Read a Subset of Data in MDF Datastore

You can use the read function to obtain a subset of data from the datastore. The size of the subset is determined by the ReadSize property of the MDF datastore object. By default, data from an entire file are read in one call. The power of a datastore comes from reading through multiple files sequentially within the file set. As you read, the datastore automatically bridges from one file to the next until all data from all files are read.

for ii = 1:3
    data = read(mds);
    whos("data")
    data(1:5,:)
end

  Name          Size             Bytes  Class        Attributes

  data      10000x2             246364  timetable

ans=5×2 timetable
        time     Signed_Int16_LE    Unsigned_UInt32_LE
        _____    _______________    __________________

        0 sec           0                   0         
        1 sec           1                   1         
        2 sec           2                   2         
        3 sec           3                   3         
        4 sec           4                   4

  Name          Size             Bytes  Class        Attributes

  data      10000x2             246364  timetable

ans=5×2 timetable
        time     Signed_Int16_LE    Unsigned_UInt32_LE
        _____    _______________    __________________

        0 sec           0                   0         
        1 sec           1                   1         
        2 sec           2                   2         
        3 sec           3                   3         
        4 sec           4                   4

  Name          Size             Bytes  Class        Attributes

  data      10000x2             246364  timetable

ans=5×2 timetable
        time     Signed_Int16_LE    Unsigned_UInt32_LE
        _____    _______________    __________________

        0 sec           0                   0         
        1 sec           1                   1         
        2 sec           2                   2         
        3 sec           3                   3         
        4 sec           4                   4

Reset MDF Datastore

At any time, you can call the reset function to start over at the beginning of the data set.

reset(mds)

Configure Number of Records to Read from MDF Datastore

You can use the ReadSize property to specify how much data to read on each call. ReadSize can be specified as a numeric value to read a fixed number of data points. ReadSize lets you control how much data is loaded into memory when you have a data set larger than available memory. It is recommended to use custom read sizes that are small enough to fit in memory, but still as large as possible to reduce processing overhead and improve performance.

mds.ReadSize = 5

mds = 
  MDFDatastore with properties:

  Datastore Details
                         Files: {
                                '/tmp/Bdoc24b_2725827_3770987/tpf6b9d6b0/vnt-ex10761765/File01.mf4';
                                '/tmp/Bdoc24b_2725827_3770987/tpf6b9d6b0/vnt-ex10761765/File02.mf4';
                                '/tmp/Bdoc24b_2725827_3770987/tpf6b9d6b0/vnt-ex10761765/File03.mf4'
                                 ... and 2 more
                                }
                 ChannelGroups: 
                                  GroupNumber    AcquisitionName       Comment           ... and 10 more columns    
                                  ___________    _______________    _____________                

                                       1           <undefined>      Integer Types    {0x0 double}
                                       2           <undefined>      Float Types      {0x0 double}


                      Channels: 
                                        Name           GroupNumber    DisplayName        ... and 17 more columns    
                                  _________________    ___________    ___________                

                                  "Float_32_LE"             2             ""         {0x0 double}
                                  "Float_64_LE"             2             ""         {0x0 double}
                                  "Signed_Int16_LE"         1             ""         {0x0 double}

                                ... and 3 more rows

  Options
          SelectedChannelNames: {
                                'Signed_Int16_LE';
                                'Unsigned_UInt32_LE';
                                'time'
                                }
    SelectedChannelGroupNumber: 1
                      ReadSize: 5
                       ReadRaw: 0
                  ValidityRule: "include"
                 IncludeEvents: 1

for ii = 1:3
    data = read(mds)
end

data=5×2 timetable
        time     Signed_Int16_LE    Unsigned_UInt32_LE
        _____    _______________    __________________

        0 sec           0                   0         
        1 sec           1                   1         
        2 sec           2                   2         
        3 sec           3                   3         
        4 sec           4                   4

data=5×2 timetable
        time     Signed_Int16_LE    Unsigned_UInt32_LE
        _____    _______________    __________________

        5 sec           5                   5         
        6 sec           6                   6         
        7 sec           7                   7         
        8 sec           8                   8         
        9 sec           9                   9

data=5×2 timetable
         time     Signed_Int16_LE    Unsigned_UInt32_LE
        ______    _______________    __________________

        10 sec          10                   10        
        11 sec          11                   11        
        12 sec          12                   12        
        13 sec          13                   13        
        14 sec          14                   14

Configure a Time Range to Read from MDF Datastore

You can also specify ReadSize as a duration to read data points by elapsed time. Note that when the read type is changed, the datastore resets to the beginning of the data set.

mds.ReadSize = seconds(5)

mds = 
  MDFDatastore with properties:

  Datastore Details
                         Files: {
                                '/tmp/Bdoc24b_2725827_3770987/tpf6b9d6b0/vnt-ex10761765/File01.mf4';
                                '/tmp/Bdoc24b_2725827_3770987/tpf6b9d6b0/vnt-ex10761765/File02.mf4';
                                '/tmp/Bdoc24b_2725827_3770987/tpf6b9d6b0/vnt-ex10761765/File03.mf4'
                                 ... and 2 more
                                }
                 ChannelGroups: 
                                  GroupNumber    AcquisitionName       Comment           ... and 10 more columns    
                                  ___________    _______________    _____________                

                                       1           <undefined>      Integer Types    {0x0 double}
                                       2           <undefined>      Float Types      {0x0 double}


                      Channels: 
                                        Name           GroupNumber    DisplayName        ... and 17 more columns    
                                  _________________    ___________    ___________                

                                  "Float_32_LE"             2             ""         {0x0 double}
                                  "Float_64_LE"             2             ""         {0x0 double}
                                  "Signed_Int16_LE"         1             ""         {0x0 double}

                                ... and 3 more rows

  Options
          SelectedChannelNames: {
                                'Signed_Int16_LE';
                                'Unsigned_UInt32_LE';
                                'time'
                                }
    SelectedChannelGroupNumber: 1
                      ReadSize: 5 sec
                       ReadRaw: 0
                  ValidityRule: "include"
                 IncludeEvents: 1

for ii = 1:3
    data = read(mds)
end

data=6×2 timetable
        time     Signed_Int16_LE    Unsigned_UInt32_LE
        _____    _______________    __________________

        0 sec           0                   0         
        1 sec           1                   1         
        2 sec           2                   2         
        3 sec           3                   3         
        4 sec           4                   4         
        5 sec           5                   5

data=11×2 timetable
         time     Signed_Int16_LE    Unsigned_UInt32_LE
        ______    _______________    __________________

        0 sec            0                    0        
        1 sec            1                    1        
        2 sec            2                    2        
        3 sec            3                    3        
        4 sec            4                    4        
        5 sec            5                    5        
        6 sec            6                    6        
        7 sec            7                    7        
        8 sec            8                    8        
        9 sec            9                    9        
        10 sec          10                   10

data=16×2 timetable
         time     Signed_Int16_LE    Unsigned_UInt32_LE
        ______    _______________    __________________

        0 sec            0                    0        
        1 sec            1                    1        
        2 sec            2                    2        
        3 sec            3                    3        
        4 sec            4                    4        
        5 sec            5                    5        
        6 sec            6                    6        
        7 sec            7                    7        
        8 sec            8                    8        
        9 sec            9                    9        
        10 sec          10                   10        
        11 sec          11                   11        
        12 sec          12                   12        
        13 sec          13                   13        
        14 sec          14                   14        
        15 sec          15                   15

Close MDF Files

Close access to the MDF files by clearing the MDF datastore variable from workspace.

clear mds