Main Content

Index and View Tall Array Elements

Tall arrays are too large to fit in memory, so it is common to view subsets of the data rather than the entire array. This page shows techniques to extract and view portions of a tall array.

Extract Top Rows of Array

Use the head function to extract the first rows in a tall array. head does not force evaluation of the array, so you must use gather to view the result.

tt = tall(table(randn(1000,1),randn(1000,1),randn(1000,1)))
tt =

  1,000×3 tall table

      Var1        Var2        Var3  
    ________    ________    ________

     0.53767      0.6737     0.29617
      1.8339    -0.66911      1.2008
     -2.2588    -0.40032      1.0902
     0.86217     -0.6718     -0.3587
     0.31877     0.57563    -0.12993
     -1.3077    -0.77809     0.73374
    -0.43359     -1.0636     0.12033
     0.34262     0.55298      1.1363
       :           :           :
       :           :           :
t_head = gather(head(tt))
t_head =

  8×3 table

      Var1        Var2        Var3  
    ________    ________    ________

     0.53767      0.6737     0.29617
      1.8339    -0.66911      1.2008
     -2.2588    -0.40032      1.0902
     0.86217     -0.6718     -0.3587
     0.31877     0.57563    -0.12993
     -1.3077    -0.77809     0.73374
    -0.43359     -1.0636     0.12033
     0.34262     0.55298      1.1363

Extract Bottom Rows of Array

Similarly, you can use the tail function to extract the bottom rows in a tall array.

t_tail = gather(tail(tt))
t_tail =

  8×3 table

      Var1        Var2        Var3  
    ________    ________    ________

     0.64776     0.47349    -0.27077
    -0.31763      1.3656     0.43966
       1.769     -1.6378    -0.50614
      1.5106      2.0237    -0.18435
     0.16401     0.77779       0.402
    -0.28276     -0.5489     0.53923
      1.1522    -0.12601    -0.73359
     -1.1465     0.29958    -0.26837

Indexing Tall Arrays

All tall arrays support parentheses indexing. When you index a tall array using parentheses, such as T(A) or T(A,B), the result is a new tall array containing only the specified rows and columns (or variables).

Like most other operations on tall arrays, indexing expressions are not evaluated immediately. You must use gather to evaluate the indexing operation. For more information, see Lazy Evaluation of Tall Arrays.

You can perform these types of indexing in the first dimension of a tall array:

  • B = A(:,…), where : selects all rows in A.

  • B = A(idx,…), where idx is a tall numeric column vector or non-tall numeric vector.

  • B = A(L,…), where L is a tall or non-tall logical array of the same height as A. For example, you can use relational operators, such as tt(tt.Var1 < 10,:). When you index a tall array with a tall logical array, there are a few requirements. Each of the tall arrays:

    • Must be the same size in the first dimension.

    • Must be derived from a single tall array.

    • Must not have been indexed differently in the first dimension.

  • B = A(P:D:Q,…) or B = A(P:Q,…), where P:D:Q and P:Q are valid colon indexing expressions.

Additionally, the number of subscripts you must specify depends on how many dimensions the array has:

  • For tall column vectors, you can specify a single subscript such as t(1:10).

  • For tall row vectors, tall tables, and tall timetables, you must specify two subscripts.

  • For tall arrays with two or more dimensions, you must specify two or more subscripts. For example, if the array has three dimensions, you can use an expression such as tA(1:10,:,:) or tA(1:10,:), but not linear indexing expressions such as tA(1:10) or tA(:).

Tip

The find function locates nonzero elements in tall column vectors, and can be useful to generate a vector of indices for elements that meet particular conditions. For example, k = find(X<0) returns the linear indices for all negative elements in X.

For example, use parentheses indexing to retrieve the first ten rows of tt.

tt(1:10,:)
ans =

  10×3 tall table

      Var1        Var2        Var3  
    ________    ________    ________

     0.53767      0.6737     0.29617
      1.8339    -0.66911      1.2008
     -2.2588    -0.40032      1.0902
     0.86217     -0.6718     -0.3587
     0.31877     0.57563    -0.12993
     -1.3077    -0.77809     0.73374
    -0.43359     -1.0636     0.12033
     0.34262     0.55298      1.1363
       :           :           :
       :           :           :

Retrieve the last 5 values of the table variable Var1.

tt(end-5:end,'Var1')
ans =

  6×1 tall table

      Var1  
    ________

       1.769
      1.5106
     0.16401
    -0.28276
      1.1522
     -1.1465

Retrieve every 100th row from the tall table.

tt(1:100:end,:)
ans =

  10×3 tall table

      Var1         Var2         Var3  
    _________    _________    ________

      0.53767       0.6737     0.29617
      0.84038    -0.041663    -0.52093
      0.18323       1.3419    0.052993
     0.079934     -0.40492     -1.6163
      0.26965      -1.5144     0.98399
    -0.079893      -1.6848    -0.91182
      0.47586      -2.1746      1.1754
       1.9085     -0.79383     0.18343
        :            :           :
        :            :           :

Extract Tall Table Variables

The variables in a tall table or tall timetable are each tall arrays of different underlying data types. Standard indexing methods of tables and timetables also apply to tall tables and tall timetables, including the use of timerange, withtol, and vartype.

For example, index a tall table using dot notation T.VariableName to retrieve a single variable of data as a tall array.

tt.Var1
ans =

  1,000×1 tall double column vector

    0.5377
    1.8339
   -2.2588
    0.8622
    0.3188
   -1.3077
   -0.4336
    0.3426
      :
      :

Use tab completion to look up the variables in a table if you cannot remember a precise variable name. For example, type tt. and then press Tab. A menu pops up:

Screenshot showing MATLAB command line, with "tt." entered and a tab completion menu of possible values.

You can also perform multiple levels of indexing. For example, extract the first 5 elements in the variable Var2. In this case you must use one of the supported forms of indexing for tall arrays in the parentheses.

tt.Var2(1:5)
ans =

  5×1 tall double column vector

    0.6737
   -0.6691
   -0.4003
   -0.6718
    0.5756

See Access Data in Tables or Select Times in Timetable for more indexing information.

Concatenation with Tall Arrays

In order to concatenate two or more tall arrays, as in [A1 A2 A3 …], each of the tall arrays must be derived from a single tall array and must not have been indexed differently in the first dimension. Indexing operations include functions such as vertcat, splitapply, sort, cell2mat, synchronize, retime, and so on.

For example, concatenate a few columns from tt to create a new tall matrix.

[tt.Var1 tt.Var2]
ans =

  1,000×2 tall double matrix

    0.5377    0.6737
    1.8339   -0.6691
   -2.2588   -0.4003
    0.8622   -0.6718
    0.3188    0.5756
   -1.3077   -0.7781
   -0.4336   -1.0636
    0.3426    0.5530
      :         :
      :         :

To combine tall arrays with different underlying datastores, it is recommended that you use write to write the arrays (or calculation results) to disk, and then create a single new datastore referencing those locations:

files = {'folder/path/to/file1','folder/path/to/file2'};
ds = datastore(files);

Assignment and Deletion with Tall Arrays

The same subscripting rules apply whether you use indexing to assign or delete elements from a tall array. Deletion is accomplished by assigning one or more elements to the empty matrix, [].

“( )” Assignment

You can assign elements into a tall array using the general syntax A(m,n,...) = B. The tall array A must exist and have a nonempty second dimension. The first subscript m must be either a colon : or a tall logical vector. With this syntax, B can be:

  • Scalar

  • A tall array derived from A(m,…) where m is the same subscript as above. For example, A(m,1:10).

  • An empty matrix, [] (for deletion)

“.” Assignment

For table indexing using the syntax A.Var1 = B, the array B must be a tall array with the appropriate number of rows. Typically, B is derived from existing data in the tall table. Var1 can be either a new or existing variable in the tall table.

You cannot assign tall arrays as variables in a regular table, even if the table is empty.

Extract Specified Number of Rows in Sorted Order

Sorting all of the data in a tall array can be an expensive calculation. Most often, only a subset of rows at the beginning or end of a tall array is required to answer questions like “What is the first row in this data by year?”

The topkrows function returns a specified number of rows in sorted order for this purpose. For example, use topkrows to extract the top 12 rows sorted in descending order by the second column.

t_top12 = gather(topkrows(tt,12,2))
Evaluating tall expression using the Local MATLAB Session:
Evaluation completed in 0.067 sec

t_top12 =

  12×3 table

      Var1       Var2       Var3  
    ________    ______    ________

     -1.0322    3.5699     -1.4689
      1.3312    3.4075     0.17694
    -0.27097    3.1585     0.50127
     0.55095    2.9745       1.382
     0.45168    2.9491     -0.8215
     -1.7115    2.7526     -0.3384
    -0.21317    2.7485      1.9033
    -0.43021    2.7335     0.77616
    -0.59003    2.7304     0.67702
     0.47163    2.7292     0.92099
    -0.47615     2.683    -0.26113
     0.72689    2.5383    -0.57588

Summarize Tall Array Contents

The summary function returns useful information about each variable in a tall table or timetable, such as the minimum and maximum values of numeric variables, and the number of occurrences of each category for categorical variables.

For example, create a tall table for the outages.csv data set and display the summary information. This data set contains numeric, datetime, and categorical variables.

fmts = {'%C' '%D' '%f' '%f' '%D' '%C'};
ds = tabularTextDatastore('outages.csv','TextscanFormats',fmts);
T = tall(ds);
summary(T)
Evaluating tall expression using the Local MATLAB Session:
- Pass 1 of 2: Completed in 0.16 sec
- Pass 2 of 2: Completed in 0.19 sec
Evaluation completed in 0.46 sec

Variables:

    Region: 1,468×1 categorical
        Values:

            MidWest        142   
            NorthEast      557   
            SouthEast      389   
            SouthWest       26   
            West           354   

    OutageTime: 1,468×1 datetime
        Values:

            Min    2002-02-01 12:18
            Max    2014-01-15 02:41

    Loss: 1,468×1 double
        Values:

            Min            0     
            Max            23418 
            NumMissing     604   

    Customers: 1,468×1 double
        Values:

            Min            0          
            Max            5.9689e+06 
            NumMissing     328        

    RestorationTime: 1,468×1 datetime
        Values:

            Min            2002-02-07 16:50 
            Max            2042-09-18 23:31 
            NumMissing     29               

    Cause: 1,468×1 categorical
        Values:

            attack                294  
            earthquake              2  
            energy emergency      188  
            equipment fault       156  
            fire                   25  
            severe storm          338  
            thunder storm         201  
            unknown                24  
            wind                   95  
            winter storm          145

Return Subset of Calculation Results

Many of the examples on this page use gather to evaluate expressions and bring the results into memory. However, in these examples it is also trivial that the results fit in memory, since only a few rows are indexed at a time.

In cases where you are unsure if the result of an expression will fit in memory, it is recommended that you use gather(head(X)) or gather(tail(X)). These commands still evaluate all of the queued calculations, but return only a small amount of the result that is guaranteed to fit in memory.

If you are certain that the result of a calculation will not fit in memory, use write to evaluate the tall array and write the results to disk instead.

See Also

| | | | |

Related Topics