doc2sequence

Convert documents to sequences for deep learning

collapse all in page

Syntax

sequences = doc2sequence(enc,documents)
sequences = doc2sequence(emb,documents)
sequences = doc2sequence(___,Name,Value)

Description

sequences = doc2sequence(enc,documents) returns a cell array of the numeric indices of the words in documents given by the word encoding enc. Each element of sequences is a vector of the indices of the words in the corresponding document.

example

sequences = doc2sequence(emb,documents) returns a cell array of the embedding vectors of the words in documents given by the word embedding emb. Each element of sequences is a matrix of the embedding vectors of the words in the corresponding document.

example

sequences = doc2sequence(___,Name,Value) specifies additional options using one or more name-value pair arguments.

example

Examples

collapse all

Open Live Script

Load the factory reports data and create a tokenizedDocument array.

filename = "factoryReports.csv";
data = readtable(filename,'TextType','string');
textData = data.Description;
documents = tokenizedDocument(textData);

Create a word encoding.

enc = wordEncoding(documents);

Convert the documents to sequences of word indices.

sequences = doc2sequence(enc,documents);

View the sizes of the first 10 sequences. Each sequence is a 1-by-S vector, where S is the number of word indices in the sequence. Because the sequences are padded, S is constant.

sequences(1:10)
ans=10×1 cell array
    {[           0 0 0 0 0 0 0 1 2 3 4 5 6 7 8 9 10]}
    {[  0 0 0 0 0 0 11 12 13 14 15 2 16 17 18 19 10]}
    {[    0 0 0 0 0 0 20 2 21 22 7 23 24 25 7 26 10]}
    {[        0 0 0 0 0 0 0 0 0 0 0 27 28 6 7 18 10]}
    {[        0 0 0 0 0 0 0 0 0 0 0 0 29 30 7 31 10]}
    {[    0 0 0 0 0 0 0 32 33 6 7 34 35 36 37 38 10]}
    {[      0 0 0 0 0 0 0 0 0 39 40 36 41 6 7 42 10]}
    {[    0 0 0 0 0 0 0 0 43 44 22 45 46 47 7 48 10]}
    {[      0 0 0 0 0 0 0 0 0 0 49 50 17 7 51 48 10]}
    {[0 0 0 0 52 8 53 36 54 55 56 57 58 59 22 60 10]}
Open Live Script

Convert an array of tokenized documents to sequences of word vectors using a pretrained word embedding.

Load a pretrained word embedding using the fastTextWordEmbedding function. This function requires Text Analytics Toolbox™ Model for fastText English 16 Billion Token Word Embedding support package. If this support package is not installed, then the function provides a download link.

emb = fastTextWordEmbedding;

Load the factory reports data and create a tokenizedDocument array.

filename = "factoryReports.csv";
data = readtable(filename,'TextType','string');
textData = data.Description;
documents = tokenizedDocument(textData);

Convert the documents to sequences of word vectors using doc2sequence. The doc2sequence function, by default, left-pads the sequences to have the same length. When converting large collections of documents using a high-dimensional word embedding, padding can require large amounts of memory. To prevent the function from padding the data, set the 'PaddingDirection' option to 'none'. Alternatively, you can control the amount of padding using the 'Length' option.

sequences = doc2sequence(emb,documents,'PaddingDirection','none');

View the sizes of the first 10 sequences. Each sequence is D-by-S matrix, where D is the embedding dimension, and S is the number of word vectors in the sequence.

sequences(1:10)
ans=10×1 cell array
    {300×10 single}
    {300×11 single}
    {300×11 single}
    {300×6  single}
    {300×5  single}
    {300×10 single}
    {300×8  single}
    {300×9  single}
    {300×7  single}
    {300×13 single}
Open Live Script

Convert a collection of documents to sequences of word vectors using a pretrained word embedding, and pad or truncate the sequences to a specified length.

Load a pretrained word embedding using fastTextWordEmbedding. This function requires Text Analytics Toolbox™ Model for fastText English 16 Billion Token Word Embedding support package. If this support package is not installed, then the function provides a download link.

emb = fastTextWordEmbedding;

Load the factory reports data and create a tokenizedDocument array.

filename = "factoryReports.csv";
data = readtable(filename,'TextType','string');
textData = data.Description;
documents = tokenizedDocument(textData);

Convert the documents to sequences of word vectors. Specify to left-pad or truncate the sequences to have length 100.

sequences = doc2sequence(emb,documents,'Length',100);

View the sizes of the first 10 sequences. Each sequence is D-by-S matrix, where D is the embedding dimension, and S is the number of word vectors in the sequence (the sequence length). Because the sequence length is specified, S is constant. 

sequences(1:10)
ans=10×1 cell array
    {300×100 single}
    {300×100 single}
    {300×100 single}
    {300×100 single}
    {300×100 single}
    {300×100 single}
    {300×100 single}
    {300×100 single}
    {300×100 single}
    {300×100 single}

Input Arguments

collapse all

Input word embedding, specified as a wordEmbedding object.

Input word encoding, specified as a wordEncoding object.

Input documents, specified as a tokenizedDocument array.

Name-Value Arguments

collapse all

Specify optional pairs of arguments as Name1=Value1,...,NameN=ValueN, where Name is the argument name and Value is the corresponding value. Name-value arguments must appear after other arguments, but the order of the pairs does not matter.

Before R2021a, use commas to separate each name and value, and enclose Name in quotes.

Example: 'Length','shortest' truncates the sequences to have the same length as the shortest sequence.

Unknown word behavior, specified as the comma-separated pair consisting of 'UnknownWord' and one of the following:

  • 'discard' – If a word is not in the input map, then discard it.

  • 'nan' – If a word is not in the input map, then return a NaN value.

Tip

If you are creating sequences for training a deep learning network with a word embedding, use 'discard'. Do not use sequences with NaN values, because doing so can propagate errors through the network.

Padding direction, specified as the comma-separated pair consisting of 'PaddingDirection' and one of the following:

  • 'left' – Pad sequences on the left.

  • 'right' – Pad sequences on the right.

  • 'none' – Do not pad sequences.

Tip

When converting large collections of data using a high-dimensional word embedding, padding can require large amounts of memory. To prevent the function from adding too much padding, set the 'PaddingDirection' option to 'none' or set 'Length' to a smaller value.

Padding value, specified as the comma-separated pair consisting of 'PaddingValue' and a numeric scalar. Do not pad sequences with NaN, because doing so can propagate errors through the network.

Data Types: single | double | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64

Sequence length, specified as the comma-separated pair consisting of 'Length' and one of the following:

  • 'longest' – Pad sequences to have the same length as the longest sequence.

  • 'shortest' – Truncate sequences to have the same length as the shortest sequence.

  • Positive integer – Pad or truncate sequences to have the specified length. The function truncates the sequences on the right.

Data Types: single | double | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64 | char | string

Output Arguments

collapse all

Output sequences, returned as a cell array.

For word embedding input, the ith element of sequences is a matrix of the word vectors corresponding to the ith input document.

For word encoding input, the ith element of sequences is a vector of the word encoding indices corresponding to the ith input document.

Tips

  • When converting large collections of data using a high-dimensional word embedding, padding can require large amounts of memory. To prevent the function from adding too much padding, set the 'PaddingDirection' option to 'none' or set 'Length' to a smaller value.

Version History

Introduced in R2018b

See Also

| | | | | | | | | | |

Topics