Main Content

attentionLayer

Dot-product attention layer

Since R2024a

expand all in page

    Description

    A dot-product attention layer focuses on parts of the input using weighted multiplication operations.

    Creation

    Syntax

    layer = attentionLayer(numHeads)
    layer = attentionLayer(numHeads,Name=Value)

    Description

    layer = attentionLayer(numHeads) creates a dot-product attention layer and sets the NumHeads property.

    example

    layer = attentionLayer(numHeads,Name=Value) also sets the Scale, HasPaddingMaskInput, HasScoresOutput, AttentionMask, DropoutProbability, and Name properties using one or more name-value arguments.

    Properties

    expand all

    Attention

    Number of heads, specified as a positive integer.

    Each head performs a separate linear transformation of the input and computes attention weights independently. The layer uses these attention weights to compute a weighted sum of the input representations, generating a context vector. Increasing the number of heads lets the model capture different types of dependencies and attend to different parts of the input simultaneously. Reducing the number of heads can lower the computational cost of the layer.

    The value of NumHeads must evenly divide the size of the channel dimension of the input queries, keys, and values.

    Data Types: single | double | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64

    Multiplicative factor for scaling dot product of queries and keys, specified as one of these values:

    • "auto" — Multiply the dot product by 1/sqrt(D), where D is the number of channels of the keys divided by NumHeads.

    • Numeric scalar — Multiply the dot product by the specified scalar.

    Data Types: single | double | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64 | char | string | cell

    Flag indicating whether the layer has an input that represents the padding mask, specified as 0 (false) or 1 (true).

    If the HasPaddingMaskInput property is 0 (false), then the layer has three inputs with the names "query", "key", and "value", which correspond to the input queries, keys, and values, respectively. In this case, the layer treats all elements as data.

    If the HasPaddingMaskInput property is 1 (true), then the layer has an additional input with the name "mask", which corresponds to the padding mask. In this case, the padding mask is an array of ones and zeros. The layer uses or ignores elements of the queries, keys, and values when the corresponding element in the mask is one or zero, respectively.

    The format of the padding mask must match that of the input keys. The size of the "S" (spatial), "T" (time), and "B" (batch) dimensions of the padding mask must match the size of the corresponding dimensions in the keys and values.

    The padding mask can have any number of channels. The software uses only the values in the first channel to indicate padding values.

    Flag indicating whether the layer has an output that represents the scores (also known as the attention weights), specified as 0 (false) or 1 (true).

    If the HasScoresOutput property is 0 (false), then the layer has one output with the name "out", which corresponds to the output data.

    If the HasScoresOutput property is 1 (true), then the layer has two inputs with the names "out" and "scores", which correspond to the output data and the attention scores, respectively.

    Attention mask indicating which elements to include when applying the attention operation, specified as one of these values:

    • "none" — Do not prevent attention to elements with respect to their positions. If AttentionMask is "none", then the software prevents attention using only the padding mask.

    • "causal" — Prevent elements in position m in the "S" (spatial) or "T" (time) dimension of the input queries from providing attention to the elements in positions n, where n is greater than m in the corresponding dimension of the input keys and values. Use this option for auto-regressive models.

    • Logical or numeric array — Prevent attention to elements of the input keys and values when the corresponding element in the specified array is 0. The specified array must be an Nk-by-Nq matrix or a Nk-by-Nq-by-numObservations array, Nk is the size of the "S" (spatial) or "T" (time) dimension of the input keys, Nq is the size of the corresponding dimension of the input queries, and numObservations is the size of the "B" dimension in the input queries.

    Data Types: single | double | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64 | logical | char | string

    Probability of dropping out attention scores, specified as a scalar in the range [0, 1).

    During training, the software randomly sets values in the attention scores to zero using the specified probability. These dropouts can encourage the model to learn more robust and generalizable representations by preventing it from relying too heavily on specific dependencies.

    Data Types: single | double | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64

    Layer

    Layer name, specified as a character vector or string scalar. For Layer array input, the trainnet and dlnetwork functions automatically assign names to layers with the name "".

    The AttentionLayer object stores this property as a character vector.

    Data Types: char | string

    Number of inputs to the layer, returned as 3 or 4.

    If the HasPaddingMaskInput property is 0 (false), then the layer has three inputs with the names "query", "key", and "value", which correspond to the input queries, keys, and values, respectively. In this case, the layer treats all elements as data.

    If the HasPaddingMaskInput property is 1 (true), then the layer has an additional input with the name "mask", which corresponds to the padding mask. In this case, the padding mask is an array of ones and zeros. The layer uses or ignores elements of the queries, keys, and values when the corresponding element in the mask is one or zero, respectively.

    The format of the padding mask must match that of the input keys. The size of the "S" (spatial), "T" (time), and "B" (batch) dimensions of the padding mask must match the size of the corresponding dimensions in the keys and values.

    The padding mask can have any number of channels. The software uses only the values in the first channel to indicate padding values.

    Data Types: double

    Input names of the layer, returned as a cell array of character vectors.

    If the HasPaddingMaskInput property is 0 (false), then the layer has three inputs with the names "query", "key", and "value", which correspond to the input queries, keys, and values, respectively. In this case, the layer treats all elements as data.

    If the HasPaddingMaskInput property is 1 (true), then the layer has an additional input with the name "mask", which corresponds to the padding mask. In this case, the padding mask is an array of ones and zeros. The layer uses or ignores elements of the queries, keys, and values when the corresponding element in the mask is one or zero, respectively.

    The format of the padding mask must match that of the input keys. The size of the "S" (spatial), "T" (time), and "B" (batch) dimensions of the padding mask must match the size of the corresponding dimensions in the keys and values.

    The padding mask can have any number of channels. The software uses only the values in the first channel to indicate padding values.

    The AttentionLayer object stores this property as a cell array of character vectors.

    This property is read-only.

    Number of outputs of the layer.

    If the HasScoresOutput property is 0 (false), then the layer has one output with the name "out", which corresponds to the output data.

    If the HasScoresOutput property is 1 (true), then the layer has two inputs with the names "out" and "scores", which correspond to the output data and the attention scores, respectively.

    Data Types: double

    This property is read-only.

    Output names of the layer.

    If the HasScoresOutput property is 0 (false), then the layer has one output with the name "out", which corresponds to the output data.

    If the HasScoresOutput property is 1 (true), then the layer has two inputs with the names "out" and "scores", which correspond to the output data and the attention scores, respectively.

    The AttentionLayer object stores this property as a cell array of character vectors.

    Examples

    collapse all

    Open Live Script

    Create a dot-product attention layer with 10 heads.

    layer = attentionLayer(10)
    layer = 
  AttentionLayer with properties:

                   Name: ''
              NumInputs: 3
             InputNames: {'query'  'key'  'value'}
               NumHeads: 10
                  Scale: 'auto'
          AttentionMask: 'none'
     DropoutProbability: 0
    HasPaddingMaskInput: 0
        HasScoresOutput: 0

   Learnable Parameters
    No properties.

   State Parameters
    No properties.

Use properties method to see a list of all properties.
    Open Live Script

    Create a simple neural network with cross-attention.

    numChannels = 256;
numHeads = 8;

net = dlnetwork;

layers = [
    sequenceInputLayer(1,Name="query")
    fullyConnectedLayer(numChannels)
    attentionLayer(numHeads,Name="attention")
    fullyConnectedLayer(numChannels,Name="fc-out")];

net = addLayers(net,layers);

layers = [
    sequenceInputLayer(1, Name="key-value")
    fullyConnectedLayer(numChannels,Name="fc-key")];

net = addLayers(net,layers);
net = connectLayers(net,"fc-key","attention/key");

net = addLayers(net, fullyConnectedLayer(numChannels,Name="fc-value"));
net = connectLayers(net,"key-value","fc-value");
net = connectLayers(net,"fc-value","attention/value");

    View the network in a plot.

    figure
plot(net)

    Figure contains an axes object. The axes object contains an object of type graphplot.

    Algorithms

    expand all

    References

    [1] Vaswani, Ashish, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. "Attention is all you need." In Advances in Neural Information Processing Systems, Vol. 30. Curran Associates, Inc., 2017. https://papers.nips.cc/paper/7181-attention-is-all-you-need.

    Extended Capabilities

    Version History

    Introduced in R2024a

    See Also

    | | | | | |

    Topics