Attention relationship
An informal label here for an attention weight linking token representations during a model computation.
Illustrative conversation
- How does the model connect “it” to a function mentioned several lines earlier?
- Attention relationships let the later token weigh relevant earlier representations while the model processes the sequence.
Definition
In a Transformer, self-attention computes weighted relationships between token representations so information from one position can influence another. Different attention heads and layers can emphasize different patterns, helping the model connect references, syntax, instructions, and distant evidence. The projection parameters are learned, while attention weights depend on the input and are computed during inference and should not be confused with durable application memory.
Reviewed