OmniPulse | AI, tech and markets

Attention forms a weighted combination of information from different positions in a sequence. Transformer architectures use attention alongside other components to construct representations.

About 2 min to read · Take extra time for practice · AI-assisted learning notes · Expanded October 17, 2026

First, the key terms.

Query: A learned representation used to look for relevant information.
Key: A representation used to score a match.
Value: Information combined according to the resulting weights.

The idea, explained

Scaled dot-product attention can be written as softmax(QKᵀ / √dₖ)V. Queries and keys determine scores; softmax turns those scores into weights over values. Multi-head attention learns several such transformations. Position information and feed-forward layers are also important: attention alone is not a complete language model.

STEP 1

Compute relevance scores

Each token’s representation is transformed into queries, keys and values. Query-key comparisons produce scores. A normalization step creates weights; masking can prevent access to positions that should be unavailable.

STEP 2

Combine information

The weights determine how much each value contributes to an output representation. Multiple heads can learn different relations. Other components, including feed-forward transformations and position information, also contribute; attention is not the whole architecture.

Follow a worked example.

In a sentence with an ambiguous pronoun, attention can combine information from earlier tokens when representing the current token. This is a computation over learned vectors, not evidence that the system has a human interpretation of the sentence.

Walk it through, one step at a time

Use a toy set of two values, 10 and 20, with attention weights 0.25 and 0.75.
Their weighted sum is 0.25 × 10 + 0.75 × 20 = 17.5. Real values are vectors, so this combination happens coordinate by coordinate.
The weights explain this numerical mixing operation, not necessarily the final answer’s causal reasoning or factual reliability.

Where people get stuck

An attention weight should not automatically be treated as a faithful explanation of a model’s final decision.

Now try it yourself.

Pause before opening each answer. Explain your reasoning in your own words.

1. Check the core idea

What is the role of V in the attention equation?

Reveal the answer & reasoning

V contains the value vectors being combined. The query-key scores determine how much each value contributes.

2. Apply what you learned

If a token receives a high attention weight, does that prove it caused the final answer?