Transformer Attention: Q, K and V

What they ask

The first technical for an NLP or AI role is usually one team member and it opens on the architecture. Draw the transformer for me: what is in the encoder stack, what is in the decoder stack, and what is different between them. Then, block by block:

• Which attention is in which block. Self attention in the encoder, masked (causal) self attention in the first decoder block, cross attention in the second. Say why the mask exists: at training time the whole target sequence is present, and without the mask position t would read position t+1 and the model would learn nothing useful.
• What Query, Key and Value actually are. Three learned projections of the same input. The query asks "what am I looking for", the keys advertise "this is what I contain", the dot product of the two gives the weights, and the values are what gets mixed. I will ask why we divide by the square root of d_k and what happens to the softmax if we do not.
• In cross attention, where does each one come from. The decoder supplies the query from its own hidden state; the key and the value come from the encoder output. Candidates who have only read about self attention usually stall here.
• Multi head attention: why several heads instead of one wide one, and what the concatenation plus output projection is for.
• Pretraining objectives. Masked language modelling for encoder only models, next token prediction for decoder only, denoising or span corruption for encoder decoder. Then: which family would you pick for retrieval, for classification, for generation, and why.

What they look for

That you can move between the picture and the maths without hand waving. They are patient with a candidate who says "this part I have only read about" and hard on one who invents. Expect the interview to switch between Arabic and English mid sentence; answer in whichever you think faster in.