AI Model Internals · Training Workshop

Transformers, from the inside

Tokens · Attention · Inference — a hands-on field guide, taught by one of the people who wrote the paper.

An engineering training for ML engineers & curious builders · ~90 min

html-ppt skill · transformer trainingtokyo-night

Agenda · 14 slides

What we'll cover

01 · Tokens

What the model actually reads & the cost of text.

02 · Embeddings & Attention

Vectors, Q/K/V, and why "attention is all you need".

03 · Architecture

Encoder & decoder blocks, multi-head self-attention.

04 · Inference

Autoregressive generation, KV cache, batching, latency.

Module 01 · Tokens

The model doesn't read words —it reads tokens

Tokenization of text into token IDs

  • Token = a unit of text

    Roughly 3–4 characters for English. "transformers" ≈ trans|form|ers. A sub-word, not always a whole word.

  • BPE tokenizer

    Byte-Pair Encoding merges the most common character pairs into vocabulary entries. Bigger vocab → fewer tokens per word.

  • Why you care

    Billing & context are measured in tokens, not words.

Module 02 · Representations

Tokens become vectors

Embedding matrix

Each token ID looks up a row in a big matrix → a vector in d_model dims (e.g. 4096).

Meaning as geometry

king − man + woman ≈ queen. Directions in vector space encode semantics.

Positional encoding

Attention is order-agnostic, so we add sine-wave position signals so the model knows word order.

No hard recurrence or convolution — just lookup tables and position math. That's part of why it parallelizes so well.

Module 02 · The core idea

Attention = "how much should I look at everyone else?"

Self-attention with tokens connected by weighted lines

  • Q · K · V

    Query asks a question, Key says "what I am", Value is "my content". Similarity(Q,K) → weight for each Value.

  • Softmax of dot products

    Weights are normalized so each position pays attention that sums to 1.

  • Self-attention

    Every token weighs every other token — including itself — in a single forward pass.

Module 02 · Scaling attention

Multi-head attention

h heads in parallel

Split the d_model into h subspaces (e.g. 8–32). Each head attends to a different relationship — syntax, coreference, proximity, subject–object.

Concat & project

All heads' outputs are concatenated and linearly projected back to d_model.

Think of it as several "attention experts" each monitoring a different type of relationship, then voting together.

Module 03 · Architecture

Encoder–Decoder stack

Transformer encoder-decoder architecture diagram

  • Encoder

    Bidirectional self-attention — reads the whole input at once. e.g. BERT-style encoders.

  • Decoder

    Masked self-attention (can't see the future) + cross-attention into the encoder. e.g. GPT-style decoders.

  • Block = attention + MLP

    Each block: multi-head attention → add & norm → feed-forward → add & norm. Stack N of them.

Module 04 · Inference

Generation is autoregressive

Autoregressive token-by-token generation with KV cache

  • One token at a time

    The model predicts the next token, appends it, and repeats — a loop until <EOS>.

  • Time to first token

    The first token (prefill) is the slow part. After that, streaming feels fast.

  • It's a loop

    Output length = number of forward passes. That's why "long outputs are slow."

Module 04 · Faster inference

KV cache — why it gets faster

The problem

Each step recomputes attention over ALL previous tokens. Naive: O(n²) work total.

The fix

Keys & Values from past steps are cached. Only the new Query needs computing → roughly O(n) per step.

Memory cost: KV cache grows with sequence length × layers × heads × head-dim — a big chunk of a model's VRAM during long chats.

Module 04 · Throughput

Batching = many requests, one pass

Batch inference

Group N requests and run them as one matrix multiply → far higher GPU utilization.

The tradeoff

More throughput per token, but every request waits for the slowest in the batch → higher latency.

Continuous batching

Modern servers evict finished sequences and slot new ones in mid-step — key to fast chat.

Module 04 · Memory & speed

Quantization: fewer bits, faster

FP16 baseline

A 7B-parameter model in FP16 ≈ 14 GB of weights.

4-bit (INT4)

Weights dropped to 4 bits ≈ 3.5 GB — ~4× smaller, runs on a single consumer GPU, small quality hit.

Post-training quantization rounds weights to fewer bits. Bigger speed & memory win for a small accuracy cost.

Module 04 · Advanced speed

Speculative decoding: guess, then verify

Draft

A small fast model guesses the next K tokens in one shot.

Verify

The big model checks all K at once in a single parallel pass.

Accept / reject

Accepted guesses are free; on a miss, resample. Net: ~2–3× decode speed-up.

Recap

The whole chain, one slide

Text → inputTokenizer → token IDs → embeddings + position
UnderstandingStacked blocks: multi-head self-attention + MLP + add & norm
OutputAutoregressive next-token loop until <EOS>
SpeedKV cache (O(n) per step) + batching (throughput) + quantization (memory) + speculative (latency)
The mantraAttention is all you need.

Wrap-up · Hands-on challenge

Go count some tokens.

Homework: pick a model, measure its KV cache for a 4K-token conversation, and rough out its FP16 vs INT4 memory — then bring the numbers and we'll compare.

← → navigate · S presenter · T themes · F fullscreen