01 Chapter / 12
From Prediction Tasks to Data Contracts
Given “春风”, how do we build inputs and targets that predict the next character and the end of the text, rather than copy the current character?
Learning Objectives
- Gradual self-regression from the condition probabilities definition containing EOS and distinguishing prefixes from full document probabilities
- Split and create a train-only Unicode sorting list by document, using UNK0/BOS1/EOS2 and separate target ignore values -100
- Construct moving input, target and valid tags according to the right-hand fill rule of the project, distinguishing between real EOS and filled roles
- Reassemble and export shapes with B, T, V, D and run independent contract checks
02 Chapter / 12
From Probabilities to Vector Representations
How do real-valued scores become probabilities and a loss, and how do embedding gradients accumulate when a character appears more than once?
Learning Objectives
- Average cross-americ americium and its p-octogenetic target gradient from stable softmax to appear to be active gradually token
- Probability, loss and orientation of three types of fractions and independent review using equal mobility and centre differentials
- Write the embedded checklist as a single heat matrix multiplier to calculate the gradient token to add on the same parameter line
- Loss of information on symmetrically aggregated reverse positioning order, distinguishing between characters embedded in the context of the next chapter
03 Chapter / 12
Work Through Attention Step by Step
How do two positions use Q, K and V to compute a readout and propagate errors back through all three projections?
Learning Objectives
- Recalculate two-dimensional attention and scalable hypothetical boundaries from the input of three separate projective-by-axis structures Q, K, V
- Gradually extrapolate all Q/K/V and common input gradients from AV, softmax
- Move side by side, comb the boundary and the full coordinates with limited differential independent inspection forward and backwards
- Distinguishing matching weights, reading content and predicting probabilities with the reverse of the same key and changing value
04 Chapter / 12
Causality, Multiple Heads and Position
How does attention exclude future tokens while multiple heads and positions distinguish the same character in different contexts?
Learning Objectives
- To induce ji and prove that right fill is in the future of valid query, save real EOS with target - 100 excluding filling loss
- Split multiple headings by index, hand-sized, read-and-balanced W_O output
- Embedded learning absolute position access, calculated the gradient to be shared by location and limited the scope of application of the no-location reverse
- Check location information leaks and position error with future disturbance, close mask contrast and whole/cache equivalent
05 Chapter / 12
From Residual Connections to a Complete Decoder Block
How do normalization, residual connections and a feed-forward network form a complete decoder block with checkable gradient paths?
Learning Objectives
- Gradual calculation of population differentials LayerNom and accurate GELU feed
- Write pre-LN forward and residual gradient for two branches
- Diffusion of whole machine parameters by shape
- Performed by checking the authentication block with algebra non-variable, numerical and structural
06 Chapter / 12
Backpropagation and Optimization
How does a scalar loss produce gradients throughout the computation graph and become one AdamW update?
Learning Objectives
- extrapolate softmax cross-entropy and head weight gradient from the calculation diagram
- Gradually extrapolate the gradient of LayerNorm input and emulation parameters
- Unvariable and Center Limited Disparities Inverted Error
- Explain AdamW's deviation correction, decoupling decay and cropping different positions
07 Chapter / 12
Implement a Trainable Decoder
How do embeddings, stacked blocks and an output head become a model, and how do we detect unregistered parameters or inconsistent targets?
Learning Objectives
- Embed words, location, multi-layered blocks, finalLN and head into an open interface
- Manual limit to verify complete logits and alignment losses
- Distinguishing the boundaries of microcalculation, module registration and optimizer holdings
- Run all parameter connections and fixed batch proposed acceptances and identify conflicting targets
08 Chapter / 12
Training, Data and Reproducible Recovery
After an interruption, how do we restore the model, optimizer, sampling and random state so that the next update is identical?
Learning Objectives
- Split and construct each character with a fixed document ID and a single score for EOS
- To extract right padding, ignore objectives and valid token weighted assessment
- Run training and eval/no_grad validation along the real API
- Save and test the complete restoration of models, optimizers, samplers and torch RNG
09 Chapter / 12
Evaluation: What Has the Model Actually Learned?
When validation loss falls, how do we rule out test leakage, incorrect denominators and mismatched ablations?
Learning Objectives
- Weighted loss and confusion from the probabilities of negative logarithmic characters
- Designing a clean test set for validation, decomposition and comparison
- Description of uncertainty in limited samples with document-level resampling
- Trace quality results to the same model, data and configuration
10 Chapter / 12
Generation and Caching with the Same Model
How does the same model generate one token at a time, and how do we verify that cached and full-prefix computation agree?
Learning Objectives
- Distribute greedy, temperature and top-k generation by conditions
- Explains the KV cache on a layer-by-line basis and the full prefix calculates the equivalent price
- Calculate cache memory and calculate complexity by word
- Errors detected in position deviation, window cropping and random state
11 Chapter / 12
Architecture and Cost: Toward Modern Implementations
When using RoPE, GQA or online softmax, which changes optimize the same function and which change the model itself?
Learning Objectives
- Calculate main project parameters, activations, KV and matrix multiplier different books
- Queued ROPE relative position points and GQA cache savings
- Distinguishing function differences between RMSNom, door control feed and original model
- Rebuild the same attention without saving the whole line with a block of online Softmax
12 Chapter / 12
Final Project: Deliver a Reproducible Transformer
Given only the project archive, how can someone independently reproduce training, recovery, evaluation and generation for the same build?
Learning Objectives
- Independent assembly of data, models, optimization, restoration, evaluation and generation complete chain
- Check correctness with a clear acceptance matrix, not just a demonstration.
- Report the benefits, failures and data boundaries of the reference operation
- Project packages with environment, Hash, original results and recovery orders
Transformer in Practice
Explore attention, position, and generation through a single prediction.
Intermediate · 12 lessons · Approx. 48 hours
Scope and Experimental Limits
Starting with next-token prediction, derive attention, causality, position, residual connections, and backpropagation. Twelve lessons build on the same character-level Transformer: implementation, full-parameter training, exact recovery, cached generation, and evaluation. Complete code, numerical checks, progressive assignments, and project validation make the limits of small-corpus experiments explicit.
Calculate attention by hand, then implement, train, and validate a complete character-level Transformer.
View Learning Plan
Course Details
View Learning Plan