LEARNING PLAN/ Intermediate
Transformer in Practice
Calculate attention by hand, then implement, train, and validate a complete character-level Transformer.
Start with a question, derive the mechanism, then test your implementation with new inputs.
Times are author estimates that include derivation and practice. They have not been measured with learners; work through the sections at your own pace.
WHAT YOU WILL BUILD
What will you build to demonstrate understanding?
- Deference conditions from mission definition, cross-entropy and attention gradient
- Full decoding model for achieving multi-heading, absolute position, pre-LN and GELU
- Complete full parameter training, effective token rating, check point exact recovery and cache generation
- Use numeric contrasts, malfunctions, digestion of the same version as independent document acceptance
What should you know before starting?
- Could create Python functions, lists, dictionaries and run local programs
- Acquainted with basic calculations of limited probability, vector and matrix; specific usage of the course
- Knows the meaning of one-dimensional guides; multiple inverse steps are derived from each of the instructions
Use the prerequisites to choose a starting point. You can read freely and return to earlier lessons whenever you need them.
THE LEARNING ROUTE
A clear path, one step at a time.
Saved in this browser only. Marking a lesson complete does not mean the project has passed verification.
Definitions and Expressions
Creates a common interface between predictive objects, probability, embedding and attention.
From Prediction Tasks to Data Contracts
Given “春风”, how do we build inputs and targets that predict the next character and the end of the text, rather than copy the current character?
From Probabilities to Vector Representations
How do real-valued scores become probabilities and a loss, and how do embedding gradients accumulate when a character appears more than once?
Work Through Attention Step by Step
How do two positions use Q, K and V to compute a readout and propagate errors back through all three projections?
Causality, Multiple Heads and Position
How does attention exclude future tokens while multiple heads and positions distinguish the same character in different contexts?
Construct and Learn
Connects to the complete decoder, leads to a gradient and saves a reversible training state.
From Residual Connections to a Complete Decoder Block
How do normalization, residual connections and a feed-forward network form a complete decoder block with checkable gradient paths?
Backpropagation and Optimization
How does a scalar loss produce gradients throughout the computation graph and become one AdamW update?
Implement a Trainable Decoder
How do embeddings, stacked blocks and an output head become a model, and how do we detect unregistered parameters or inconsistent targets?
Training, Data and Reproducible Recovery
After an interruption, how do we restore the model, optimizer, sampling and random state so that the next update is identical?
Evaluation and delivery
Comparative evaluation, generation and extension mechanisms to deliver the same version of projects.
Evaluation: What Has the Model Actually Learned?
When validation loss falls, how do we rule out test leakage, incorrect denominators and mismatched ablations?
Generation and Caching with the Same Model
How does the same model generate one token at a time, and how do we verify that cached and full-prefix computation agree?
Architecture and Cost: Toward Modern Implementations
When using RoPE, GQA or online softmax, which changes optimize the same function and which change the model itself?
Final Project: Deliver a Reproducible Transformer
Given only the project archive, how can someone independently reproduce training, recovery, evaluation and generation for the same build?
BUILD / TEST / TRANSFER
Runable Character Transformer Experimental Station
Starting with original documents and random weights, self-decoding models and complete training chains are achieved; evaluation, cache generation and location are completed with the same checkpoint and source code, data inventory, original results and failure reports are submitted.
Runtime Requirements
Python 3.12 and PyTorch 2.6.0 are recommended. A CPU is sufficient; see the project README for installation instructions.
Read the README after extracting the archive, then follow its environment and run instructions. Reference implementations and learning tasks are provided separately.
Project Checklist
This is a personal checklist, not an automatic grade. Keep your inputs, results, failures and reproduction commands.