LEARNING PLAN/ Intermediate
Train a Language Model from Scratch
Starting with original short texts and byte encoding, build a training-only BPE tokenizer, a complete decoder, recoverable training, and consistent evaluation. Eight lessons connect one offline CPU project through numerical derivations, fault counterexamples, and recorded results.
Start with a question, derive the mechanism, then test your implementation with new inputs.
Times are author estimates that include derivation and practice. They have not been measured with learners; work through the sections at your own pace.
WHAT YOU WILL BUILD
What will you build to demonstrate understanding?
- Freezing of language and coded identity by source group, detection of specific data leaks
- Hand-count UTF-8 and determine BPE, certify byte rounding and account for unit of expression
- Train self-written decoder from random parameters to re-establish environmental recovery tracks
- Unified boundary and window calibre assessment, failure to maintain full scale matching and generation
What should you know before starting?
- Writes Python functions, dictionary, slices and runs local items
- Knowledge of probabilities, logarithms, matrix shapes, guides and chain rules
- Transformer field attention, causality, forward feed and reverse transmission completed; course focus training system closed
- Transformer Chapter 4: Causation and locationIt explains why removal targets, cause mask and right fill do not leak the future.
- Transformer Chapter 5: Full Decoding BlockCould read every step along pre-LN, fragment and front feed.
- Transformer Chapter 6: Reverse and UpdateThe gradient can be explained from the cross-entropy chain along the output head, normalization and residue chain.
- Transformer Chapter 7: Complete DecoderThe parameters that can be distinguished from those held by the microcalculation, parameter registration and the optimizer.
Use the prerequisites to choose a starting point. You can read freely and return to earlier lessons whenever you need them.
THE LEARNING ROUTE
A clear path, one step at a time.
Saved in this browser only. Marking a lesson complete does not mean the project has passed verification.
Language and Encoding
Fixed data division, byte units and codes that are only compiled in the training set.
Define the Corpus and Task Before the Model
How should we split and score the same documents so that held-out loss answers one clearly defined question?
Bytes, Unicode and Prediction Units
How do a Chinese character, its UTF-8 bytes and model tokens differ, and why does encodability not guarantee valid generation?
From Byte-pair Statistics to Reversible BPE
When adjacent byte pairs have equal counts or overlap, how can BPE remain deterministic and reversible?
Window and Training
Connect effective targets, complete models and re-enable training.
Count Each Target Once: Windows and Batches
After splitting a long document into windows, how do we score each character and end marker exactly once?
Decoder Gradients and Resource Accounting
How can decoder shapes and gradient records reveal “training” that updates only the output layer?
Training as a Recoverable State Transition
Why can the next update differ even after loading the same weights and seed?
Evaluation and budget
The analysis leaves room for failure in generation, with a clearer picture of the scale changes under the budget.
Evaluation and Generation Failures under a Frozen Protocol
When test loss improves but generated text is corrupted, which statistics and output failures should we check separately?
From Small-scale Comparisons to Credible Budget Conclusions
When wider models or more updates change performance, how do we separate the effects of parameters, corpus coverage and total training budget?
BUILD / TEST / TRANSFER
Micro-language modeling stations from bytes to training receipts
Thirty-two original composite shorts have been used to independently complete the byte/BPE code, complete preLN decoder full parameters, state restoration, freeze assessment and four scale tracks. The compressed package contains data, model copies, thirteen tests, fixed check points and complete laboratory receipts.
Runtime Requirements
Python 3.12 and PyTorch 2.6.0 are recommended. A CPU is sufficient; see the project README for installation instructions.
Read the README after extracting the archive, then follow its environment and run instructions. Reference implementations and learning tasks are provided separately.
Project Checklist
This is a personal checklist, not an automatic grade. Keep your inputs, results, failures and reproduction commands.