01 Chapter / 8
Define the Corpus and Task Before the Model
How should we split and score the same documents so that held-out loss answers one clearly defined question?
Learning Objectives
- Write document, template groups, data division and training targets into language contracts that can be verified item by item
- Distinguishing document averages, token averages and false generalizations caused by leaks with complete numerical examples
- Recreate project data summaries from original bytes and group lists and reject cross-assembly repetitions
02 Chapter / 8
Bytes, Unicode and Prediction Units
How do a Chinese character, its UTF-8 bytes and model tokens differ, and why does encodability not guarantee valid generation?
Learning Objectives
- Revert UTF-8 bytes in bits and separate characters, bytes and token lengths
- Verify the encoded coverage of the fixed byte table for new characters while constructing an inverse to the creation that cannot be decoded
- Target numbers, even baselines and byte costs by project number and boundary agreement
03 Chapter / 8
From Byte-pair Statistics to Reversible BPE
When adjacent byte pairs have equal counts or overlap, how can BPE remain deterministic and reversible?
Learning Objectives
- Ordered BPE training according to established numbers, parallel and non-overlapping replacement rules
- Demonstrate the byte order of the merger and construct cases of error replacement or test participation
- Boundaries for shorter accounting sequences, increased vocabulary and token path probabilities
04 Chapter / 8
Count Each Target Once: Windows and Batches
After splitting a long document into windows, how do we score each character and end marker exactly once?
Learning Objectives
- Inputs from boundary sequences to target error and tail window index, demonstrating that valid targets are covered just once
- Distinguishing from the window context and future leaks, verifying right fill in causally non-variable
- Accounting for real batch target numbers, sample status and goal weighted assessment denominators
05 Chapter / 8
Decoder Gradients and Resource Accounting
How can decoder shapes and gradient records reveal “training” that updates only the output layer?
Learning Objectives
- Accounting for all parameters in the matrix shape of actual preLN, absolute position, GELU decoder
- Explain the gradient from cross-entropy along the output projection and residue chain, and detect pseudo-closed rings that only train the output layer
- Define the trainingable range of small CPU experiments using real parameters, gradients and state memory records
06 Chapter / 8
Training as a Recoverable State Transition
Why can the next update differ even after loading the same weights and seed?
Learning Objectives
- Implements a clear gradient, forwards, reverses, cropping and AdamW updates on a line-by-line basis, distinguishing between loss and budget indicators
- The first step of Adam W is to explain why the restoration of weight alone cannot restore the optimal trajectory.
- Save and verify models, optimizers, encoding, data, sampling and random state, and recreate the environment next
07 Chapter / 8
Evaluation and Generation Failures under a Frozen Protocol
When test loss improves but generated text is corrupted, which statistics and output failures should we check separately?
Learning Objectives
- Recreate token PPL from the document-by-document NLL with every original bytes of regular paths containing EOS
- The use of the unigram baseline identification indicator calibre and training statistics
- Record separately loss, generation number, UTF- 8 legitimacy and observable failure on the freeze model
08 Chapter / 8
From Small-scale Comparisons to Credible Budget Conclusions
When wider models or more updates change performance, how do we separate the effects of parameters, corpus coverage and total training budget?
Learning Objectives
- Design re-established twinning experiments, distinguishing between the quantity of parameters, unique language materials, cumulative training targets and calculation of budgets
- Recapitulate all 12 points of the four real tracks and explain the effective range of observations of width and updating.
- Fake scaling inferences from recognition of unknown loss thresholds, low sample preparation versus budget mismatch
Train a Language Model from Scratch
Connect data, training, and evaluation in a reproducible pipeline.
Intermediate · 8 lessons · Approx. 32 hours
Scope and Experimental Limits
Eight lessons connect corpus contracts, UTF-8, reversible BPE, windows, a fully trained decoder, AdamW, exact recovery, held-out evaluation, and controlled scale comparisons. Prior knowledge of Transformer mechanisms is assumed; the project includes a self-contained copy of their implementation. Worked numerical examples, standard-library checks, progressive exercises, and explanatory quizzes accompany every lesson. Results on a small synthetic corpus validate the engineering; they do not establish general language ability or large-model scaling laws.
Starting with original short texts and byte encoding, build a training-only BPE tokenizer, a complete decoder, recoverable training, and consistent evaluation. Eight lessons connect one offline CPU project through numerical derivations, fault counterexamples, and recorded results.
View Learning Plan
Course Details
View Learning Plan