01 Chapter / 8
Define Tasks, Trajectories and Rewards
Which type, length and numerical conditions must a two-step arithmetic trajectory satisfy before it earns a reward?
Learning Objectives
- Establishment of two-step arithmetic tasks and strict validation protocols from limited sample space
- Evidence to distinguish between intermediate steps, final answers and real causal reasoning
- Manual baseline for data disaggregation, assault testing and even-handedness strategy completed
02 Chapter / 8
Derive the Full-trajectory Policy Gradient from Finite Sums
When a reward is not directly differentiable with respect to parameters, how do finite trajectory sums yield a policy gradient?
Learning Objectives
- Line-by-line extrascore Historic Equivalent with REINFORCE
- Triangulation checks using a parsing gradient, limited differentials and tracks
- Distinguishing training targets, sample gradients and one optimizer update
03 Chapter / 8
Within-group Comparisons and Valid Advantage Estimates
Why can subtracting a group’s mean reward change the expected gradient, and when is a leave-one-out baseline unbiased?
Learning Objectives
- Impartial conditions for demonstrating an independent baseline and retaining a baseline
- Accurates a change in the zoom bias and standard deviation with its own group average
- Explanatory lapses in zero groups, associated sampling and different task difficulties
04 Chapter / 8
Reference Policies, KL and Clipping: Distinct Objectives
KL, importance weights and clipping can all constrain updates. Which objective or estimator does each actually change?
Learning Objectives
- Scorre estimate for full transmission trajectory KL
- Hand-counting probabilities, weights of importance and positive or negative advantages
- Clarify the border between this project on-policy realization and PPO, GRPO
05 Chapter / 8
Connect Complete Trajectories to a Real Transformer
How do two numerical actions map to causal positions in a full Transformer and define one trajectory probability?
Learning Objectives
- From input number, response position and causal matrix to complete path
- Reconciling 2768 parameter ledgers and 10-digit support to one.
- Achieve a uniform interface for sampling, re-scoring, listing and freezing references
06 Chapter / 8
A Real RL Update: Sampling, Verification and Backpropagation
How do 40 sampled trajectories become one full-parameter update through verification, leave-one-out advantages and rescoring?
Learning Objectives
- Completely track 40 steps to enter a parameter update
- Separation of supervisory gains, RL benefits, reference penalties from optimist effects
- Read evidence of thinning reward zero gradients, cropping and 17 volume updates
07 Chapter / 8
Report Training Gains, Test Regressions and Sampling Budgets Separately
When training answers improve but every held-out task fails, what can additional sampling and parameter learning each establish?
Learning Objectives
- Failed to read the same training upgrade and combination that freezes running
- IRD pass@k and distinguish between budget increases and parameters learning
- Check the conclusions with precise enumeration, independent sampling, formatting pressure and zero-satisfactory upper limit
08 Chapter / 8
Reproduce from a Clean Directory and Report Failures
How can we reproduce the experiment from a clean directory and report both training gains and failures to generalize?
Learning Objectives
- Rebuilding complete data, training, evaluation and format checks in the stand-alone project catalogue
- Establish a list of the same evidence as the source code, check points, indicators and tests
- Define course endpoints with limited costing and clear non-operational projects
Reasoning and Reinforcement Learning
Design verifiable tasks, rewards, and feedback to understand reasoning optimization.
Advanced · 8 lessons · Approx. 34 hours
Scope and Experimental Limits
Eight lessons connect strict task contracts, exact gradients, group-estimator bias, reference policies, complete-response likelihoods, full-network RL updates, fixed-budget evaluation, and independent evidence. The project includes exact standard-library enumeration and a real 2,768-parameter decoder: 40 SFT steps, 100 RL steps, and updates to all 17 tensors. Held-out intermediate-sum failures are retained. This is neither a natural-language reasoning LLM nor a complete PPO or GRPO reproduction.
Use the same two-step arithmetic trajectories to derive REINFORCE, leave-one-out group baselines, and KL divergence. Train a small Transformer and report training gains, compositional extrapolation failures, and format sensitivity separately.
View Learning Plan
Course Details
Course Contents
01 / Define Tasks, Trajectories and Rewards02 / Derive the Full-trajectory Policy Gradient from Finite Sums
03 / Within-group Comparisons and Valid Advantage Estimates
04 / Reference Policies, KL and Clipping: Distinct Objectives
05 / Connect Complete Trajectories to a Real Transformer
All 8 lessons ↗
View Learning Plan