LEARNING PLAN/ Progress
Reasoning and Reinforcement Learning
Use the same two-step arithmetic trajectories to derive REINFORCE, leave-one-out group baselines, and KL divergence. Train a small Transformer and report training gains, compositional extrapolation failures, and format sensitivity separately.
Start with a question, derive the mechanism, then test your implementation with new inputs.
Times are author estimates that include derivation and practice. They have not been measured with learners; work through the sections at your own pace.
WHAT YOU WILL BUILD
What will you build to demonstrate understanding?
- Definition of trajectory support, rigorous authentication and diagnostic data disaggregation
- Rebuild score, leave a baseline and KL gradient and check bias with an enumeration
- Two-step response to genuine shared parameters Transformer and actual RL update
- Distinguishing learning gains from the same frozen weights, searching budgets, extrapolation failures and failure of formats
What should you know before starting?
- Operation of Python functions, lists and local files
- Reads limited summation, logarithm and basic orientation; condition probability, score and group estimation are derived in the body
- Completion of Transformer field operations and reverse transmission base; recommended training after reading large models
- Transformer Chapter 4: Causation and locationThis indicates the visible range of each action after the given prefix.
- Transformer Chapter 6: Reverse and UpdateCan read the reverse message and parameter update for PyTorch losses.
- Transformer Chapter 7: Complete DecoderCan check decoder parameters and position-by-position output; a true trajectory model for this course.
Use the prerequisites to choose a starting point. You can read freely and return to earlier lessons whenever you need them.
THE LEARNING ROUTE
A clear path, one step at a time.
Saved in this browser only. Marking a lesson complete does not mean the project has passed verification.
Tasks and Estimators
Fixed trajectories and incentives, strategic gradients and group baseline conditions.
Define Tasks, Trajectories and Rewards
Which type, length and numerical conditions must a two-step arithmetic trajectory satisfy before it earns a reward?
Derive the Full-trajectory Policy Gradient from Finite Sums
When a reward is not directly differentiable with respect to parameters, how do finite trajectory sums yield a policy gradient?
Within-group Comparisons and Valid Advantage Estimates
Why can subtracting a group’s mean reward change the expected gradient, and when is a leave-one-out baseline unbiased?
Objective and update
Distinguishing KL from cropping and connecting the sampling and full parameter update of the decoder.
Reference Policies, KL and Clipping: Distinct Objectives
KL, importance weights and clipping can all constrain updates. Which objective or estimator does each actually change?
Connect Complete Trajectories to a Real Transformer
How do two numerical actions map to causal positions in a full Transformer and define one trajectory probability?
A Real RL Update: Sampling, Verification and Backpropagation
How do 40 sampled trajectories become one full-parameter update through verification, leave-one-out advantages and rescoring?
Evaluation and delivery
Separate training, setting aside, searching budgets, running back and retaining real failures.
Report Training Gains, Test Regressions and Sampling Budgets Separately
When training answers improve but every held-out task fails, what can additional sampling and parameter learning each establish?
Reproduce from a Clean Directory and Report Failures
How can we reproduce the experiment from a clean directory and report both training gains and failures to generalize?
BUILD / TEST / TRANSFER
2-step arithmetic trajectory enhancement learning lab
Accurate reference to the standard library + local complete Transformer; sample, verify, retain a return, KL, update, listing and freeze evaluation is fully functional.
Runtime Requirements
Python 3.11+ and PyTorch 2.6.0. Single-threaded CPU execution; no GPU or API key is required.
Read the README after extracting the archive, then follow its environment and run instructions. Reference implementations and learning tasks are provided separately.
Project Checklist
This is a personal checklist, not an automatic grade. Keep your inputs, results, failures and reproduction commands.