LEARNING PLAN/ Progress
LLM Post-training
Connect SFT, preference data, Bradley–Terry models, KL divergence, and DPO using an enumerable reference and a complete miniature decoder. Eight lessons derive response probabilities and gradients, preserving independent evidence of preference gains, probability leakage, and format-transfer failures.
Start with a question, derive the mechanism, then test your implementation with new inputs.
Times are author estimates that include derivation and practice. They have not been measured with learners; work through the sections at your own pace.
WHAT YOU WILL BUILD
What will you build to demonstrate understanding?
- Different targets defined by demonstration and comparison, with accurate response-only masking
- Indication of BT gradient, KL Best Policy and DPO full response to losses
- Full parameter training 11472 parameters decoder and validation of fixed references with path score
- Post-definition training conclusions with stand-alone, absolute path quality and format failure
What should you know before starting?
- Can develop Python programs using limited probability, logarithmic, lead and chain rules
- Transformer field decoder, causality and global parameter reverse dissemination completed
- Can run the local PyTorch2. 6 CPU environment; theoretical reference code only needs a standard library
- Transformer Chapter 4: Causation and locationIt is possible to check the extent of the conditions for returning to answer.
- Transformer Chapter 6: Reverse and UpdateCan read chain guidance and complete parameters update.
- Transformer Chapter 7: Complete DecoderEnables the logits to be checked against the target position along the complete decoder; this is used for this course sequence.
Use the prerequisites to choose a starting point. You can read freely and return to earlier lessons whenever you need them.
THE LEARNING ROUTE
A clear path, one step at a time.
Saved in this browser only. Marking a lesson complete does not mean the project has passed verification.
Oversight and comparison
Distinguishing between demonstration and preference and constructing verifiable targets for limited response.
What Does Post-training Change?
A demonstration gives one response, while a preference compares two. What can each tell us, and what can each change?
SFT: From Response Likelihood to Full-parameter Gradients
After joining a prompt and response, how do we score only response tokens and EOS, then backpropagate through every parameter?
Preference Data: What Are We Comparing?
What reward relationships remain identifiable when preference data contains ties, duplicates or cycles?
Incentives and strategies
The incentive is poor, KL is the best strategy, and DPO is the best.
Bradley–Terry Reward Models and Identifiable Differences
If preferences constrain only reward differences, why must we handle the reward zero point and scale separately?
Derive the Optimal Policy under a KL Constraint
How do we derive the optimal policy from a reward-minus-KL objective, and why does the formula require care at zero reference probability?
DPO: From Reward Substitution to Response Log-ratios
Why can increasing the winner-to-loser probability ratio still reduce the winner’s absolute probability?
Sequence and evaluation
Received complete decoder and checked probabilistic outflow with a failure to migrate formatting.
Connect SFT and DPO to a Complete Decoder
When connecting a four-response derivation to a full decoder, how do we check masks, full-path scores and a frozen reference?
Evaluation: Preference Gains, Probability Leakage and Format Failures
If preference metrics improve while response likelihood or performance on new formats worsens, how do we report all of these results?
BUILD / TEST / TRANSFER
Preference mechanism and complete decoder double-realization test stand
A limited four-response reference provides a range between precise expectations and full parameters, the real decoder feed completes SFT60 steps with DPO40 steps, and freezes the loading of the evaluation and five format migration probes. Pack the source code, original data lists, 17 tests and all success/failure receipts.
Runtime Requirements
Python 3.12 and PyTorch 2.6.0 are recommended. A CPU is sufficient; see the project README for installation instructions.
Read the README after extracting the archive, then follow its environment and run instructions. Reference implementations and learning tasks are provided separately.
Project Checklist
This is a personal checklist, not an automatic grade. Keep your inputs, results, failures and reproduction commands.