01 Chapter / 8
What Does Post-training Change?
A demonstration gives one response, while a preference compares two. What can each tell us, and what can each change?
Learning Objectives
- Defining conditionality strategies, monitoring models, preference comparisons and independent assessments as different data objects
- Calculating correct rate quality, short quality and desired utility on four response tasks per item
- Post-training status and zoning compacts to build non-exaggeration model capabilities
02 Chapter / 8
SFT: From Response Likelihood to Full-parameter Gradients
After joining a prompt and response, how do we score only response tokens and EOS, then backpropagate through every parameter?
Learning Objectives
- Gradually decipher the p−y gradient for softmax cross-entropy and update all parameters along a sexual policy
- Answer to Dotoken with the correct location of the response-only misallocation mask, and keep the first answer to Token and EOS
- Distinguishing between oversight objectives by answer, by token and by model repeat frequency
03 Chapter / 8
Preference Data: What Are We Comparing?
What reward relationships remain identifiable when preference data contains ties, duplicates or cycles?
Learning Objectives
- Write comparison with winner/loser, draw and duplicate as checkable data contracts
- Estimated success rate by repeated comparison frequency numbers and incentive of boundary assumed with loop preferences against the location markers
- Distinguishing candidate overlays, labelling rules and stand-alone utility, constructing tests that reveal location or length deviations
04 Chapter / 8
Bradley–Terry Reward Models and Identifiable Differences
If preferences constrain only reward differences, why must we handle the reward zero point and scale separately?
Learning Objectives
- From Bradley-Terry's assumption, hessian seems stable, gradients and regularization.
- Manual correctness and shortness independent comparison, verification of the solver gradient
- Explanation of infinity-scale pressure, additional non-recognition and calibration limits that can be divided into preferences
05 Chapter / 8
Derive the Optimal Policy under a KL Constraint
How do we derive the optimal policy from a reward-minus-KL objective, and why does the formula require care at zero reference probability?
Learning Objectives
- Best strategy to reward KL reduction in gradual resolution of expectations on limited support reference distribution
- An independent global best of proof by KL non-negative, checking temperature and incentive scale connection
- Construct zero-reference support and reverse KL vice case, defining situations that are not directly covered by the formula
06 Chapter / 8
DPO: From Reward Substitution to Response Log-ratios
Why can increasing the winner-to-loser probability ratio still reduce the winner’s absolute probability?
Learning Objectives
- Reverse KL 's best strategy to reward and replace BT, and eliminate the hints from the equations.
- DPO gradient to distinguish four candidate linear eliminations from complete decoder inverse chain
- The manual reference is the reverse of the absolute probability decline in Winner, defining the beta and length to match
07 Chapter / 8
Connect SFT and DPO to a Complete Decoder
When connecting a four-response derivation to a full decoder, how do we check masks, full-path scores and a frozen reference?
Learning Objectives
- Rebuild the first answer/ EOS mask and full path scores by real battling interface with response_logps
- Account for 11472 parameter decoder and verify SFT/DPO full parameter gradient and freeze
- Run independent input assessment, and locate the difference between the re-allocation of candidate conditions and the absolute probability of answering them
08 Chapter / 8
Evaluation: Preference Gains, Probability Leakage and Format Failures
If preference metrics improve while response likelihood or performance on new formats worsens, how do we report all of these results?
Learning Objectives
- Simultaneous interpretation of SFT, hidden comparison of DPOs, limited desired utility and sequence of gready results
- Recalculation of target conflict with real receipts, distinguishing between candidate KL and full generation distribution diagnosis
- Failed to retain full format migration and undetected boundary on the freeze model, completed reviewable delivery
LLM Post-training
Start with tasks and failure cases to shape the behavior you need.
Advanced · 8 lessons · Approx. 32 hours
Scope and Experimental Limits
Begin with a finite conditional policy and derive supervised likelihood, comparison models, regularized rewards, KL-optimal distributions, and DPO gradients. Apply them to full-parameter, response-only SFT and complete-response DPO in a custom decoder. Numerical references, a CPU sequence project, progressive exercises, independent evaluation, and failure records are included. Original synthetic tasks do not represent commercial LLMs; candidate normalization and perfect greedy scores do not establish general alignment ability.
Connect SFT, preference data, Bradley–Terry models, KL divergence, and DPO using an enumerable reference and a complete miniature decoder. Eight lessons derive response probabilities and gradients, preserving independent evidence of preference gains, probability leakage, and format-transfer failures.
View Learning Plan
Course Details
View Learning Plan