DPO: From Reward Substitution to Response Log-ratios

Task, Prerequisites, and Derivation Scope

This chapter links the best strategies of chapters IV, BT, and V, KL, to a direct preference for the optimization of the DPO. Readers should be able to recreate the poor sigmoid and the index tilting that is being supported under the reference. We will not describe “No explicit reward network” as without statistical assumptions; the DPO still uses preferred models, fixed references and a set of comparisons. The project has two deliverables: the eight parameter strategy provides a poor score and a limited margin across parameters; and the complete microdecoder uses the true prompt/response path probabilities training all model parameters. Both shared a definition of loss, but the model was different from the numeric evaluation space.

The reference strategy is q, the strategy is pi, $\beta$ Strictly correct, adopting chapter V $\tau$ = $\beta$ The answer-by-answer reward can be written by the best strategy formula. $\beta$ Multiply $\log(\pi^*/q)$ Plus. $\beta$ $\log Z(x)$. When the same x compares w with l, public $\beta$ $\log Z$ It is a variable relationship between optimal distribution and corresponding incentives; replaces it with a limited parameter strategy. $\pi^*$ Post-optimization preferences appear to be modelling and estimation steps, and it cannot be said that the intermediate parameters of arbitrary training have been best served by the original incentive targets.

Substituting BT and Defining a Computable Loss

The BT wins with the sigmoid reward. $\beta$ By “winner's logpi minus logq” and then “loser’s logpi minus logq.” The negative probability of each observation is softplus(−m). On average, for a fixed comparison set, it is DPO losses. Here q must score with Pi on the same tip, the same answer text, the same tokenizer and the boundary rules; there is no common event meaning to reduce logprob in two models in different formats. Reference is not another random string, nor is there a global constant.

If $\log\pi_w$ and $\log\pi_l$ For intermediate variables, the loss to the former is $\beta$ Multiply (sigmoid(m) minus one), for which the opposite number is given; reference entry does not participate in guidance. $\theta$. The weight of a low or misdirected symbol is already high and low, which is a dynamic coefficient that logistic seems to give itself; it cannot be replaced with the words " Forever add the probability of Winner to the fixed amount". When all pairs have common miniature parameters, the gradient contribution needs to be added, not the optimal response that they do not affect each other separately.

Example 1: A Policy Equal to the Reference Still Has a Learning Signal

Makes the policy starting point to be fully copied, and the relative log ratio is zero, m=0, loss log2, while sigmoid(m) is reduced by one in two if. $\beta$ = 0.5 The loss of the Winner logprob guide is - 0.25 For Loser is + 0.25. The loss is not zero and the gradient is not zero; the DPO changes the relative preference from the reference starting point. “The training won't move.” Mixes the function values with the lead numbers.

In a limited linear strategy, the logpi difference for the same x is equal to the logit difference because softmax public unicoded items are consumed. Take x=2, Winner is one, Loser is zero, and the gradient updates only the bias and slope of these two actions: Winner [−0.25,−0.5], Loser [0.25,0.5], and the rest zero. The learning rate is 0.2 hours when Winner parameters increase [0.05,0.1], Loser decreases the same amount. The independent inspection is the center of eight parameters, and the independent check is the new margin is positive and loss drops. A zero gradient for some of the parameters is the result of the formula here, not the break chain.

Complete decoder cannot directly replicate the simplification of “Only two output lines”. Different answers have different contexts after the first token, and subsequent softmax reclassifications are no longer shared; the gradients sought by logprob go through embedding, attention, location, forward feed and output head. The project sequences provide two complete answers, one for response+EOS and one for ZXBXZ, and then for automatic microdistribution of loss, checking all seventeen parameters with limited gradients and actual updates. This realization will be carried over to API under Chapter VII.

Example 2: Better Relative Preference Does Not Guarantee Higher Winner Probability

Sets the reference for Winner and Loser to 0.4 and 0. 2, respectively. The current strategy is 0.2 and 0. 02, with the rest of the quality distributed to other answers. Winner compares the reference rate to 0. 5, while Loser is 0. 1, with the difference log ratio to log5; $\beta$ = about 0 804719 at 0.5, BT predicts a preference of about 0 690983, loss about 0. 369640. Compared to the reference log2, DPO improved, but the absolute probability for Winner decreased from 0.4 to 0 2.

This example is not an anomaly: the DPO data bind Winner to the reference correction of Loser, without fixing the quality of all other answers separately. If Loser falls faster, margins can improve, even if Winner drops. The independent check is to print two absolute probability, two log comparison, and eventually margin, one that declares all desired behaviour to be increased by looking only at the square. In open text, failure to compare answers may absorb probabilities makes loss preferable to independent generation evaluation necessary to be preserved together.

Beta and Optimization: Why Monotonicity Is Not Guaranteed

$\beta$ Connecting the incentive scale to the price of deviation from reference in the deduction; it also zooms in margin and gradient in the realization of loss. $\beta$ More robust KL costs, more recent strategy references; but practical training is limited step-to-step data, changing $\beta$ At the same time, it changes the gradient scale and saturation speed. KL is not always measured. $\beta$ Strictly single. Items are limited reference columns $\beta$ Validation results for 0.1, 0.5, 2 do not use small grids as a rule. Sequence layout is fixed $\beta$ 0.1, don't test it.

$\beta$ At zero in margin and zero in the strategy gradient; therefore, implementation should be rejected $\beta$, cannot be understood as “Zero.” $\beta$ Zero automatically tracks the reward indefinitely. The real “NoneKLIncentive maximizes” limit is not a direct zero-in-the-code factor for training dynamics with the DPO parameterization. Another extreme is $\beta$ Largely, a small amount of logs saturated the sigmoid over changes and were more sensitive to floating point stability and learning rates; the project used softplus to avoid direct calculation of negative logsigmoids.

Response Probability Must Cover the Complete Event

In response to self-return, logpi(y|x) is equal to the sum of each token in response and EOS log probability, and the hint totoken is not counted as a condition. The length of Winner and Loser are still aggregated because they are the probability of full string events. If the token average is changed, the score will be redefined to no longer the same full answer probability as the original DPO derived; this variant can be studied, but the name must be changed and the target stated, and the exact theoretical conclusion cannot continue to be quoted after the length has been removed.

The total logprob naturally changes in length, but the correction of reference offsets the long effects common to some of the two models and does not guarantee that all preferred data length deviations will be eliminated. For example, the SFT model contains long answers, the DPO prefers to be short, and EOS behavior will be influenced by a direct gradient. The sequence items strictly retain EOS, first answer totoken and different lengths mask, and will report separately on the correctness rate for each NLL answer; three are not the same target and will not be in conflict with a “Combined score”.

The limited reference strategy is fully standardized in four actions, allowing the full distribution of KL to be accurately calculated; decoder can calculate a valid probability for a given string path with a limited length, but it cannot list all the text. In theory, an open path at the end of the budget generation can be registered as abort result, closing the total quality of the limited tree creation probability; this project does not seek peace with this huge tree. The re-entry of the four fixed candidates is used only for candidate diagnostics, and cannot claim to have measured the entire text KL or satisfy some global KL constraint.

The Fixed Reference Is Part of the Objective

Freezing the reference requires three things: the number snapshots are not changed, the reference gradient maps are not built forward, and the model is in a determined evaluation mode. No_grad alone does not prevent its parameter object from being updated elsewhere; there is no event freeze; only copying the variable name may still point to the same variable list or module. The project limited strategy is read in no_grad, with all reference parameters since the serial version of Deepcopy readquires_grad for False and eval, while the DPO is reading in n_grad and comparing the complete state before and after training.

If each step of reference follows the current strategy, even if every time detach, margin returns zero, the log loss will remain log2, but the gradient may still drive the strategy, because only the guides for the reference branch are blocked; if both branches are linked to the same calculation chart, the gradient may be zero. Both errors can be misjudged by Loss alone. Independent checks should also assert object identity, parameters freeze, and reference to Hash and strategy updates, rather than just checking whether a single mark is limited.

# response_scores
def response_scores(logits,targets,mask):
    raise NotImplementedError('dpo')

# dpo_loss
def dpo_loss(policy_w,policy_l,reference_w,reference_l,beta):
    raise NotImplementedError('dpo')

# response_scores
def response_scores(logits,targets,mask):
 if logits.ndim!=3 or targets.shape!=logits.shape[:2] or mask.shape!=targets.shape:raise ValueError('shape mismatch')
 if not torch.isfinite(logits).all():raise ValueError('finite logits required')
 safe=targets.masked_fill(~mask,0);return (logits.log_softmax(-1).gather(-1,safe[...,None]).squeeze(-1)*mask).sum(-1)

# dpo_loss
def dpo_loss(policy_w,policy_l,reference_w,reference_l,beta):
 if not beta>0:raise ValueError('positive beta')
 margin=beta*((policy_w-policy_l)-(reference_w.detach()-reference_l.detach()));return torch.nn.functional.softplus(-margin).mean()

This Lesson's Project Contribution

This chapter delivers DPO losses, full gradients and two calculations that reveal misunderstandings. $\beta$ 0.5, learning rate 0.2, with a fixed SFT snapshot as reference; the sequence version uses 60 steps SFT and 40 steps DPO. $\beta$ 0.1 calculates the real answer path with a full parameter decoder. The two budgets and models are different and cannot mix their numbers into the same performance curves. The next chapter links mask, logprob, freeze and state preservation to a real implementable training project and retains all format migration failures.

The correction of the full answer ratio cannot be omitted. $\log\pi_w$ Less $\log\pi_l$ Again, sigmoid is optimized by another preferred classification target; it does not maintain a relative fixed reference incentive variable. Even if the training of Pair has increased its accuracy, it cannot be interpreted in the same way. The project also saves two points for policy and reference, and the test strategy is equal to zero for all margins at the time of reference, and detects misuse, inconsistencies or answers mask.

The theoretical preference model assumes that the observation success rate is given by a sigmoid with a bad reward, and that the DPO inherits the same mismatch if the real data contains loops or labelling rules that are missing from the context. Saves the obvious BT for not eliminating the BT's representational hypothesis. Limited data may also be completely distinguishable in parameter space, and continued optimization will increase the margin, reduce its losses, without ensuring that other response quality is controlled.

The only reference for guidance is that of a constant. $\beta$ It's still a fixed hyper-parameter, and it's not inverted. $\beta$, new targets are needed, otherwise models may simply reduce training losses by changing scales instead of changing desired behaviour. $\beta$ and steps are not automatically reconciled in training. $\beta$ It is not a contradiction, but two separate experiments; their absolute losas values cannot be directly horizontally superior.

The average offline pair also has sample weights. Winner of a tip compares three times with three Loser, not the same feedback from three independent users; these comparisons share Winner and context and are statistically relevant. The current training uses 18 pair rights as an empirical objective, and tests only report this set of designated relationships, not as an accurate confidence range for independent samples. To estimate overall human preferences, additional sampling and labelling models are required, and cannot be replaced by a synthesizing rule in this course.

Reverse transmission to EOS is particularly important: when short answers are better than long answers, the probability of lifting the end may be the primary path for a model to satisfy preference. If EOS is missing, the model claims to rate the full answer, the model may learn to start correctly but not stop; if all EOS are deleted as padding, both DPO and SFT will miss the end event. Chapter VII validates the EOS item by item at the level of the effective target member, not just whether the string prefix is correct.

import math
softplus=lambda t:max(t,0)+math.log1p(math.exp(-abs(t)))
sigmoid=lambda t:1/(1+math.exp(-t))
beta=.5;q_w,q_l=.4,.2;p_w,p_l=.2,.02
m=beta*((math.log(p_w)-math.log(q_w))-(math.log(p_l)-math.log(q_l)))
assert p_w<q_w and m>0 and softplus(-m)<math.log(2)
print('winner fell',q_w,'to',p_w,'margin',m,'preference probability',sigmoid(m),'loss',softplus(-m))
grad_w=-beta/2;grad_l=beta/2
assert grad_w<0 and grad_l>0
print('at reference loss',math.log(2),'winner/loser logprob gradients',grad_w,grad_l)