Derive the Optimal Policy under a KL Constraint

Task and Prerequisites: Fix the Reward Before Optimizing the Distribution

Chapter IV, which combines comparative probabilities, now uses the real-digit reward r as a given input for the time being, and examines another question: how to change the strategy so that expectations are increased without being arbitrarily removed from the reference strategyq. Readers need to be limited in their search for, logarithm, guidance and binding extreme values; LaGrand’s specific operations will be done gradually, without assuming that the reader already knows the conclusions. We seek four candidate distributions under one fixed x and then apply them context by context. Real-parameter sharing strategies may not achieve all best distributions by x at the same time, and this expression limit must be separated from the solution at the distribution level.

Fixed limited action collection, reference probability $q_a$ All strict and one-for-one, reward. $r_a$ are limited. probability of selection $p_a$ Not negative, and one. Defines the KL as each item $p_a\log(p_a/q_a)$ and agreed to $p_a$ The target J is to expect a reduction in the reward. $\tau$ It's a lot of kl. $\tau$ Strictly correct. Here “Forward” refers to the course 's use of p-q' instead of calling the other side the same punishment. $\tau$; it is not a study rate or an alias to the softmax sampling temperature.

Deriving a Candidate Solution from First-Order Conditions

First on all. $p_a$ Find a place to stay inside the house. Multiply the constraints with one. $\lambda$ Add target, write down expectations of reduced reward $\tau$ KL plus $\lambda$ Multiply probability and minus one. $p_a$ Guidance, rewards. $r_a$,p logp/q is guided by logp/q plus one to bind $\lambda$. The number of guides is zero. Got it. $\log(p_a/q_a)$ equals $r_a$ Divide $\tau$, add a constant that is not action-related. After indexing $p_a$ equals $q_a$ By e. $r_a$ Divide $\tau$ Subherent by the same constant.

Normalization determines the constant: definition Z is all $q_a$ By e. $r_a$ Divide $\tau$ At the end of the spectrum, Z is strict and limited, so the constant is divided by Z. $p^*$ The distribution with q in favor of the incentive index. Positive q and limited r ensure each $p^*$ True, the candidates are indeed internal and one. We do not consider only first-order conditions as proof of excellence; below we compare arbitrary and viable distributions to candidates directly and cover borders and global dimensions.

An Independent Proof of Global Optimality

From $p^*$ Definition, rewards can be rewritten as $\tau$ Multiply $\log(p^*/q)$ More. $\tau$ $\log Z$. Add to random p. J: expected $\tau$ $\log Z$ (a) remain unchanged due to probability and one; $\tau$ Weighted by p $\log(p^*/q)$ Less. $\tau$ Weighted by p $\log(p/q)$. The two q entrys are removed and the J(p) = $\tau$ $\log Z$ − $\tau$ $D_{\mathrm{KL}}(p\Vert p^*)$ It's a precise algebra equation, not some kind of training approximation. As long as KL is not negative, $p^*$ To achieve global maximum. $\tau$ $\log Z$ 。

KL Non-negotiated zero items for p alone. Definition support set S ={a: p_a>0 }, zero press 0 log 0=0 The continuous limit of treatment. t>0 ，−log t≥ 1−t: The guide for the difference between the two sides is 1−1/t Yes. t=1 The front negative after positive, the minimum is zero. For each ace. S Take t=p*_a/p_a, by p_a S ♪ Up and down, get ♪ $D_{\mathrm{KL}}(p\Vert p^*)\ge 1-\sum_{a\in S}p_a^*\ge0$. The first step to the right cannot be written to zero: if p=(1, 0 )、p*=( 0.7, 0.3 It's... 0.3, and the actual KL=−log 0.7 ≈ 0.356675 This chapter q is positive, with limited rewards > 0, so p* each is strict if S Missing any action, then 1 −Σ_S p*> 0, cannot be equated; if S It's the whole set, and the first one is equal. p*_a/p_a = 1. So the equivalent is just... p=p*, give the only global best. This check is different from the location count and covers the simple border of probability.

Example 1: The Complete Optimum for Two Candidates

q= [1/2,1/2], reward r= [1,0], $\tau$ = 1. Unalified weights are [e/2,1/2], Z=(e+1)/2, $p^*$ = [e/(e+1),1/(e+1)], about [0.731059,0.268941]. Expected reward is 0. 731059, KL about 0. 110944, target difference of approximately 0.620115; another path to calculate $\log Z$ Also about 0. 620115. The two values are equal to the direct proxy check of deductions.

If KL chooses only the first one, but KL is log2, then the target is about 0. 306853, and it's below $p^*$. Reference q itself rewards 0 5, KL zero, target 0 5, also below $p^*$. The best thing to see is not to keep the reference intact, nor to just maximize the reward. It's fixed in advance. $\tau$ The compromise; if the mission really allows a complete rejection of reference, the target should be changed instead of claiming that this is the solution “It's not perfect.”.

Will $\tau$ Changed to 0. 5, $p^*$ Turns sigmoid II and one minus sigmoid II, about [0.880797,0.119203], more focused on high rewards. $\tau$ (b) The differential in index incentives is reduced at times of high intensity and distributed q; $\tau$ When you move to zero, the quality is concentrated on the action that rewards the greatest and is distributed in the q-scales within the largest group. This is the limit for a limited incentive, not implying that the number code can be directly applied. $\tau$ Set zero; formulae and realizations require strict discipline, with log space calculations to be stabilized at extreme hours.

Reward Origin and Scale

Give all r+cs under the same x, and the weight of non-aligned weights is multiplied by e by c $\tau$ Substrate, with total equivalency after standardization, the optimal strategy remains unchanged; the target maximum is only flatc. This freedom is consistent with the joint incentive bias in BT. If you multiply the reward by k, you can also add the positive $\tau$ Multiply k, divided by reward. $\tau$ No change. $p^*$ Same as always. Multiply incentives and do not change. $\tau$, the strategy is more focused and cannot be termed the equivalent of the same question of optimization.

That's why the BT reward unit agreed to go to the back of the DPO. $\beta$..we fixed sigmoid temperature one, $\tau$ As a conversion measure to reward the strategy. Different incentive models may be trained in different scales, and direct re-use of the same KL coefficient may not be comparable; re-calibration should be based on clear objectives and validation protocols. The current project is precisely counting KL on limited responses, and there are problems with the entire text space that cannot be exhausted in the sequence, and Chapter VII will make a strict distinction between arithmetical diagnosis and a true distribution of KL.

Example 2: Zero Support Makes High Rewards Unreachable

Sets the reference q= [1,0], rewards r= [0,100]. If the candidate p gives any positive probability to the second action, there will be a p2 x logp2 in KL, divided by zero, scattered into positive and negative goals. Only p2=0 can be maintained, so the best is [1,0], regardless of how high the second reward may be. The condition for supporting the formula was not established here. It cannot be directly zero and then the strategy automatically discovers the second action.

The reverse is a material condition. Limited softmax mathematics gives positive probability to all actions, but the actual float-point probability may spill over, and codes should prioritize log probabilities and be stabilized and normalized. Project reference small strategy ranges are mild, and the boltzmann function also openly rejects bad and bad q. $\tau$. The rejection does not amount to “Repair” zero support; if the task allows for a change of support, the policy space or smooth reference needs to be redefined and acknowledged as changing the target. An independent check is a list of possible support, not just a function that returns the vector.

The Direction of KL Divergence Matters

If the punishment is changed to $D_{\mathrm{KL}}(q\Vert p)$, the item in the number of guides becomes $q_a$ Divide $p_a$ Not. $\log(p_a/q_a)$ Add one, the best is no longer the same index. Neither KL is zero at p=q, nor negative, but the penalties for rare events are different when far from reference. Seeing a number in the trial log called “KL” is not enough to know which distribution is responsible for expectations, which is placed in a log fraction, and whether candidates are singled out only.

In the language model, the full distribution of KL is a sum of all possible complete responses; the single fixed response log ratio is only one random variable, which may be negative and does not amount to KL. If you sample from the current strategy, logs are expected to move forward to KL with the exception of q and must maintain the same conditions, boundaries and sampling distribution. Limited projects can do four actions, so there are no sampling errors; the full decoder candidate diagnosis is only re-assembled in four specified strings and cannot report KL on this basis for the entire language model.

The Connection to Parametric Training

Distribution $p^*$ The best theory is a x-by-x approach, but the linear strategy to share eight parameters does not necessarily accurately represent it on all x. Even if it can be expressed, limited sub-gradient updates may not arrive. Full decoder has more limitations: indications, limited data, preference model errors, and optimization errors may exist. Parsing provides a baseline for testing the bridges and limited calculations that drive DPOs, rather than a guarantee that any training will run “That's what's going to happen.”.

This chapter item increases the Boltzmann function, a double-circuit calculation of the exact KL with the best value, and a zero-backed failure denial. Chapter VI will write the BT incentive as an optimal strategy against a reference log, using the same hint to be replaced by a unicoded constant, resulting in a DPO loss that is directly calculated in policy terms. Readers should first be able to state that the reward is given here and the next chapter is derived from a comparison between the two levels, avoiding misusing variables as not requiring data preferences or model conditions.

“KLConstraint” should be precisely distinguished from “LessKLPunishment”. The hard constraints will first provide that KL is allowed to exceed a budget and then maximize the incentive within a viable area; this chapter is fixed $\tau$ Optimizes the form of punishment without a prior commitment that KL will eventually be a certain number. $\beta$ 0.1 cannot be used to claim that the model actually deviates just as good as 0.1 nats. Chapter VIII describes the deviation using the actual KL instead of the coefficient name.

When stabilizing the optimal distribution, you form logq and r for each action. $\tau$, take the maximum value of these numbersa, subtract all a and then index it. Public e sub-dimensions of the molecules are reduced to the same result as the direct formula, but avoid high rewards or small ones. $\tau$ Makes the index spill. The maximum a is just a numerical benchmark, not a new incentive threshold, and does not change support. If q itself has been rounded to zero in the float point, it should keep the earlier logq instead of restoring the lost information to the zero-recording hypothesis of stabilization.

A limited-generation tree provides a conceptual bridge from four actions to text. Each node is a prefix that has been created, with the next token conditional probability; the root-to-EOS path probability is a side probability multiplies. If the maximum length is set, the end of a non-EOS trail should be registered as abort result, which together with all completed paths will constitute a complete sample space. Only this full spatial definition and reference will have the same meaning.

When model parameters serve more than one x at the same time, a x-by-x solution is just a target distribution group. Sharing parameter training is equivalent to finding a compromise within an expressable range, not necessarily reaching the upper limit of each Z. Using the maximum value of free distribution less actual parameterization targets can include both expression and optimization gaps and, if reward is also drawn, there are several layers of estimated error.

For the same incentive threshold, the index factor has to dissipate and the optimal distribution must return to the q; this is a very strong single measure. If the function produces even output, it means that the reference factor has been lost. $\tau$ It is also important to approach q rather than the default homogeneity, only coincidentally. Project Boltzmann alone checks both support and best value, and readers can independently check their transplant codes with this degradation case.

import math
q=[.5,.5];r=[1.,0.];tau=1.
weights=[q[i]*math.exp(r[i]/tau) for i in range(2)];Z=sum(weights);p=[v/Z for v in weights]
kl=lambda a,b:sum(x*math.log(x/y) for x,y in zip(a,b) if x)
J=lambda a:sum(x*y for x,y in zip(a,r))-tau*kl(a,q)
assert abs(J(p)-tau*math.log(Z))<1e-12
for i in range(1,1000):
    a=[i/1000,1-i/1000]
    assert abs(J(a)-(tau*math.log(Z)-tau*kl(a,p)))<1e-12
    assert J(a)<=J(p)+1e-12
shift=[q[i]*math.exp((r[i]+3)/tau) for i in range(2)];shift=[v/sum(shift) for v in shift]
assert max(abs(a-b) for a,b in zip(p,shift))<1e-12
print('optimal',p,'reward',p[0],'KL',kl(p,q),'objective',J(p),'logZ',math.log(Z))