Reference Policies, KL and Clipping: Distinct Objectives

Freeze the Reference Distribution Before Measuring Deviation

This chapter fixes a reference strategy q, which assigns a strict positive probability to each of the permissible tracks and does not update in training. Theta is controlling the current strategyp and follows the same two-step syntax. $p(y)\log[p(y)/q(y)]$ Weighted and different. The two directions are: KL(pPresent.q) weighted by the current strategy, KL(qPresent.p) weighted by reference; this text is used only for the former. The same reference supports a defined logarithm, but if one refers to a path that is really zero and is now positive, KL will be endless and cannot hide the target with an arbitrary decimal.

The goal of optimization is to subtract from the expected strict reward beta double KL, Beta being a non-negative number. Beta controls the trade-off between the reward unit and Nat ' s punishment, not the learning rate. Beta zero returns the net reward; Beta is generally more referencing nearby, but there is no limited Beta guarantee that every input will be kept intact. The project uses as reference the deep copy of the complete model after supervision training and the de-gradation of gradients; it is not an old strategy to update each step, and the two roles must be separated.

An Extra Term for Parameter-Dependent Returns

Write the target as the current expectations. The reward seems to be... $R-\beta\log(p/q)$. The return itself depends on theta and cannot be explained by applying “Awards are fixed” chapter II directly. $p(R-\beta\log(p/q))s$, return the number of brackets $-\beta p s$. The second score is zero because the score averages, so the ultimate expectation is still to stop the gradient ' s return by score. The overall expectation is eliminated, not the extra item on each sample is zero.

So it is possible to get lp and reference lq from the current sampling, calculate a advantage with which they are tectonic returns and detach, and then only allow the current lp to be transmitted in reverse. If you direct the return by applying the full lp, there will be an additional error multiplier; if you simply guide the sample KL values to the self-direction and ignore the distribution of samples, it will not be a full trajectory of KL gradients.

Example 1: KL Divergence and Rewards for Two Paths

Sets the reference q to [0.5,0.5], currently p to [0.8,0.2], and rewards [1,0]. The two logarithms of log1.6 are about 0.470004, log0.4 - 0.916291, respectively. KL is about 0.192745, non-negative by p weighting; the logarithmic ratio of a single line to 2 is negative, so that the average logarithm of a sample or a group does not automatically account for KL code error. It is important to distinguish KL from its random estimate.

If Beta equals 0 1, the target is 0.8−0.0192745, or about 0 7807255; the reference target is 0 5. The current strategy is indeed better, but does not mean that the best measure of probability is to continue to be the best one. The best q is slanted by the incentive index when fixed. The second back course is fully guided; the lesson is updated with actual sampling to reach the current parameter group goal and does not claim the best overall advantage of reaching that distribution level.

The Old Policy and the Reference Policy Serve Different Roles

The old strategy p_old is a frozen sample distribution that produces a trail. If you want to re-use this data to evaluate the new p, you have to compare it to the full trajectory. $w=p/p_{old}$ Weighted to convert old distribution expectations into new distribution expectations. The condition is that the old distributions are supported where the new distributions are of good quality, and the value of the ratios can be accumulated. The next batch is updated only once after each sample, and immediately re-taken from a new strategy.

The reference q defines the regular target and remains intact throughout the RL phase. The binding meaning changes with the q step also as the current p p, where the sampling point KL is always zero; the p_old is frozen as a monitoring starting point, and again refers to each batch as equally wrong. The project examines references using an independent copy of the model, requires_grad=False, and an updated pre- and post-model byte summary. Reference and the current structure do not represent references to arguments that can be shared; if both point to the same volume, optimizing the current model will change the reference quietly.

Example 2: The Advantage Sign Determines the Clipping Branch

PPO commonly uses proxy targets $wA$ Multiply smaller values than crop probabilities, psilon fixed positive, clip intercepts w [1−epsilon,1+epsilon]. Take the epsilon. 2 Advantages A=2 margin w=1.4, no reduction of item 2. 8 - No, no, no. 4 Option 2. 4, saturates the local gains that increase the probability of positive action. A=−2 、 w=0.6, undecided - 1.2 the term of the contract - 1.6, Choose-- 1.6, limit the benefits of over-reducing the probability of negative advantage movements.

But take A=−2 、 w=1.4 at the time of the loss, without a reduction - 2.8 the term of the contract - 2.4 The smaller one is still... 2.8, is not saturated. You can't cut w all the time and then multiply A. That's to throw out Min branch and negative advantage. You can't have this local agent claiming it. “KLNever exceed a threshold”; PPO cropping is not a strict global trust domain constraint. The update of shared neurological parameters may change many unexacted paths at the same time, and the proxy goal is only an optimised replacement under a specific sampling distribution.

Token Probability Ratios Differ from Full-Trajectory Ratios

The complete path ratio is the multiplier of the probability ratio per step, and the logarithm is the sum of the logarithms per step. Ten consecutive steps 1.1 accumulate in about 2 59, so that close word-by-word ratio does not directly push the entire path closer to one. Actual PPOs, GRPOs often design proxy, advantage, length one and KL estimates that their formulas and target weights are always itemized and cannot call any “Group incentives to reduce mean value” code the same algorithm.

DeepSeekMath is the original source of relative team optimization. The PPO paper describes old data agents and tailoring. Based on these sources, this course selects a smaller, fully verifiable REFFORCE: a batch of updated, standard-based, old data-free, cut-off losses. This selection is made so that the target, estimated deviations, and the full path KL can be fully accounted for; the tailoring is an independent teaching examination, and there is no claim in the report that the PPO or full GRPO has been actually trained.

An Independent Check of KL Divergence

A limited path can weigh KL against the probabilities of each step by a prefix, and it can be broken in the chain of track KL: first step KL, plus the weighted second step KL for the current first step. This relationship comes from spreading logarithms and exchanging limited summation; the second weight must be the first step probability of the current p. If the reference q weighted step 2 prefix is used, the volume is changed. The project evaluation uses a more direct 100-track full listing with probabilities and close to one joint check; the floating point error allows about 10 negative six dimensions.

This chain relationship is actually developed: the first step is to write the probability of p(u), q(u), and the second step is p(v|u), q(v|u). The log ratio in the path item is broken into log [p(u)/q(u)] plus [p(v|u)/q(v|u)]. The first part is to find a peace, with only the first step KL; the second part is to fix the u-to-v request for KL and then to match p(u). Each order of exchange is legal because there are only 100 limited items and does not involve an unlimited number limit.

For example, first step p [0.8,0.2], q evenly, the conditions under the first prefix are distributed equally so that KL is zero and the condition under the second prefix is 0.5 nt, and the trajectory KL is 0.1927445 plus 0.2 times 0. 5 About. 292745. If you wrongly use the q prefix weight. 5 It's gonna count to zero. 442745, the difference is not a sample noise, but a misspelled target. Check the weighted distribution of the prefix to avoid mislabelling a different KL volume as a achievement error. The two-action tree here is only a reconciliation formula. The actual network is still 10 action two steps, and the full listing does not depend on this hand set value.

Only a sample of logp minus logq is evaluated in its entirety. The former cost is positive for the batch, while the latter only increases with step-by-step indexes by 100 paths in two steps and ten. Real and long text cannot be copied. It should not be possible for this project to extrapolate KL to a large production model to give an easy distribution of KL. Chapter VII uses this limited and precise tool to reveal that training has been successful at the same time as testing degradation and to preserve capability boundaries beyond its original goal.

import math
p=[.8,.2];q=[.5,.5]
kl=sum(a*math.log(a/b) for a,b in zip(p,q))
assert abs(kl-.19274475702175753)<1e-12
def clipped(w,a,eps=.2): return min(w*a,max(1-eps,min(1+eps,w))*a)
assert abs(clipped(1.4,2)-2.4)<1e-12
assert abs(clipped(.6,-2)+1.6)<1e-12
print(kl,clipped(1.4,-2))