Within-group Comparisons and Valid Advantage Estimates

The Baseline as an Expectation Identity

Chapter II is estimated by the gradient of the incentive score, but the difference may be large. This chapter gives returns less baselines; the baseline is not real values, but reference numbers. $b(x)$ Without relying on the current trajectories, it multiplys the baseline by zero, minus the expected gradient. The baseline is neutral even if the values are wrong; it affects the variance rather than the constant equation. Baselines are allowed to be estimated by other independent samples, provided that they are independent of the current sample after giving X and stop the gradient when calculating score losses.

The extreme example is that the baseline is equal to the current reward, with all advantages zero and the gradient always zero; it is clearly no longer equal to the general non-zero target gradient. Another common confusion is that “No prejudice.” should be understood as being in a better direction for each batch of updates. The average value of repeats samples when describing fixed parameters without bias may still be poor estimates for one batch; the parameter trajectory is not an unbiased copy of the exact gradient trajectory after changes in non-linear optimizers, tailoring and subsequent parameters.

Example 1: A Minimal Two-Action Control Variate

Makes the correct action half probability, rewards one, rewards zero. For the correct action logit score, half correct and half negative. Without baseline, the estimater takes half probability, half zero probability, average value E [G] = 1/2 × 1/2+1/2 × 0=1/4;second order E [G²] = 1/2 ×( 1/2) ²= 1/8 is defined as: Var(G) = E [(G−E[G] )²]= E [G²] − E [G] Two, so here's the... 1/8−(1/4) ²= 1/16. After reducing the constant baseline by half, when correct $(1-1/2)(1/2)=1/4$, when error $(0-1/2)(-1/2)=1/4$, the variance is zero and the average is unchanged.

This example does not allow for the introduction of a baseline to expect the reward to always make all parameters the lowest gradient difference. Var(G) = E [G²] − E [G] The mean square in 2 is not related to b, so the minimum difference equals the minimum second order rule. $E[(R-b)^2s^2]$ , cross-referenced to b to guide $b^*=E[Rs^2]/E[s^2]$, the pre-determining denominator is positive and the rectangular is present. The vector score, if the target is the sum of the coordinates, also spreads squarely to obtain b*= E [R‖s‖²] / E [‖s‖²]; this does not guarantee that each of the coordinates is best. Only score square weights and incentives satisfy the appropriate relationship is equal to the average average reward. This project uses an interpretable set of averages and does not claim to be the best control variable for the difference in the gradient of the entire neural network.

Why Including the Sample in Its Group Mean Shrinks the Estimate

The same input independently draws a G trajectory, so that Article I is rewarded as $R_i$, score $s_i$, the group mean value is the sum of all rewards and divided by G. Use $R_i-\bar R$ As an advantage, the baseline contains its own rewards and is therefore not independent. Roll out expectations: the 1/G item contributing to the original gradient of its own, and the other G−1 entry multiplys zero for independence. Less each expectation is the original gradient. $1-1/G$, the average number of groups is still this scale.

The conclusions are for estimates of standard deviations, separate distribution with input, fixed parameter sampling, and the same weight for each. G To reduce it to seven-eighths at 8 o'clock. 7/8 The positive constant scaling can sometimes adjust the absorption of learning rates, but it cannot be called a strict and unbiased estimate of the original target. If the Task Force is not small, different input will be scaled differently, and even the task weights will change, and it is no longer a simple global learning rate that can be modified.

Leave-One-Out Group Means Restore Independence

Definition of a baseline remaining in Article i as other G−1 An average of the given input and the current parameter. Independent sampling guarantees the independence of the other tracks from article i, so the baseline is set by article i score to zero. For each article, it is averaged and the target gradient is estimated without bias. It is also equal to its own average advantage by G divided. G−1, so the code can be calculated efficiently at a time with its own aggregate reduction, without really recycling the G-list.

Project run. py 's loo_advantages group flat batches to reshape, requiring G at least two and total elements to be removed from G. It seeks each group first and the current value is reduced to G−1. All task groups must be placed continuously; if the different task samples are disrupted first, they are still in eight groups, which may occasionally constitute an independent baseline, but no longer the agreed input control variable, differences in difficulty will increase the variance and interpretations may not be consistent with the instructions. Sample numbers and input records are more useful than keeping only one average reward.

Example 2: Computing Advantages for Four Trajectories

Grouping rewards [1,0,0,0]. The average value of the group is 0 25, with its own advantages [0.75,-0.25,-0.25,-0.25]; leaving a baseline for [0,1/3,1/3,1/3], with its own advantages [1,-1/3,-1/3,-1/3], which is the same for 4/3. Both sets of advantages are zero, but “Advantages and Zeros” cannot prove the expected gradient itself, because it has to multiply the respective score. Zero and only describe the algebraic nature of the group 's centralization.

If the reward is converted to [0,0,0,0] or [1,1,1,1], the two central advantages are zero. They do not have relative information in the group and they do not equal to the entire strategic gradient: other groups that may be drawn can still contribute to non-zero. The project actually includes the probability of sampling logarithms in return for punishment, so the rewards are the same but with different probabilities, and there may still be signals in the positive parts; the training start strategy is fully consistent with the reference, with zero penalties, and without success, the first batch may indeed not be updated at all.

Standard-Deviation Normalization Is Not a Free Operation

Some of the relative methods of centralizing the incentives by removing the standard deviations from the group plus the small constant. The standard deviation is the current whole random variable, dependent on the current trajectory and associated with score, so that the previous independent baseline evidence cannot be directly applied. Even without its own baseline, a random denominator cannot be expected to be a constant. This may control the scale, but optimizing the weighting of objects and tasks changes; it must be studied as a different estimate, rather than using “Values are more stable” as proof of maintaining the original goal.

For example, both sets of the same input have only one success, but a set of incentive margins has been artificially scaled tenfold. The gradient is also magnified tenfold without a standard deviation; the standard deviations are almost offset by one. This is not necessarily correct, but they have different meanings for the reward scale. The GRPO proposed by DeepSeekMath also involves old tactics, cropping, KL, and specific advantage definitions, and this course uses its group comparative perspective alone, and the real project achieves a single REINFORCE without a full GRPO recovery name.

How Independence Assumptions in Sampling Can Fail

If the same group of beam searches obtains several powerful and relevant paths, or if they are copied eight times after a single trajectories have been drawn, other tracks are no longer independent and a baseline is left with current action information. Each reward is identical in the case of replication, with zero centralization, regardless of whether the true gradient is not zero. The conditions are also recalculated and cannot be demonstrated separately.

Fixed random seeds allow the experiment to be repeated, not the same random numbers to be repeated for all members. Torch. Generator controls successive random streams, and multiminormial produces separate samples of each row in the batch. Tests with the same initial seed should be reruned by the same track that allows drifting; not every member in the training will reset the same seed to produce a fully relevant sample.

# estimate
def estimate(rewards,scores,mode='loo'):
    raise NotImplementedError('loo')

# estimate
def estimate(rewards,scores,mode='loo'):
 n=len(rewards)
 if n<2 or len(scores)!=n:raise ValueError('at least two paired samples')
 if mode not in ['none','self','loo']:raise ValueError('unknown baseline')
 total=sum(rewards);out=[0.]*len(scores[0])
 for i,(reward,score)in enumerate(zip(rewards,scores)):
  baseline=0 if mode=='none'else total/n if mode=='self'else (total-reward)/(n-1)
  for j,s in enumerate(score):out[j]+=(reward-baseline)*s/n
 return out

This Lesson's Deliverable: Enumerable Evidence of Deviation

Stdlib Lab.py lists all the lists of three action strategies and G equals two, three, and four, totalling nine, twenty-seven, and eighty-one groups, respectively. Each set of probability is a multiplier of action probabilities, adding each advantage to the average of score; leaving one estimate by coordinates is equal to the solvency gradient, with its own mean value equal to its fixed scaling. This is not a beautiful conclusion from a large number of Monte Carlo samples, but rather an exact weighting of limited space, influenced only by a floating point slippage error.

Readers should also take the initiative to remove the condition of independence: if a group is forced to copy, there will be a zero gradient. If a G is removed, leaving a zero denominator, the code must reject instead of adding a small psilon. For a zero reward, the correct action is to keep zero information rather than to surreptitiously replace a false success. The next chapter will add the reference strategy to the reward, maintaining all sampling, independence and stop gradient conditions in this chapter in order to induce mathematics to real training.

from lab import finite, group_expectation
z=[.3,-.2,.1];g=4
t=group_expectation(z,g);truth=finite(z)["gradient"]
assert max(abs(a-b) for a,b in zip(t["leave_one_out_gradient"],truth)) < 1e-12
print(t)