Derive the Full-trajectory Policy Gradient from Finite Sums

Write Down the Quantity to Optimize

For the time being, this chapter uses no neurological network. Only a limited strategy of three actions is available. $z=(z_0,z_1,z_2)$ It's three real points, the probability of being downsized by softmax; the reward is fixed at one point in action zero and the rest zero. $J(z)$ The expectation here is to repeat the average of the pumping action obtained by the same probability distribution, not the direct equivalent of the three incentives. When entered, this already contains the most central difficulty of the project ' s strategic learning: the sampling distribution itself depends on the parameters to be updated.

Readers know the guide and the principle of the commercial. $Z=\sum_j e^{z_j}$, $p_i=e^{z_i}/Z$. The k points are guided by the molecules' instructions, the denominator's index entries, and the numeric ones are sorted as $\partial p_i/\partial z_k=p_i(\mathbf1_{i=k}-p_k)$. The logarithmic probability lead is obtained by dividing it by a positive probability. It is called score, which indicates the local sensitivity of the logarithmic probability to the parameter that has been pumped, not the reward score.

Probability Normalization Implies a Zero-Mean Score

The score of each action is weighted by the probability of the action, and it has a sum of probabilities. Limited and can be guided on a case-by-case basis, while all probabilities are consistent on a single basis, with zero. This is the feed point for the back baseline and KL. The condition is that the current parameters are close to supporting non-frozen parameters, and that the current ten figures softmax are always strict and meet the conditions. It cannot control the hard threshold of “Is this allowed?” without the same guidance formula.

Yeah. $J=\sum_i p_i r_i$ Guides, rewards are fixed in this chapter, and no arguments are guided. Using the probabilistic guide is equal to the probabilities multiplied by the probabilities of the logarithm, the tactical gradient is the incentive to multiply the score expectations. This is an exact constant equation in a limited sample space, without the need for schilling samples to become endless. $r_i\nabla\log p_i$ It is only a random estimate of this expectation, which may be in poor direction; the precise constant equation must be separated from the limited sample estimate.

Example 1: A Complete Three-Action Gradient Calculation

The first three points are zero, so one third of the probability for each action is expected to be rewarded by one third. $p_0(1-J)=2/9$; split between the other two $p_k(0-J)=-1/9$. Three and zero, matching all points to the same constant does not change the softmax peer symmetry. Walking down this gradient is small enough to improve the relative score of the correct action; not three scores rise indefinitely together.

When you draw the correct action, the single sample gradient is [2/3,-1/3,-1/3], with a probability of one third; when you draw the wrong action, zero is rewarded, and the single sample gradient is zero, and [2/9,-1/9,-1/9] is combined. The zero gradient at the time you draw the wrong action cannot be interpreted as “The error action does not need to be reduced”, while the correct action is upgraded to the level where the other probabilities are squeezed. Once you add the baseline, the error action can also be given a non-zero sample gradient, but the overall expectation remains the same, and chapter III will prove precisely.

Then we'll score asymmetrically. [0.3,-0.2,0.1], item lab.py output actual probability, parsing gradient, and center limit difference. The center score adds or subtracts the k score by a smaller volume, h, recalculating the J, divided by 2h. When the function is smooth and the h is moderate, the cut error is a second scale; h is too small to be dominated by a float offset error. The project uses a negative five-point ratio of h to ten and examines the error rather than in general terms. “Numerical guidance is the exact value.” 。

From Individual Actions to Complete Trajectories

Now back to input x and two-step response y. Use. $\theta$ The self-regression definition states that the probability of the first step is determined by input, and that the second step is conditioned on the first step already written, with the full probability multiplying both. Taking a logarithm turns the multiplier into a sum, then guides the shared parameter, and gets two positions score. A successful end reward goes on the whole and means that the front and back actions take credit; not just updating the output of the last number. Both steps go through the same Transformer, and the parameter gradients are also transmitted through the middle invisible state.

The reminder part is not a tactical extraction response, and does not add the hint-word probability to the apparent response. End action forces a probability of one, zero logarithm, zero gradient in this agreement, and should not be added to a false negative logarithm for 14 groups of softmax. If a free text agreement is changed, EOS will become a real random move, requiring a redefinition of sample space, cut rules and approximation. This project deliberately fixes two steps to see these boundaries clearly and does not pose fixed length conclusions as any long text.

Example 2: Calculating Two Paths with a Shared Prefix

Sets the probability of a first step four at 0. 2, and the probability of a second step eight after four is 0. 3, or the path [4,8,EOS] is 0. 06, or log0.2 plus log0. 3. If the final validation is passed, the strategy gradient sample is the sum of the first step score and the second step score. If the second step log0.3 is used, there is no direct first step to select credit, the target is no longer the probability of a complete path; That's why the path_logps in project have to add two places together.

Another [4,7,EOS] shares the first step, which is different and failed. The local gradients of the two paths are not separate from the shared parameters, because the same input and prefix are calculated using the same graph. After sampling, the response numbers are re-estimated as fixed data to legal score; do not try to find a normal guide for the output number of the discrete multinomial. What we seek is a sample of the probabilities of the number that has been processed through the desired constant equation.

Signs, Averaging, and Update Order in the Code

Optimizers usually minimize losses, so the maximum incentive is given to the target of the sample with a negative number. A single sample loses a negative incentive times the average of logarithm probabilities, the gradients are reduced after inverse transmission, and the gradients for reward targets rise. If used and not average, the updated scale is multiplied by batches at the same learning rate; the volume cannot be changed, but the experiment is still described as exactly the same optimal setting.

The training cycle must first empty the previous gradients, calculate the current map, reverse disseminate, check the limited values, and then crop and update them. PyTorch defaults on the cumulative gradient; forgets that emptiness can be mixed into the old batch. Incentives, baselines and reference probability stop the gradient in the estimater as required, chapter IV explains why; model current log probability retains the gradient. Detach does not allow models “No study.” to be used, but rather avoids miscalculating the weight of the estimate device as another path for minor loss.

Independent Checks and Conclusions That Must Not Be Substituted

The project provides three-track reconciliations: stdlib's three actions resolve the difference between centre; full 100 paths seek direct automatic fractions of summation; and the same one holds only the score path after stopping the gradient for probabilities and returns. The latter two paths equal the algebra check not for sample accuracy, but rather for target gradients.

These checks only demonstrate consistency in the local gradients of the specified function. It does not guarantee that limited step optimization is best on the global scale, nor that training topic upgrades can be moved to unincorporated input. The accompanied models, limited step lengths, cropping, Adam status and sampling differentials are all involved in the actual trajectory. The project therefore retains initialization seeds, monitoring steps, RL steps and a gradient book for each phase; Chapter VII uses the freeze check point to measure the true results rather than using them. “Gradient is correct” Alternative “Model Society” 。

from lab import finite
r=finite([.3,-.2,.1])
assert r["max_error"] < 1e-9
assert abs(sum(r["gradient"])) < 1e-12
print(r)