Backpropagation and Optimization

Objectives, Boundaries, and the Computation Graph

This chapter answers “How the loss tells each parameter where to go.” The premise is that the pre-LN block, matrix multipliers, single-digit guides and deflection meanings of Chapter V; this chapter extends the multi-chain approach to head and LayerNom, with a limited difference for independent verification. You should be able to distinguish the precise gradient of the loss at the end, the approximate value of the gradient checked, and the rules for updating the optimizer selection. The latter are not the same object, nor can they provide evidence of a loss decline once as a whole.

Calculates the chart with no rings: node is the middle number or volume, which means downstream calculations use upstream values. The parameter is the input that needs to be guided, the word number is separated data and does not guide the number itself. The mark loss is recorded as $\ell$ Node $a$ , and the upper gradient is recorded as $\bar a=\partial\ell/\partial a$, it describes the first step of the effect of changing the node on the ultimate loss. If a node is used by two paths, the two chain contributions must be added and not retained for the last one.

For example... $u=ax,v=u+x,\ell=v^2/2$ of which $a$ is the parameter, $x$ For the given input. First $\bar v=v$ And more. $\bar u=\bar v$;entry gradients are given along the direct plus sub-routing $\bar v$ Give it along the sidewalk. $a\bar u$ Total $(1+a)v$; argument gradients are: $x\bar u=xv$. This is the reverse of the debris structure. Auto-minority saves the necessary intermediate values and reverses this local rule, not recombine a “Gradient Model”.

From Softmax Cross-Entropy to the Output Matrix

Take a valid forecast position. Hide the vector as $h\in\mathbb R^d$ output weight is $W\in\mathbb R^{d\times V}$ of which $V$ is the scale of the phrase sheets;logits path vectors $s=hW$. The real next character number is $y\in\{0,\ldots,V-1\}$. Definition $Z=\sum_j e^{s_j}$ Probability $p_j=e^{s_j}/Z$ Negative logarithms $\ell=-\log p_y$, the logarithms in this chapter are natural and the unit of loss is nat.

Here. $\mathbf1_{i=y}$ If you have an equal number, you will have one, otherwise zero. $\partial p_j/\partial s_i=p_j(\mathbf1_{i=j}-p_i)$;rewards $-1/p_y$ Multiply $j=y$ This is the second independent deduction. $\sum_i\bar s_i=\sum_i p_i-1=0$, which corresponds to all logits plus the same constant does not change probability and loss.

Numerical realization cannot be directly indexed to large logits. $m=\max_j s_j$, write down the price of loss $-s_y+m+\log\sum_j e^{s_j-m}$. This is an algebra constant variant, with no more than one index, to reduce the risk of overflow; not to close the probability. The hand-plug gradient can still be calculated from the original constant equation and does not require “Maximum value position” to be used as an additional discrete decision for a true model. A limited float point remains rounded and extreme margins may spill over, so code is used to match temperature input.

For multiple active positions, a weight gradient of the same shape is created by location and then reconciled or averaged by the number of valid words; Chapter VIII will deal strictly with padding. Use one position first, so no extra bits or phrases are removed. If the frame head. weight store shape is $[V,d]$, the gradient is a top shift. The shape is legal only for multiplication, whether the target moves right or whether the average denominator is correct or not.

Example 1: Full Gradients and One Descent Step for Two Head Types

Take $h=(1,2)$ ， $W$ The two lines are the same. $(0.2,-0.1)$ and $(0,0.3)$ Target zero. Logits is yes. $(0.2,0.5)$;minus index is $(e^{-0.3},1)$, the probability of being normalized $(0.4255575,0.5744425)$. The loss is $\log(1+e^{0.3})\approx0.8543552$, so logits gradient is $(-0.5744425,0.5744425)$ 。

The first line of weight gradient is to hide the first coordinates one times that gradient, and the second two times that action takes two lines. $(-0.5744425,0.5744425)$ 、 $(-1.1488850,1.1488850)$. The first of the hidden gradients is negative 0:5744 times zero 2 plus positive:05:744 times negative:01, negative:17233; the second is positive:05:744 times 0.33 and positive:17:7233. If the target is miswritten into one, both types of gradients will be replaced by another, and the procedure can still fall but training for the wrong task.

Use of learning rates $\eta=0.1$ One normal gradient down, press $W_{new}=W-\eta\nabla_W\ell$ Updates. The new two-line weights are about $(0.257444,-0.157444)$ and $(0.114889,0.185111)$;newlogits about $(0.487221,0.212779)$ The target group has moved from backward to ahead, losing about 0:5653. This decline is an observation of the value input, not a theory that any arbitrary learning rate is valid; excessive learning rates may still undermine more complex networks.

import math
h=[1.0,2.0]; W=[[0.2,-0.1],[0.0,0.3]]; target=0
def loss(w):
    s=[sum(h[i]*w[i][j] for i in range(2)) for j in range(2)]
    top=max(s)
    return -s[target]+top+math.log(sum(math.exp(v-top) for v in s))
s=[sum(h[i]*W[i][j] for i in range(2)) for j in range(2)]
e=[math.exp(v-max(s)) for v in s]; probs=[v/sum(e) for v in e]
gs=[p-(j==target) for j,p in enumerate(probs)]
gw=[[x*g for g in gs] for x in h]
step=1e-5; errors=[]
for i in range(2):
    for j in range(2):
        plus=[r[:] for r in W]; minus=[r[:] for r in W]
        plus[i][j]+=step; minus[i][j]-=step
        numerical=(loss(plus)-loss(minus))/(2*step)
        errors.append(abs(numerical-gw[i][j]))
new=[[w-0.1*g for w,g in zip(row,grad)] for row,grad in zip(W,gw)]
assert max(errors)<1e-8
assert abs(sum(gs))<1e-12
assert loss(new)<loss(W)
print('loss',loss(W),'after',loss(new),'max gradient error',max(errors))

Why LayerNorm Backpropagation Must Differentiate the Mean and Variance

Enter the length following chapter V $d$ mean $\mu$, centralize coordinates $c_i=x_i-\mu$ Population difference $v=\sum_i c_i^2/d$ ， $r=\sqrt{v+\varepsilon}$ ， $z_i=c_i/r$, output $o_i=\gamma_i z_i+\beta_i$. Stabilised constants maintained $10^{-5}$. Now give the upper gradient $a_i=\partial\ell/\partial o_i$, Definition $u_i=a_i\gamma_i$, $\bar z_i=u_i$;simulation parameter gradients, respectively $a_i z_i$ and $a_i$ 。

To expand the input gradient, $\delta x_i$ is any minor disturbance. The mean change is $\delta\mu=\sum_i\delta x_i/d$ # So, why don't you # $\delta c_i=\delta x_i-\delta\mu$. The square is guided by $\delta v=2\sum_i c_i\delta c_i/d$. Because of centralised coordinates and zeroes, multiplied $\delta\mu$ ♪ One to go away, one to go ♪ $2\sum_i c_i\delta x_i/d$. The changes are input dependent and cannot be set to zero at the time of backpropagation.

Loss first step changes to $\delta\ell=\sum_i u_i\delta z_i$. The first one goes to each of the top. $\delta x_i$ Factor $u_i/r$;average items to all coordinates common coefficients $-\overline u/r$;margin to first $i$ Coordinate coefficient $-c_i\overline{uc}/r^3$. The horizontal line represents the arithmetic average along the feature axis and does not represent the cross-batch average. So you read the full input gradient:

Summarize all coordinates, first part zero, second part contains $\sum_i c_i=0$, so enter the gradient and zero. This is a reverse check of constant change. Another frequently misused check is “Gradient and standardized vector orthodoxy”: When stabilization constants are not zero, normalization is not static, so the internal build-up is generally not rigid. The stabilization constant cannot be deleted for the purpose of passing an inspection that applies only to eps=0. The gamma, Beta gradient shared in multiple locations needs to be collected along batches and times and cannot leave only the last position.

Example 2: Incorrect Gradients from Omitting the Statistical Path

Take $x=(1,2,3)$, default gamma I, Beta zero, loss is only the first coordinates of the unicoded output, i.e. $\ell=z_1$. Upstream $u=(1,0,0)$;average 2, $c=(-1,0,1)$ ， $v=2/3$ ， $r=\sqrt{2/3+10^{-5}}$ ， $\overline u=1/3$ ， $\overline{uc}=-1/3$. By formula, the gradient is $2/(3r)-1/(3r^3)$ 、 $-1/(3r)$ 、 $-1/(3r)+1/(3r^3)$ - I'm sorry. $(0.20413,-0.40825,0.20411)$ The sum of three is zero.

Failed to achieve average and variance as constants, direct only to the current coordinates of molecules and give $(1/r,0,0)$ - I'm sorry. $(1.22474,0,0)$. Not only is it very different from the difference between the centre, the gradient and the zero; it miscalculates loss changes along the direction that all features move together. This double evidence is more reliable than checking only one coordinates, positive or negative: one check comes from precision variables and the other from independent numerical disturbances.

import math
x=[1.0,2.0,3.0]; eps=1e-5
mu=sum(x)/3; c=[v-mu for v in x]
r=math.sqrt(sum(v*v for v in c)/3+eps)
u=[1.0,0.0,0.0]; mean_u=sum(u)/3
mean_uc=sum(a*b for a,b in zip(u,c))/3
g=[(a-mean_u)/r-b*mean_uc/r**3 for a,b in zip(u,c)]
def loss(values):
    avg=sum(values)/len(values)
    variance=sum((v-avg)**2 for v in values)/len(values)
    return (values[0]-avg)/math.sqrt(variance+eps)
for delta in [1e-3,1e-5,1e-7]:
    numerical=[]
    for i in range(3):
        plus=x[:]; minus=x[:]; plus[i]+=delta; minus[i]-=delta
        numerical.append((loss(plus)-loss(minus))/(2*delta))
    error=max(abs(a-b) for a,b in zip(g,numerical))
    print(delta,error)
    assert error<1e-5
assert abs(sum(g))<1e-12
print('exact formula',g,'wrong frozen-statistics',[1/r,0,0])

From Local Gradients Back to the Complete Block

The front feed linear layer reverses the same rule as just head: the weight gradient is the extra volume of the input and output gradient, the input gradient times the weight is reset; the shared bias gradient is found along all positions. Accurate GELU leads are obtained by multiplying rules $g^{\prime}(a)=\Phi(a)+a\phi(a)$ of which $\phi(a)=e^{-a^2/2}/\sqrt{2\pi}$ It's a normal density, and $\Phi^{\prime}=\phi$ This is guided by the upper limit of the points. First you go through the second linear layer, then you go through this element-by-element number, then you go through the first level, down to the first level, and then you add the branch input gradient to the main path of the debris.

The center's differential. $[L(\theta+\delta)-L(\theta-\delta)]/(2\delta)$ Approximate guides. If there is a three-order continuous guide in the adjacent area, the number of bits that Taylor has started is equal to the cut error. $O(\delta^2)$;floating points result in follow-up $1/\delta$ A larger rounding effect, so the step is not as small as it is. Closes the dropout before checking, uses double accuracy, fixes the same input, and avoids updating the optimist between positive and negative disturbances. The check in this chapter validates the point and does not substitute for global deconstruct.

Completing Backpropagation Through the Attention Branch

To end a block of chain law, fix a single head shape: sequence length is $T$ ♪ Head wide ♪ $d_h$, query, key and value matrix $Q,K,V$ Both. $[T,d_h]$ Scores $S=QK^{\mathsf T}/\sqrt{d_h}$ Yes. $[T,T]$. The cause-and-effect mask specifies which locations are legal, and the legal position follows the path softmax to get a probability matrix $P$, head output is $O=PV$. The capital case here. $V$ The value matrix is not the size of the pre-vocabulary in this section; the following formula is used locally and the output sheet is still headed by head.

Upstream gradient for a given head output $G$ Expand $O_{ia}=\sum_jP_{ij}V_{ja}$. A value element is guided and all query lines are aggregated, get $\bar V=P^{\mathsf T}G$; guidance for a probabilistic element and aggregation of output characteristics, obtained $\bar P=GV^{\mathsf T}$. The first shape is a head width matrix, the second is a fractional array. The next line applies the derived softmax aberrations, and the simplified formula for cross-entropy cannot be applied directly to any upstream gradient.

The first one comes from a cumulative item. $\sum_k\bar P_{ik}\,\partial P_{ik}/\partial S_{ij}$, leave the instruction and then put the public probability factor in place. The second two sets respectively guide the query and the key in the fraction, and the zoom factor remains unchanged. When you return to project, the query path is given to input gradients. $\bar QW_Q^{\mathsf T}$, the key and the value path are given to the same item, and the three are transferred to the unicode output. The puzzle is just rehearsing the coordinates, reversed; the output projection is still based on linear outlier rules. The result mask is a fixed constraint rather than a trainingable weight, and the reverse transmission does not give the probability of disallowing the position. The order of examination should first verify the single line softmax, then the matrix multiplier, and finally the complete branch; otherwise, a total gradient error cannot tell you which layer first deviates.

Check the fraction gradients and zeroes per row: check the first pair of columns to sum up, and the second to include probability and probability. If only it is legal in its first position, the probability is one, the gradient of the line ' s query and key is likely to be zero; this is not a breakchart. The parameters have an accessible gradient path, a batch of gradients is not zero, and changes in the post-update parameter are three different assertions. The two must be checked before input containing multiple legal past positions, using a limited differential to judge suspected errors, rather than requiring all elements to be non-zero each time.

AdamW: An Update Rule, Not a Consequence of the Chain Rule

Sort all actual models to be updated into parameter vectors $\theta$ In the $t$ Compute the current gradient before next update $g_t$ AdamW maintains a sliding average by coordinates $m_t$ Average peace gradient $v_t$; the starting value is zero. $\beta_1,\beta_2$ Between zero and one, learning rate $\eta>0$ Decay coefficient $\lambda\ge0$, optimizer stabilize constant $\epsilon_a>0$ And it's with the eps that are being normalized are two different super-parameters.

Why are you trying to correct the bias? If the gradient is always expected to be the same at each step, you can expand the one-step forward coefficient. $1-\beta_1^t$, therefore, no correction of expectations has been made to the zero start value; except for this coefficient, which removes the initialized bias under the model. The second step is calculated in the same way for the square gradient. This reasoning explains the correction factor and does not demonstrate that the gradient distribution of actual unstable training is constant, let alone that optimization is inevitable.

Take a single parameter starting value two, gradient zero one, learning rate zero one, decay zero one, two beta at zero nine and zero nine nine, optimise ps $10^{-8}$ The first step is zero point and zero point and zero point and zero point and zero point and zero point and zero point and zero point point and zero point and zero point and zero point and zero point and zero point and zero point and zero point and zero point and zero point and zero point and zero point and zero point and zero point, and that's about one point and eight and eight. $\lambda\theta$ First, the gradient is added and the rectangular and denominators change. That is another update that cannot be called a local equivalent.

Global gradient crop first calculated $q=\sqrt{\sum_k g_k^2}$, threshold $c>0$ Again. $g^{\mathrm{clip}}_k=g_k\min(1,c/q)$, the zero-varius ratio is agreed to be one. The gradient of three to four is five, and the threshold becomes zero-six and zero-eight. Cropping is an optimist idea, not a change in the original loss resolution gradient, nor is it generally a guide for a fixed new target. AdamW is used to crop and update the rectangular; it affects history and does not merely record weights while saving the check point.

import math
parameter,grad,lr,decay=2.0,0.1,0.01,0.01
b1,b2=0.9,0.999
m=(1-b1)*grad; v=(1-b2)*grad*grad
mh=m/(1-b1); vh=v/(1-b2)
new=(1-lr*decay)*parameter-lr*mh/(math.sqrt(vh)+1e-8)
assert abs(new-1.989800001)<1e-10
raw=[3.0,4.0]; norm=math.sqrt(sum(x*x for x in raw))
clipped=[x*min(1.0,1.0/norm) for x in raw]
assert abs(sum(x*x for x in clipped)-1.0)<1e-12
print('AdamW first step',new,'clipped gradient',clipped)

Project Contributions and Later Dependencies

The project first clears the old gradient, gets lost, reverses, checks limitedness, tailors globally, and then calls the optimist to update; if it is updated backwards, it is updated by an empty gradient or the previous gradient. Automatic fractions can cover all layers of reverse transmission, but a critical path must be reviewed through the head outside of this chapter and the LayerNom small case. Chapter VII examines whether each type of parameter in the complete model is involved in the calculation and Chapter VIII saves weights, rectangularities, steps and sample states; the number of guides in this chapter cannot replace these system inspections.