Work Through Attention Step by Step

Task: Verify Both the Forward and Backward Pass of a Retrieval Step

The first two chapters have defined token, embedded matrix, softmax and cross-entropy gradients, but each location has not been read for other positions. Now construct a single focus: each position is matched by a query with a key, mixing the value vectors of each location. Eventually, one mark of loss will be traced back to Q, K, V, and then back to the project and input that generates them. Here only the full reference case of the two locations is calculated, without cause masking; chapter IV limits visible information and adds multiple head and position.

First, the matrix multipliers, the chain-of-mark rules in chapter II and the shared gradients are used to sum up. $X\in\mathbb R^{T\times D}$ ， $T$ It's the number of locations. $D$ is the input width; $W_Q,W_K\in\mathbb R^{D\times d}$ ， $W_V\in\mathbb R^{D\times r}$. Query $Q=XW_Q$ Keys $K=XW_K$ Both. $T\times d$ value $V=XW_V$ Yes. $T\times r$ 。 $d$ The point of query and key count must be the same, value width $r$ It can be different here. $V$ Indicates a value matrix, the size of which does not participate in the formula in this chapter and avoids mixing with the other meanings of the same letter in the previous chapter.

1. Assigning Roles to Three Projections Before Defining Retrieval

Queries specify what features the current line is looking for, and the key specifies which features each candidate position matches, and the value specifies what is read. These are the computational duties of three linear maps, not three textual or artificially defined semantic. Attention is given to the fact that the three are generated by the same X, but the weight matrix is independent of each other. If the three matrices are accidentally tied to the same parameter, the model family has changed and cannot be considered equivalent simply because the shapes are identical.

Query Location $i$ And the candidate position. $j$, calculate point points $S_{ij}=\sum_a Q_{ia}K_{ja}/\sqrt d$. Each query should be regularized separately from all candidates: $A_{ij}=\exp(S_{ij})/\sum_k\exp(S_{ik})$ And finally... $O_{i\ell}=\sum_jA_{ij}V_{j\ell}$. A matching score, followed by a line weight, and then a weighted reading, with a three-step index having different meanings. If you are re-allocated, another resource allocation is obtained, which usually does not guarantee the read weight and one for each query.

A’s line is unicoded to “Read Which Locations”; p Chapter II is unicoded token: “What next is the category?” O is a mixture of numeric vectors, not necessarily non-negative, and not one for each peacekeeping operation. Even if A is painted as a probabilities map, it cannot be used directly as a prediction token, or as proof of the correct answer.

“Convert” should also be converted from a shape mark to a physical index. If Q acts [1,2], [0,1], K acts [1,0], [2,1], the first line and the second key count are 1 and 4 respectively, and the second action zero and 1 respectively. Correct QK is changed to [[1,4], [0,1]; missing is transferred to [[5,2], [2,1]. The main case selects the diagonal matrix for ease of calculation, but it hides certain conversion errors, and therefore an independent inspection must increase this asymmetric input.

2. The Assumptions Behind Scaling

Explains the motion of scaling using a clear random model: assumes that each coordinates of query and key are equal to zero, one variance, that the query and key are independent of each other, and that the different coordinates are separate from each other. $q_ak_a$, equal value zero, second order rule is $\mathbb E(q_a^2)\mathbb E(k_a^2)=1$ The difference is one. The difference between the coordinates is zero. $d$ The difference of the item added is $d$ divided by $\sqrt d$ This is a basis for calculating the usual scale when the difference returns to one.

These independent and variable conditions are not the law that the training network always meets. For example, when queries and keys come from the same input, or even the two, the dots contain squares, the mean value is no longer the zeros assumed above. Zooming is still our fixed architecture agreement, but “The difference is one.”’s argument can only be set up accurately in a declared random model. It does not guarantee that the softmax will never concentrate or that the training will not spread; initial distribution, standardization and learning rates will also change numerical behaviour.

Example 1: A Complete Forward Pass with Two Positions and Two-Dimensional Values

Take $X$ A matrix of second-order units, $d=r=2$ Set. $a=\sqrt2\ln3$ ， $W_Q=aI$ ， $W_K=I$ ， $W_V=\operatorname{diag}(2,4)$ Because... $X$ is the matrix of units, $Q=aI$ ， $K=I$ ， $V$ Two behaviors. $[2,0]$ 、 $[0,4]$ 。 $QK^{\mathsf T}$ Still is $aI$, divided by $\sqrt2$ ， $S$ Two behaviors. $[\ln3,0]$ and $[0,\ln3]$. The unit matrix is a deliberately selected manual input and is not embedded in the actual language model that you must use.

The first line index is [3,1] Total four, weight. [3/4,1/4] Second act [1/4,3/4]. When reading the first line, three-quarters times [2,0] Plus one quarter. [0,4] Okay. [3/2,1]; second line of achievement [1/2,3]. Select a metric probe J=O First row 1-D+ 2×O Second line, second dimension. Got it. J=3/2+6 = 15/2 The probe is designed to give a clear purpose to the subsequent reverse, not to treat this as a language model loss.

There are two different types of check forwards. Check the non-negative lines and lines of A on a line-by-line basis, confirming that they are in the direction of unimonization. Then combo borders with a contour: because each row has a weightless weight and is equal to one, the first dimension of O must be between zero and two dimensions of two vectors, and the second dimension between zero and four. The output of [1.5,1] and [0.5,3] are satisfied. This range check is different from recombination and reveals errors that can be avoided or directly weigh fractions.

The boundary of the contour is bound only to read values and this layer of reading output. The value itself has been entered by W_V projection, not necessarily in the same direction as the original character vector; the output projection for the next chapter will be re-lined again. Therefore, “The weight of a position is greater for an input” cannot be interpreted directly as a causal contribution to the final forecast with the same proportion. To test the impact, it can be clearly disturbed by fixing other inputs and tracking all subsequent branches; attention re-entry can be seen in the middle but cannot skip the full calculation chain.

3. Backpropagating Step by Step from AV

Use $dV$ 、 $dA$ Waiting for short expression $J$ The gradient of the corresponding matrix does not represent a time differential. $O_{i\ell}=\sum_jA_{ij}V_{j\ell}$. Set a value coordinate $V_{j\ell}$, it'll be searched by all $i$ Read, so $dV_{j\ell}=\sum_iA_{ij}G_{i\ell}$. Fixed weights $A_{ij}$, which works in all dimensions of the vector, so $dA_{ij}=\sum_\ell G_{i\ell}V_{j\ell}$. The matrix is... $dV=A^{\mathsf T}G$ ， $dA=GV^{\mathsf T}$. Each search corresponds to all paths that pass through the variable in the calculation chart.

You can't just direct the molecules of the softmax. You can't skip the index for the same line. $A_j=e^{S_j}/Z$ ， $Z=\sum_ke^{S_k}$ Fixed. $S_k$, in the commercial code, the molecular item is $\delta_{jk}e^{S_j}$, the denominator guide is $e^{S_k}$. So the number of guides is $[\delta_{jk}e^{S_j}Z-e^{S_j}e^{S_k}]/Z^2$ Get it sorted. $A_j(\delta_{jk}-A_k)$ 。 $\delta_{jk}$ This output is called a barber matrix for all inputs.

Insulation: the weighted gradient in the first line is ( 3/4) × 2+(1/4) × 0=3/2, so dS line 1 [(3/4)(2−3/2),(1/4)(0−3/2)] = [3/8,−3/8]. The mean value in the second row is ( 1/4) × 0+(3/4) × 8=6 So the second act... [−3/2,3/2]. The gradient and zero per row corresponds to the flat shift that the whole row fractions and the constant weight do not change. Only the weights are kept. A dA will miss the denominator path, rows and usually no longer zero.

Expand fractions again $S_{ij}=\sum_aQ_{ia}K_{ja}/\sqrt d$ Fixed. $Q_{ia}$ All of them. $j$ ♪ To seek peace ♪ $dQ_{ia}=\sum_jdS_{ij}K_{ja}/\sqrt d$;fixed $K_{ja}$, all queries $i$ ♪ To seek peace ♪ $dK_{ja}=\sum_idS_{ij}Q_{ia}/\sqrt d$ So... $dQ=dS\,K/\sqrt d$ ， $dK=dS^{\mathsf T}Q/\sqrt d$. This example $K=I$ # So, why don't you # $dQ=dS/\sqrt2$ ； $Q=\sqrt2\ln3\,I$ # So, why don't you # $dK=\ln3\,dS^{\mathsf T}$. The numerical program will score all coordinates, not just the easiest one to draw.

And then back to the project. $dW_Q=X^{\mathsf T}dQ$ And the two other ones deserve it. $dW_K$ and $dW_V$ The same. $X$ Three branches each. $dX=dQW_Q^{\mathsf T}+dKW_K^{\mathsf T}+dVW_V^{\mathsf T}$. They must be added together and cannot cover the first two at the last value. The loss is then returned to the input vector through attention; only later will these gradients be fully trained if they are combined with real cross-enthon, all layers and optimizers.

Partially inverted Q, K, and V separately, not assuming that they are statistically independent. The calculus allows for first study of the changes in the function on three self-variant coordinates and then a chain rule to retake the common input. Statistical independence is a random model condition in the scalable motive, and the coordinates bias guideline is a computational breakdown, which is not the same concept. If the differential check was to change Q, then X recalculated three at the same time and compared to local dQs, the two different functions were compared and the result was not naturally consistent.

Example 2: Identical Keys Prevent the Query from Discriminating

Keep query and value, and change only two key lines to [1,1]. For any fixed query, it is equal to the points of the two rows, so the weight of the softmax is required [1/2,1/2], both lines output [1,2] The probe. J=1+2 × 2=5. The query does not distinguish between the two keys, so only changes around this point Q, the output remains unchanged. This example of degradation calculates the limits of the matching mechanism, which cannot be used “The model will learn the key itself.” Skip.

The opposite can also be reconciled with the above conclusions. dA remains [[2,0], [0,8], with an average value of one and four, respectively, and therefore two actions, [1/2,−1/2], [−2,2]. Each line is offset by the same lines, K, which is zero. But dK is not necessarily zero, because each row disturbs the same condition. Zero query gradients and non-zero-key gradients are not contradictory: they should be independent. It is wrong to set all gradients zero.

It can also be independently verified in a two-position special case. Set a row with a weight of [a,1−a] and an upper weight gradient of [u,v]. The first fraction should be a(1−a) (u−v)) and the second the opposite number. The competition for both weights comes from the common denominator: raising the first score not only raises the first weight but reduces the second weight. When u=v, reading the target is not sensitive to its allocation, the two guides are zero. If looking at softmax molecules alone, it will wrongly produce non-zero updates.

4. Independent Finite Differences and Fault Controls

When freezing other inputs, the center differentials add a coordinates h to reduce the loss by 2h. The program uses Q, K, V as an independently disturbed physical matrix to test the local functions just deduced; then takes three additional projectors back to the common X and checks dX to detect missing branches. Using double accuracy float points and two non-sync lengths, the error threshold is fixed in advance. The difference only provides this numerical evidence and interprets the reason why extrapolates still assume the general formula.

import math
import copy

def transpose(A): return [list(row) for row in zip(*A)]
def mm(A,B):
    return [[sum(a*b for a,b in zip(row,col)) for col in zip(*B)] for row in A]
def scaled(A,c): return [[c*x for x in row] for row in A]
def softmax(z):
    m=max(z); e=[math.exp(v-m) for v in z]
    return [v/sum(e) for v in e]
def forward(Q,K,V):
    S=scaled(mm(Q,transpose(K)),1/math.sqrt(len(Q[0])))
    A=[softmax(row) for row in S]
    return mm(A,V),A
Q_test=[[1.,2.],[0.,1.]]; K_test=[[1.,0.],[2.,1.]]
assert mm(Q_test,transpose(K_test))==[[1.,4.],[0.,1.]]
assert mm(Q_test,K_test)!=mm(Q_test,transpose(K_test))
G=[[1.,0.],[0.,2.]]
def objective(Q,K,V):
    O,_=forward(Q,K,V)
    return sum(x*g for row,gr in zip(O,G) for x,g in zip(row,gr))
def backward(Q,K,V):
    O,A=forward(Q,K,V)
    dV=mm(transpose(A),G); dA=mm(G,transpose(V)); dS=[]
    for i,row in enumerate(A):
        mean=sum(a*g for a,g in zip(row,dA[i]))
        dS.append([A[i][j]*(dA[i][j]-mean) for j in range(len(row))])
    assert all(abs(sum(row))<1e-12 for row in dS)
    dQ=scaled(mm(dS,K),1/math.sqrt(len(Q[0])))
    dK=scaled(mm(transpose(dS),Q),1/math.sqrt(len(Q[0])))
    return dQ,dK,dV

a=math.sqrt(2)*math.log(3)
Q=[[a,0.],[0.,a]]; K=[[1.,0.],[0.,1.]]; V=[[2.,0.],[0.,4.]]
O,A=forward(Q,K,V)
assert max(abs(A[i][j]-[[.75,.25],[.25,.75]][i][j]) for i in range(2) for j in range(2))<1e-12
assert abs(objective(Q,K,V)-7.5)<1e-12
analytic=backward(Q,K,V); fd_errors=[]
for h in (1e-5,1e-6):
    for field in range(3):
        error=0.
        for i in range(2):
            for j in range(2):
                plus=copy.deepcopy([Q,K,V]); minus=copy.deepcopy([Q,K,V])
                plus[field][i][j]+=h; minus[field][i][j]-=h
                numeric=(objective(*plus)-objective(*minus))/(2*h)
                error=max(error,abs(numeric-analytic[field][i][j]))
        fd_errors.append(error)
assert max(fd_errors)<1e-7

X=[[1.,0.],[0.,1.]]; Wq=copy.deepcopy(Q); Wk=copy.deepcopy(K); Wv=copy.deepcopy(V)
def projected_loss(x): return objective(mm(x,Wq),mm(x,Wk),mm(x,Wv))
dQ,dK,dV=analytic
branches=[mm(dQ,transpose(Wq)),mm(dK,transpose(Wk)),mm(dV,transpose(Wv))]
dX=[[sum(branch[i][j] for branch in branches) for j in range(2)] for i in range(2)]
x_error=0.; h=1e-6
for i in range(2):
    for j in range(2):
        plus=copy.deepcopy(X); minus=copy.deepcopy(X)
        plus[i][j]+=h; minus[i][j]-=h
        numeric=(projected_loss(plus)-projected_loss(minus))/(2*h)
        x_error=max(x_error,abs(numeric-dX[i][j]))
assert x_error<1e-7
same_keys=[[1.,1.],[1.,1.]]
O_same,A_same=forward(Q,same_keys,V)
dQ_same,_,_=backward(Q,same_keys,V)
assert all(abs(x)<1e-12 for row in dQ_same for x in row)
assert O_same==[[1.,2.],[1.,2.]]
print("attention_weights",A)
print("output",O,"probe_loss",objective(Q,K,V))
print("dQ",dQ,"dK",dK,"dV",dV)
print("QKV_finite_difference_max_error",max(fd_errors))
print("shared_X_finite_difference_max_error",x_error)
print("same_key_output",O_same,"same_key_dQ",dQ_same)
print("all_attention_checks_passed")

In the normal dense realization of the scale multiplied by the fee, the QK conversion will require a T2d volume operation, AV will require a T2r storage weight, and T2 numbers are required for storage. This is the level of volume under the clearly calculated model, not the time-consuming prediction of a device; the projection itself will be extra chargeable. The small reference function deliberately maintains the intermediate matrix to check that the same values, gradients and outcome tests can be reused at a later acceleration, and the same value, gradient and result test cannot be omitted at a faster speed than the equivalent price.

The upstream gradient given by the probe is very thin in this case to allow each step to be repeated. In a true language model, cross-alkanes form an additional upstream gradient through the output matrix, normalization and other layers, and the local reverse formula of attention does not change as a result. This gradient must be calculated from real loss. It cannot be fixed in this chapter for G to be inserted into model training. The reusable properties of local algorithms are derived from input output consistent with guidance definitions, not from artificial probes that can cause the target to fall.

The differential input also avoids the weight of almost complete saturation on a value. If an index has spilled, the output after a minor disturbance may be completely static on the machine, the parsing and the value are close to zero, but it is difficult to detect errors. Use this chapter as an example of a clear mix of weights, and then set extreme fractions to check stability. One check is for a chain of conductivity, and one is for a limited precision boundary, not because the former omits the latter.

5. Progressive Exercises and Solutions

# causal_attention
def causal_attention(query, key, value, query_offset=0, dropout=None):
    raise NotImplementedError('attention')

# 第3章仅实现无缓存注意力；第10章在同一函数增加offset。
def causal_attention(query,key,value,query_offset=0,dropout=None):
    if query_offset != 0:
        raise NotImplementedError('第10章再处理历史位置')
    scores = query @ key.transpose(-2,-1) / math.sqrt(query.shape[-1])
    future = torch.arange(key.shape[-2],device=key.device)[None,:] > torch.arange(query.shape[-2],device=query.device)[:,None]
    probabilities = torch.softmax(scores.masked_fill(future,float('-inf')),dim=-1)
    weights = probabilities if dropout is None else dropout(probabilities)
    return weights @ value, probabilities

This Lesson's Project Contribution and the Next Dependencies

There is now a complete forward, dissecting and independent differential reference for one head of attention, which includes a gradient that adds to the three project sharing input. All locations in this chapter are visible, so it cannot be used to claim that the next token model is cause and effect. Chapter IV will introduce causality and filling permission, multiple heading and output projection into the same calculation chain, as well as learning absolute location for the main project, checking the entire prefix interface. The full parameter training that follows will use these actual images and cannot be frozen until they are fully trained.