From Residual Connections to a Complete Decoder Block

What This Lesson Builds and What It Assumes

The first four chapters already give a hyphen, embedding, location, and causally multi-headed focus. This chapter links attention to truly stackable decoder blocks. At the end of the chapter, you should be able to explain the shape, specific figures for uniformization, the gradient circuits for residues, and the number of parameters of the whole set of models, item by item. If average values, squares and arrays are calculated and guided by one function; if the multi-chain approach is described in local variations when needed. This does not prove that any deep network will be trained.

The amount agreed to be $B$, the current sequence length is $T$, hide width as $d$, first number $h$, the width of each head is $d_h=d/h$, front feed hides the width as $f$. These are the positive integer numbers, and $h$ Integer $d$ block input. $X$ It's shape. $[B,T,d]$ real; lower case below $x$ This indicates the length of one of the locations. $d$ vectors. Statistics are calculated along the final feature axis, neither over time nor over batches. All activations are in an empty schematic number.

This item freezes to pre-LN: each sub layer is first recalculated and then returned to the original input. There are two separate LayerNom units for each block, with a non-selective and biased linear Q/K/V/O map for attention. The precise GELU is activated, and the single stabilization constant is fixed. $\varepsilon=10^{-5}$. This chapter schilling dropout is zero, so the function is determined in a forward direction; with dropout on, constant equivalence against values should be fixed randomly or under evaluation mode.

Position-Wise LayerNorm: Mean, Population Variance, and Stability Constant

Definitions $\mu$ For all of that position $d$ Arithmetic average of individual characteristics; defining centralised coordinates $c_i=x_i-\mu$; defining population patterns variance $v$ To centralize squares and divide them by $d$. This is not a statistical task to estimate an unknown overall variance from the sample, so it is not divided $d-1$ is modified without prejudice. $r=\sqrt{v+\varepsilon}$, standardize the value $z_i=c_i/r$ 。

Parameters $\gamma_i$ and $\beta_i$ It is the training-by-characterization scaling and deflection, with the default starting values of one and zero, respectively; the same set of parameters is shared in all batches and locations, but the statistical values are recalculated at each location. They do not press each position into the same vector, but they adjust the standardized coordinates. $\gamma=1,\beta=0$, the average of the results of the reclassification is zero because of the centralization coordinates and zero; its population variance is zero $v/(v+\varepsilon)$ Not the strict one.

The final conclusion can be directly inserted into the verification: standardized squares and divides $d$ is equal to the original centralised squares and divided by $d(v+\varepsilon)$ The molecules are right. $v$ ♪ When... $v$ It's close to one when it's much larger than a stable constant; $v=0$ , which is zero. Here comes the conditional value close, which cannot be reduced to a permanent unit variance. The average value of zero is no longer necessary after the analog parameter is not the default.

The peer check is an independent algebra test: one constant for all features $a$, add the new mean $a$, so the centralised coordinates remain the same, the differences are the same, and the output is the same. The positive scale does not replicate this conclusion: $x$ Multiply $s>0$, the denominator becomes $\sqrt{s^2v+\varepsilon}$, the fixed stabilization constant has not been scaled down accordingly, so it is usually only approximate. If the entire length is consolidated over time, the future hyphen will enter the statistics, undermining the causal contract of the project.

Building a Block from Two Residual Branches

Remember. $A$ It keeps the input for a multi-purpose map of causes and consequences, as defined in the previous chapter. $[B,T,d]$ shapes;keeping $F$ This is a pre-map for the next definition. The following two indicators indicate that the parameter of independent uniformization is not repeated for the same output. The first step is to generate an increment of attention by using the original input and to get an intermediate state. $U$; second step based on $U$ Pre-generation feed increment, again $U$ Added to get block output $Y$ 。

This order must be implemented on a case-by-case basis. The second one is read in the same way as the one that has been added to the increase of attention. $U$ Not at first. $X$; otherwise you're making another calculation. The two ends must be the same shape and cannot rely on radio to make a mistake. $[B,1,d]$ Increment is copied to the full. Attention is responsible for exchanging information between legitimate past positions, while the previous feed deals with the characteristics of each location separately and the weights are shared over time.

For the definition of pre-dips, a mathematical vector agreement is used to: $W_1$ Shape As $[d,f]$ ， $b_1$ Yes. $[f]$ ， $W_2$ Yes. $[f,d]$ ， $b_2$ Yes. $[d]$. Count first. $a=xW_1+b_1$, non-linear by coordinates, then multiplied $W_2$ Back to Width $d$. The actual PyTorch Linear storage matrix is the output width in front of the output, so the call times its conversion; the mathematical matrix cannot be confused with the stored shape.

Here import the error function as a special function provided by the standard library, defined as the equivalent $\operatorname{erf}(u)=2\int_0^u e^{-s^2}\,ds/\sqrt\pi$ So... $\Phi$ It is the probability that the sum will be in its current position, not a random coin toss; there is no sampling of the exact GELU forward. Negative input usually gets a small negative output, rather than a direct zero change like ReLU. If there is no non-linear layer between the two layers, the matrix multiplier can be merged into a layer of analogue map, and the extension repression does not increase the non-linear expression of this function.

Example 1: A Feed-forward Residual with a Nonzero Contribution

Take a position. $d=f=2$, Enter $x=(1,3)$, both of which are the default values. In order to separate the pre-dips and project zero attention output, there is zero concentration increase. $u=x$. Preset $W_1$ as a matrix of units, $W_2$ This is an example of a mechanism, not a recommendation to project zero when training is conducted.

The second set of homogenizations is calculated: the average is two, the centre is negative one and one, and the population is one. $a=1/\sqrt{1.00001}$ Got it. $z=(-a,a)$ of which $a\approx0.999995$. Using standard normal symmetry, $\Phi(-a)=1-\Phi(a)$;in values $g(-a)\approx-0.1586557$ ， $g(a)\approx0.8413393$. Right multiplied by half of the unit matrix, with an increase of approximately $(-0.0793278,0.4206696)$;back input, block output approximately $(0.9206722,3.4206696)$ 。

The output does not require an overall average value of zero, because reclassification takes place only at the branch entrance and the main road is kept in its original scale. $g(a)-g(-a)=a$, so the second coordinates of the increment minus the first coordinates should be the right one. $a/2$; the value is about 0:49,99975. This constant equal is a different check path from the direct error function, where the positive value is still at GELU and the negative value is cut to zero; however, if the whole function is replaced by ReLU, it will still be able to do so, and a specific value must be checked.

Example 2: Failure Controls for Variance, Position, and Stability

Still one or three vectors, if you miswrite the dichotomy $d-1$, with a two-point margin, standardization becomes negative at about 0:071 and positive at 0771. The shape is exactly right, but the values are about 30%; subsequent activations and residues will change. The correct difference should be one from “Square and di divided feature two”, which is not judged successful by the symmetrical appearance of the output. Symmetry can only check part of the error and cannot recognize the error in the denominator.

Retrieval Constant Vector $(2,2)$. The average value is two, the variance is zero; both coordinates are standardized when maintaining a stabilization constant are zero and the output equals $\beta$. If the stable constant is deleted, the denominator is zero, and the formula is not defined; this is not a negligible small rounding error. Any common equalisation of the constant vector should still be given the same output. The code calculates the correct realization, the error denominator and the zero variant boundary, so that one success does not obscure the other two types of malfunctions.

import math
EPS = 1e-5
def layer_norm(x, eps=EPS, sample_variance=False):
    mean = sum(x) / len(x)
    divisor = len(x)-1 if sample_variance else len(x)
    variance = sum((v-mean)**2 for v in x) / divisor
    return [(v-mean)/math.sqrt(variance+eps) for v in x]
def gelu(x):
    return 0.5*x*(1+math.erf(x/math.sqrt(2)))
z = layer_norm([1.0, 3.0])
y = [x+0.5*gelu(v) for x,v in zip([1.0,3.0],z)]
assert abs((gelu(z[1])-gelu(z[0]))-z[1]) < 1e-12
assert layer_norm([2.0,2.0]) == [0.0,0.0]
shifted = layer_norm([11.0,13.0])
assert max(abs(a-b) for a,b in zip(z,shifted)) < 1e-12
expected_var = 1.0/(1.0+EPS)
assert abs(sum(v*v for v in z)/2-expected_var) < 1e-12
print('preLN residual', y)
print('wrong denominator', layer_norm([1.0,3.0],sample_variance=True))
try:
    layer_norm([2.0,2.0],eps=0.0)
except ZeroDivisionError:
    print('expected failure: zero variance without eps')

Residual Connections Provide a Direct Gradient Path Without Guaranteeing Learning

Equating a position or entire length to a column vector, only for discussion of the guide. Write for any debris sublayer $y=x+R(x)$ of which $R$ is a de minimis function; $J_R$ Number of $i,j$ Item is output $R_i$ Enter $x_j$ . The bias. $x_i$ Yeah. $x_j$ , the guide is the corresponding item of the unit matrix. $J_y=I+J_R$. This constant item doesn't need to be regularized, focused and activated through the branch.

The first is the gradient that returns directly from the main road of the debris, and the second comes from the branch. They add up instead of opting for a reservation, so they can offset each other. $R(x)=-x$, which is negative, with a constant zero output and a total of zero. So “There's always a way.” is the exact structure and “The gradient never disappears.” is not rolled out by it. Multiple sublayers are stacked with their respective accompaniment matrices, and single layer additions cannot be mistaken as an entire net gradient.

If Post-LN is used, the form becomes $y=\operatorname{LN}(x+R(x))$, the guide is a unicoded range. $(I+J_R)$ This algebraic difference suggests that the two structures are different and do not necessarily constitute a better theorizing pre-LN under an arbitrary task. When all branches of this project are set aside for zero, the pre-LN block should be kept strictly in the input; if the code still exports a standardized vector, it is likely to move the unit back to the addup. The final LayerNom at the end of the model is another layer and does not belong to the constant test.

Shapes and Parameters: Counting Every Learnable Scalar

Another shape exercise: if the batch is two, three, four, two, eight, and the front bow is two times three times four; the search key is two times two times two times three times two; the attention weight is two times two times two times three times three; and then two times three times four is returned to two times three after the project. The first feed is two times three times eight, and the second linear layer is restored two times three times four. Any step of an accidental exchange of lengths and feature axes may be exposed only when they are not equal, so the test is that three and four are deliberately used.

The parameter ledger must also be separated from the active account. When the sequence becomes longer, the current activation and attention matrix grows, but the linear layer matrix does not grow with the unicoded parameters; rather, expanding the available table of positions increases the registered parameters, even if only one word is entered this time. Depending on whether the parameter is eligible for a gradient, then whether the batch gradient is not zero, and finally whether the optimizer actually holds them, there are three different questions. The latter two chapters will be tested separately and cannot skip because the total numbers match.

The four images of attention are all. $d^2$ weights per unit, total $4d^2$; wrap Q/K/V into one $[3d,d]$ The storage matrix does not change the quantity. $df+fd=2df$, the two-sided is $f+d$. Two lengths for each unit $d$ , two sets of $4d$. Add each piece $4d^2+2df+f+5d$. The header only changes the delineation of these widths, fixed $d$ The total number of parameters will not be changed when the structure remains unchanged.

In order to extend the account book to the entire Chapter 7 machine, another word-size is defined. $V$ Maximum number of locations $C$ Number of blocks $N$. The word embedded and the output of unstrapped weights $Vd$, learning absolute position embedded in $Cd$ And eventually it's down to the same thing. $2d$. The current actual input length $T$ Do not change the number of parameters in the registered list, only $T\le C$. The parameters are not shared among each other; if the same object is mistakenly returned, the number of parameters will be small.

Take $V=32,C=64,N=2,d=64,h=4,f=256$ The four pieces are projected at 16,384, with a pre-granting weight of 32,768 and a pre-release preference of 320, which is two,566 and a total of 49,728 and 56. Two,994,456; the word is embedded in the head with a head of 40,966 and the position is 4,966, the position of the final LN 128, for a total of 10,777,766. This figure is an accurate calculation of the configuration, not a visible byte, nor does it include the state of the optimifier.

def parameter_count(V,C,N,d,f):
    per_block = 4*d*d + 2*d*f + (f+d) + 4*d
    return 2*V*d + C*d + N*per_block + 2*d
assert parameter_count(32,64,2,64,256) == 107776
assert parameter_count(10,6,2,4,8) == 424
print(parameter_count(32,64,2,64,256))

Project Contributions and Later Dependencies

The present chapter reads: hero_transformer/model.py Add two sets of homogenization, one by one. FFN with pre- LN Fragmentation blocks, using the pre-brand causality focus as a certified submodule access. The submission should show that the input output is identical, the two branches are zero-time-long, etc., that changing the future dictional characters does not affect past output, and that the parameter books are consistent with the registered parameters four results. The causality check moves from the attention layer to the block as a whole, to prevent the error of the single axis or a prior feedback action over time from being re-introverted.

from hero_transformer.model import Config, TransformerLM
cfg = Config(vocab_size=32,block_size=64,d_model=64,n_heads=4,
             n_layers=2,d_ff=256,dropout=0.0)
model = TransformerLM(cfg)
actual = sum(p.numel() for p in model.parameters())
assert actual == 107776, actual
print('registered parameters', actual)

Chapter VI will be pushed back along the same calculation chart in this chapter to the gradients of LayerNom, GELU, linear layers and residues, so that the axes, stabilization constants and activation agreements cannot be replaced temporarily. Chapter VII repeats calls for multiple non-shared blocks, joins finalLN and head, with full logits and losses. The forward and parameter checks do not indicate that each parameter is connected to the gradient; that is the different nature of the latter two chapters that need to continue to be validated.

# TransformerBlock.forward（替换已提供类的此方法）
def forward(self, hidden, past_key_value=None, use_cache=False):
        raise NotImplementedError('block')

# TransformerBlock.forward（替换已提供类的此方法）
def forward(self, hidden, past_key_value=None, use_cache=False):
        attended, present = self.attention(
            self.ln_attention(hidden), past_key_value, use_cache
        )
        hidden = hidden + attended
        hidden = hidden + self.feedforward(self.ln_feedforward(hidden))
        return hidden, present