Architecture and Cost: Toward Modern Implementations

Classify the Change: Model Function or Execution Order

The first ten chapters have developed a complete decoding model that can be trained, restored, evaluated and generated. This chapter returns the usual extensions to its computational diagrams, rather than listing abbreviations. The objective is to determine whether a change has changed the expression, parameter sharing, activation function, storage, or operation schedule. First, it is modified to model matrix multipliers, paradigms, attention, caches, and underlying cosine persistence. When it comes to triangulation, the matrix and algebra relationships used are written; and the test results of hardware optimization are not taken into account as mathematical algorithms.

The extension of this chapter is a comparative study, where the default teaching project retains the absolute learning position, common heading, LayerNom, and GELU. The insertion of RoPE or GQA directly into the old checkpoint usually changes the scores given by the model, requiring corresponding model definitions and training or conversion protocols; they are not loss-free. On the contrary, the block calculation of the same softmax attention can maintain the same function under real-digit algorithms, but the floating point error, reverse transmission and performance can still be accepted separately. “More modern” Misunderstanding. “It can be replaced at any time and must be better.” 。

Track Parameters, Activations, Caches, and Operations Separately

Follow $B$ for batches, $T$ is the current length, $d$ for width, $f$ for the first time, $N$ is the number of layers $V$ As a phrase size, $C$ The total number of main project parameters is available in chapter V. $P=2Vd+Cd+N(4d^2+2df+f+5d)+2d$. This number corresponds to the length of this input $T$ Not relevant; but attention probability mass shapes $[B,h,T,T]$, which grows in squares with length. One is the lasting weight, one is the intermediate data during operation and cannot be added to an ununited “Model Size”.

It also saves gradients and optimists during training. If weights, gradients and AdamW's two totems are kept in float 32, only four and four plus eight for each of these four sets, 16 bytes. It does not take into account activation, temporary storage, distributors, and data batches. Mixed accuracy may save the weight of sovereignty extra and the frame may use different states; therefore, 16 bytes is the ledger for achieving the assumptions mentioned, which cannot be applied unconditionally to each training system, let alone direct comparison of reasoning-only memory with training memory.

For estimating the arrays by operation, each FLOP is counted separately by a single multiplying method and an addition. Q/K/V/O Total of four projectors $8BTd^2$;total of two previous projects $4BTdf$;full dense fractions and values aggregate $4BT^2d$ This estimate does not explicitly exclude softmax, GELU, standardization, bias and access. The main project is a complete matrix to be multiplied and masked, so covering the future does not mean that triangulation has been saved. The result-specific kernels can change constants and cannot be automatically pushed by a mask.

Take a scale comparison: when width and forward width are fixed, the length is doubled, the projector is about doubled, and the attention matrix is about four times more; if the volume is doubled only, both are about doubled. Actual time may be affected by the visible bandwidth, kernel activation, parallelism and cache, without these multiplyings. The book is used to present bottleneck assumptions, time-counting tests; if they are inconsistent, assumptions should be checked instead of directly writing the formula into milliseconds on the device.

RoPE: Relative Position in the Query–Key Dot Product

Look first at two-dimensional coordinates. Define the rotation matrix. $R(\alpha)$, put the query in position $m$ Rotate $m\omega$, key in position $n$ Rotate $n\omega$ ； $\omega$ is the fixed angular frequency for this coordinate. The rotation matrix meets the rotation equals the reverse matrix, so that the two points built up after the rotation depend only on the difference in position. $n-m$ The corresponding relative rotation. Here “Relative” is removed from a clear algebra instead of placing a name on the location.

Multiply this and you can check the symbols directly. $R(a)^{\mathsf T}R(b)$ the last left entry is $\cos a\cos b+\sin a\sin b=\cos(b-a)$, the upper right is $-\cos a\sin b+\sin a\cos b=-\sin(b-a)$; the other two are positive $\sin(b-a)$ and $\cos(b-a)$ So the whole matrix is... $R(b-a)$ And again. $a=m\omega,b=n\omega$ , get the top. When a different rotation is agreed, the sign of the difference may also change and cannot be separated from the matrix definition by a positive or negative number.

When each head is even in width, two pairs of coordinates, the first. $i$ Use frequency $\omega_i=10000^{-2i/d_h}$ of which $i=0,\ldots,d_h/2-1$. The frequency table is a model design agreement, not the only law that comes from the attention formula. Usually Q and K are rotated without the same rotation. It changes the way the point of entry is expressed compared to the position vector directly added to the main project; even if the two versions of parameters are close, they are not the same function.

If the query and the key are in the same position, they are rotated to the same point as if they were not rotated, which is the first check; if the two positions are added to the same integer number, the same position is constant, and the point is maintained, which is the second independent check. If the long position is still capable of calculating the sine cosine, the formula is defined and the projected quality is maintained outside the training length. “I can figure it out.” Write “Infinite Context” 。

import math
def rotate(pair, angle):
    x,y = pair
    return [math.cos(angle)*x-math.sin(angle)*y, math.sin(angle)*x+math.cos(angle)*y]
def dot(a,b): return sum(x*y for x,y in zip(a,b))
q,k,omega = [1.0,2.0],[3.0,-1.0],.17
m,n = 2,7
observed = dot(rotate(q,m*omega),rotate(k,n*omega))
relative = dot(q,rotate(k,(n-m)*omega))
shifted = dot(rotate(q,(m+11)*omega),rotate(k,(n+11)*omega))
assert abs(observed-relative)<1e-12 and abs(observed-shifted)<1e-12
assert abs(dot(rotate(q,.8),rotate(k,.8))-dot(q,k))<1e-12
print({'relative_dot':observed,'translation_difference':abs(observed-shifted)})

GQA: Which Heads Share Keys and Values?

There's a lot of them. $h_q$ A query header, and $d=h_qd_h$, with the same number of K/V heads; group queries are less focused $h_{kv}$ A K/V header, and each group is asked to share a K/V head. $h_q$ Can be. $h_{kv}$ Discreet, the number of queries per group is $g=h_q/h_{kv}$ No, no, no. $i$ Queryhead Use Number 1 $\lfloor i/g\rfloor$ A K/V header. Query can still be different, not all calculations are combined between the heads.

For example, eight query heads, two K/V headers, each with four queries, shared map zero to three zero group, four to seven group 1, and the cache is one quarter of the eight K/V head. If two sets of K/V are actually copied into eight groups before being realized, the mass interface may be “Looks like a match.”, but storage savings have been lost. The actual distribution needs to be viewed separately for logical broadcasting, ad hoc roll-out, and lasting caches.

Projection parameters are extra. Q is still wide. $d$ Map To $h_qd_h=d$ ♪ O keeps it ♪ $d$ Present. $d$ Both of them. $2d^2$ weights; K and V, respectively $d$ Map To $h_{kv}d_h$ Total, two $2dh_{kv}d_h$ So the total number of unbiased attention projection $2d^2+2dh_{kv}d_h$ Take it. $d=128,h_q=8,h_{kv}=2,d_h=16$ Q/O is 32,776, K/V is 81,912, for a total of 40,960; the usual figure is 65,536 and the parameter is five-eighths and cannot be mixed with the cache's one-quarter.

Share K/V It also changed the parameter: the original self-speech key value for each head is now bound within the group. Even if the old weight is averaged as the initial value, it is not guaranteed to keep the original output on all inputs. The main paper examines the conversion method for continuing training from an already existing check point; this course only leads to shared maps and resource books, which are not falsely described as having been completed. If the project is expanded, it should be written first. K/V Shape and map tests, and then separate quality versus budget.

RMSNorm and Gated Feed-forward Networks

RMSNom uses the square root as a measure: $r=\sqrt{d^{-1}\sum_i x_i^2+\varepsilon}$, output is usually $\gamma_i x_i/r$. It is not subtracted from the mean value, and there are usually no deviation parameters in the main item LayerNom. $x=(1,3)$, the square root of the small stabilizer $\sqrt5$ Standardized approximately $(0.447,1.342)$; LayerNom is about $(-1,1)$. Specific values indicate that they are not the same function.

After 10 additions to the entire vector, the centralised coordinates of LayerNorm are maintained, and the RMSNom denominators and molecules change and generally do not maintain the original output. The positive scale is not different when the stability items are ignored or treated accordingly; fixed stabilization constants make this a conditional approximation. Reduction in mean calculations may have engineering benefits, but it is not possible for a reduction to be made directly by an agreement to claim that the arbitrary equipment is necessarily faster and the speed still needs to be measured.

The door handles two broad-based projections multiplied by coordinates. $\operatorname{SiLU}(u)=u/(1+e^{-u})$ ， $F(x)=[\operatorname{SiLU}(xW_g)\odot(xW_u)]W_d$; the first two array shapes are $[d,f]$, and finally... $[f,d]$ ， $\odot$ represents the same position multiplied. The door-controlled branch changes the range of another route according to the input, instead of simply replacing the GELU with a function name.

The eight-eighths here are only the proportion of width that is released under the same weight budget. Realization requires integer width, and may be aligned to hardware; the ledgers are recalculated when biased, changing output dimensions or sharing matrices. Compared GELU to door-controlled prepayment, parameters and calculations should be reported at the same time. The extra half weight cannot be used as a free mechanism improvement.

Online Softmax without Storing a Full Probability Row

Set a legal score for a query as $s_j$, the value vector is $v_j$. The final output is $\sum_j e^{s_j}v_j/\sum_j e^{s_j}$. To avoid spilling, the same maximum value can be subtracted. If you read fractions by block, the maximum value of the entire line is maintained without prior knowledge. $m$ The denominator after the scaling $a$ Unredified vector molecules after scaling $o$ Attention. $o$ It's not like it's been out of focus.

Read a new score, shilling $m^\prime$ The older molecules, denominators, are used to press $e^{-m}$ Zoom. Now we're going to have to change it. $e^{-m^\prime}$ Zoom, so we're all on board. $\alpha=e^{m-m^\prime}$;new block(s) directly multiplied each $e^{s_j-m^\prime}$ Add. So that all items are re-sized to the same scale and all blocks are processed before calculation $o/a$. This step did not lose the small probability item, and the goal of mathematics was still a dense focus.

Closes the reason with a loopless variable. Assumes read as $I$, save state meets $a=\sum_{i\in I}e^{s_i-m}$ 、 $o=\sum_{i\in I}e^{s_i-m}v_i$ Multiply $\alpha$ After that, every old item becomes $e^{s_i-m^\prime}$; new blocks use the same entries $m^\prime$ And then it's just overlayed. $I\cup\mathcal B$. The first leg is directly limited and initialized; a limited number of non-overlap blocks, after they are processed, covers all the key keys, then returns to the defined focus by dividing them by the denominator.

This also gives two rebuttable checks: if a piece is read over again, part of the key is counted twice, the weight system has changed; if the mask is disposed of at will after the maximum value of the block, the full mask may introduce an undefined value. The order of the block can be adjusted, but the combination, number of occurrences and legitimacy cannot be changed. The floating point add-up does not satisfy the precise combination, so the mathematical variable remains compatible with the numerical error record and cannot be compared to the actual machine simply by extrapolation.

import math
scores, values = [1000.,1001.,999.,1002.],[2.,-1.,3.,4.]
maximum = max(scores)
weights = [math.exp(s-maximum) for s in scores]
reference = sum(w*v for w,v in zip(weights,values))/sum(weights)
m,den,num = -math.inf,0.0,0.0
for begin,end in [(0,1),(1,3),(3,4)]:
    new_m = max(m,max(scores[begin:end]))
    rescale = 0.0 if m == -math.inf else math.exp(m-new_m)
    local = [math.exp(s-new_m) for s in scores[begin:end]]
    den = rescale*den+sum(local)
    num = rescale*num+sum(w*v for w,v in zip(local,values[begin:end]))
    m = new_m
observed = num/den
assert abs(observed-reference)<1e-12
print({'dense':reference,'blocked':observed})

FleshAttention uses fractions and online reclassification to reduce reading and writing between high-bandwidth visible memory and film storage, and avoids the obvious preservation of complete attention matrices. This does not mean that the calculation of each query key is deleted altogether, nor does it mean that attention becomes linear arithmetic complexity. The Python verification above shows only the homogenization mechanism, and does not achieve an kernel or measurement acceleration. If you change to integration inner core in a project, you should report speeds and numerical errors.

The Scope and Deliverables of Extended Experiments

This chapter delivers four separate tables: Parameters, Training Status, KV Cache and Matrix Multiply Estimation; Reruning the Rotation Point Cumulative and Block Softmax two independent experiments. Select an extension of a model function to clarify new parameters and training programmes; Choose a schedule optimization to maintain a target function to write numerical differences and timing methods. The default training results of the main project continue to use the original structure and cannot be presented here as a new version of the language model that has been trained and validated. Chapter XII will integrate all completed components into a runable final delivery.