Decoder Gradients and Resource Accounting

Task and Starting Point: Reusing Verified Components

This chapter provides a complete decoder to the data pipeline and confirms that “Training in full parameters” has a probative meaning in this project. Readers should have completed the deduction of attention, causality, exterior feedback, reclassification and reverse transmission during the Transformer battle; instead of adjusting a model interface to replace those certificates, the component is integrated into the gradient. This project lm_lab/decoder.py is a self-written local copy of the course, the source path being recorded separately from Hashi; the compressed package contains a copy and does not run cross catalog import or download weight. Readers who need to re-examine their attention should be able to rely on them before checking the general ledger of this chapter.

The project is fixed on one level, four head, hidden width twenty-four, pre-broadity forty-eight, window ceiling twenty-four; BPE actual termsheet two hundred ninety. The attention Q, K, V and the output project have no bias; the forward two linear layers have bias; each has two LayerNorms with analog parameters, and one last. Positions are located using the remote array embedding, each stand-alone window starting from zero; the non-linear feed is precise GELU; dropout sets zero here. The output header does not share weights with the input embedding. Only by listing these options can the parameter formula be accurately aligned to the file.

The Shape Chain from Integer Tokens to Vocabulary Logits

Enter the x shape B×T. The integer can only be zero to V−1. Checklist E is V×d, Location table P is C×d, initial hiding status is $H_{b,t}=E_{x_{b,t}}+P_t$ shape B×T xd. Each block is re-bludgeoned to do cause-and-effects attention, plus input, then re-bludgeoned to pre-divide and then re-loaded. d×V Output matrix gets B×T × V Logits. The whole chain does not feed the target to the same location in a hidden state; the target is used only at the end of the cross-enthon.

For a valid target, logits are p, softmax probabilities are p, real category one-hot. The gradient for cross-entropy to logits is p−y; multiple active positions are divided into batch numbers. If the last hidden row is h, the output matrix is W, the output matrix gradient is extra. $h^T(p-y)$, hide the line gradient $(p-y)W^T$. The latter continues to be normalized to all Blocks, and cannot detach before the output head. A check will add a gradient with multiple numbers to the same line; the embedding line without input number can temporarily be zero, but the entire parameter is still trained.

Impairment $u=h+f(h)$ gradient is the upstream gradient multiplied by $I+J_f$ of which $J_f$ It's an axieties for side roads. The residue provides a permanent path, and must be transmitted along the side paths. De-grading attention or pre-benders in no_grad will remove the silo gradient; checking only whether the general loss drop is undetectable because the output head can still fit with fixed features. Full parameter training requires that all expected parameters requires_grad be true, integrated into optimizer, receive a limited gradient, and have actual updates under reasonable test input; these are different assertions and should not be examined only for one of them.

Example 1: Counting All 19,320 Parameters

Use V for wordsheets, d for width, f for ex-broadness, C for maximum position, L for layers. Enter and output two matrices together $2Vd$, position table Cd. Four d×d matrices per layer of attention $4d^2$;first feed matrix $2df$,bias out of f+d; twice LayerNorm with scalding and shifting, 4d. Total per layer $4d^2+2df+f+5d$; final re-alignment plus 2d. No active or gradient is counted as parameters, and no four-head error is used as a complete d×d matrix.

Forewords to V=290, d=24, f=48, C=24, L=1: The two-word matrix matrix is 13920, position table 576, attention 2304, ex-bias matrix 2304, pre-bias 72, double-dimension 96, and then singled 48, total 19320. Independent inspection I is the sum of the Numel of the actual model.parameters must be equal; independent inspection II is a grouping of names, preventing many and less offsets. The first four times the coordinates of the splitd, with no extra four times; if d cannot be divided by the header, the configuration should be rejected directly.

Example 2: The Output Learns While Lower Layers Stay Frozen

Consider a simplified output of two characteristics, two classifications: h = [1,2], the output matrix in two columns [0.1,0.3] and [−0.2,0.4].logits are [0.7,0.6], the real category is Category One. Softmax first is about 0. 524979, the gradient for logits [−0.475021,0.475021]; the gradient for h is about - 0.142506 The second is about 0%. 047502. These non-zero values mean that if h comes from earlier training parameters, the chain approach should continue to spread. If the code puts h on constant, the output matrix still has a non-zero gradient, losss may still decline, but it does not mean lower learning.

The independent value check uses a central difference: each coordinates h is added to a small amount, the cross-entropy is recalculated and the difference between two losses is divided by twice the step and should be close to the solver gradient. This standard library example only verifies the local chain-style rules of the last projection, not the full Transformer training. The fully achieved acceptance is also run in project tests: 17 name parameter sheets are given a limited non-zero number, with attention to Q/K/V/O, the position table, the two layers of forward feedback and the combination. A specific parameter element of zero is not necessarily a failure, and an inspection cannot wrongly require that all elements be zero at every step.

import math
W=[[.1,-.2],[.3,.4]]; h=[1.,2.]
def loss(h):
    z=[sum(h[i]*W[i][j] for i in range(2)) for j in range(2)]
    a=max(z); return a+math.log(sum(math.exp(v-a) for v in z))-z[0]
z=[sum(h[i]*W[i][j] for i in range(2)) for j in range(2)]
a=max(z); probs=[math.exp(v-a) for v in z]; total=sum(probs); probs=[v/total for v in probs]
gz=[probs[0]-1,probs[1]]
gh=[sum(gz[j]*W[i][j] for j in range(2)) for i in range(2)]
fd=[]; eps=1e-5
for i in range(2):
    plus=h.copy(); minus=h.copy(); plus[i]+=eps; minus[i]-=eps
    fd.append((loss(plus)-loss(minus))/(2*eps))
assert max(abs(a-b) for a,b in zip(gh,fd))<1e-9
V,d,f,C,L=290,24,48,24,1
params=2*V*d+C*d+L*(4*d*d+2*d*f+f+5*d)+2*d
assert params==19320
print('logits',z,'hidden gradient',gh,'finite difference',fd,'parameters',params)

from lm_lab.data import read_corpus
from lm_lab.engine import build,update,parameter_count
split,_=read_corpus()
model,opt,tok,windows,sampler=build(split)
receipt=update(model,opt,windows,sampler)
assert sum(p.numel() for p in model.parameters())==parameter_count(model.config)==19320
assert set(receipt['gradient_tensor_norms'])==set(dict(model.named_parameters()))
print('trainable tensors',len(receipt['gradient_tensor_norms']),'valid targets',receipt['targets'])

The Scope of Memory and Compute Estimates

If the parameters, gradients and Adam's first-order status are in float 32, the steady state alone will account for only about 16 times the number of parameters bytes, this example is 309120 bytes, approximately 3002 KiB. The number of steps of the optimizers, Python objects, frame libraries and distributors has not yet been activated, temporary work areas, and cannot be measured as process memory. The focus fractions are about B x headlines x T×T, increasing with window squares; The vocabulary is about B×T x V, and the size of the list increases peaks. This item may be small to frame costs over matrix calculations, so short wall clock hours cannot extrapolate GPU or large model efficiency.

A common training calculation approximates $6PD$ The cost of proximate and forward projection for each training token is multiplied by the constant number of parameters; it ignores part of the quadratic attention terms, activation, normalization and actual internal kernels, so this course uses it only as a budget level statement, which is not properly measured. Very small models with high, low layers, large window changes may be particularly significant. Chapter VIII reports simultaneously on parameters, updates, effective targets and wall clock time; if there are no hardware counter measurements, close numbers will not be detonated.

This chapter item adds a complete backpropagation model, parameter formulae and gradient receipts. The model structure is clear. All parameters are learned from random initialization, and there is no simplification of the "outcome layer training instead of Transformer training". The next chapter fixes the training cycle and recovery state, expanding an effective update to 80 replayable updates; we need to prove that the next batch, next loss, parameters and optimiser status before and after the interruption is consistent, rather than merely demonstrating that the preservation file can be reopened.

Normalization itself has parameters and conditions. For each position, the d-dimensional a is the coordinates and divided by d, the difference is the square deviation between the coordinates and the mean; this realization is achieved without bias in the sample. The re-export is the scaling parameter multiplied by the centralized vector by the square root of the psilon, which then equalizes the parameter. Epsilon is fixed at one in 100,000. Scaled and scaled with d elements, which explains why each LayerNorm contributes twice the d parameter, and that bias are not assimilated to “Auto Disappear” without recording it.

The forward feed first projectes each location independently from the d-D-D-D-D-D-D-D, and then returns to the GELU-D-D-D-D. Exact GELU is the input of a standard normal-state cumulative distribution function; it does not cut off completely to zero in the negative semi-axis, but is continuously adjusted. All positions are shared with the same ex-Bow weight, but not mixed; cross-position information is mixed with attention, and position identification is derived from the location table. These functions are clearly defined to understand why only the ex-Burb width changes parameters and nonlinear capacity without directly expanding the upper limit of the context.

The local output gradient can also be derived from the logarithm. The loss of individual target k is the sum of logarithmic indexes per logarithms minus z_k; the guide to z_j is the first for exp(z_j) divided by the index and the second for a log K equal, or zero, which together is the p_j minus indicator value. Stability achieves the maximum logit before it changes the probability because the molecule denominator is multiplied by one factor; the maximum value is merely a numerical benchmark in the exact equivalent expression.

Take h from an earlier linear map, h=xA. If you have just sought to hide the gh of the gradient, then the A gradient is x- and h-g-g, and the input gradient is gh-by-A. This step indicates that the limited difference in output projection is only a partial check on the chain; attention, standardization and GELU are still relative to each other, and complete automatic fractions need to be accumulated along all nodes. The project did not write all the reverses by hand, and the use of the frame autograd to perform these defined operations and test the connection of the true link cannot be taken as an inference.

Checking updates also takes into account optimizers. One measure with a gradient but not placed in the optimizer parameter group will not be modified by Step; one measurement gradient is exactly zero, and AdamW decay may change it. “The parameters have changed.” therefore cannot alone prove that the gradient of the current data has arrived. Reasonable receiving and inspection combinations include the alignment of the parameter object pool with the model naming pool, the limited gradient value, the non-zero expected number of the selected controlled batches, and actual updating differences. The current 13 tests cover the total number of configurations and gradients, respectively, and do not replace a total metric with 17 volume details.

An example of a controlled shape that is in front of the complete model is B, T, III, d, 24. Each head is six-wide, two-by-four times three times six, two-by-four times three times three, and two-by-four times twenty-four, and then two-by-three times two and two-hundred and ninety-nine. The four heads are just coordinates for the same 24-dimensional projection results, and the output width cannot be written into 96. This example allows the shape ledger to be reconciled with the target ledger and also locates errors that have misdirected the head, time or volcab axes: softmax attention along the key position, output classification softmax along the vocabulary, and two homogenization axes are not interchangeable.

The input check-list index, the concentration-alignment axis and the output classification axis are independently agreed and the same shape does not mean the same probabilities.

python -m unittest -v tests.test_pipeline.PipelineTests.test_parameter_formula_and_all_gradients tests.test_pipeline.PipelineTests.test_causality_and_padding_invariance