Implement a Trainable Decoder

Objective: From Tokens to a Fully Trainable Model

This chapter incorporates the first six chapters into a Decoder language model. Entering is a phrase number, output is a logits for each location to the next word; when matching targets is given, a target is returned to a mark loss that can be transmitted in reverse. The premise is that embedded, causally focused, pre-LN block, cross-entropy and automatic fractions are understood. This chapter does not just examine the output of one layer but rather completes a gradient link to the whole parameter, a true optimization and fixed batch diagnosis; large text generalization is reserved for subsequent data experiments.

Defines the size of the vocabulary $V$ Maximum context length $C$ Number of layers $N$ Hide width $d$ Headings $h$ Width with Front $f$. This chapter uses the learning absolute position; terms embedded are not tied to output weights; two layers of FFN are biased, Q/K/V/O and head are unbiased, and all LayerNorm maintain chapter V population differentials and stabilization constants. The training demonstration schilling dropout is zero, and models, data and optimizers are created on CPU to avoid mixing equipment differences into correct checks.

Freeze the Public Interface Before Organizing the Internals

Config receives vacab_size, Block_size, d_model, n_heads, n_layers, d_ff, dropot; respective $V,C,d,h,N,f$ Checks all widths and layers when constructed to be positive, headlines to remove hidden widths, and the probability of discards are legal. Illegal configurations are wrong at the construction stage, better than waiting for a reshape to create an anomaly that is difficult to trace. This chapter does not depend on what variable names are given within the project to block and project.

Integer tokens cannot be sent directly into the matrix. The vocabulary is embedded in the matrix. $E\in\mathbb R^{V\times d}$ Numbered rows, position table $P\in\mathbb R^{C\times d}$ Zero to Location $T-1$ Rows. The same position table is shared over batches, so the position shape $[T,d]$ It can be broadcast legally in bulk axes; it is not the same as an accidental residual broadcast that was warned in chapter V because the sharing here is a clear design. Initialization status is recorded as $H^{(0)}$ 。

The loop is: No. $k$ Before the sub-cycle, $H^{(k)}$ Shapes Still $[B,T,d]$ Location $t$ Only rely on input zero to $t$ . The initial check sheet, added to the location by location, clearly does not read the future position, so the causality of the whole machine's hidden state is summarized. It is not this push when you call only the last block, read all the pieces together with an initial input, or accidentally overwrite the original tokens in the loop.

If you give a targets, calculate negative logarithms for each active position under Chapter 6 and aggregate; no fixed batch in this chapter is padding, so it is directly averaged. Returning to the dictionaries the loss must still be calculated; loss. for logs. item() Generate only an uncharted number that cannot be repackaged as a loss. Keep without targets loss=None, which makes the reasoning call not a false label. The training does not return to a step-by-step cache, so as not to tie historical calculations, expired weights or different samples.

Example 1: A Two-Layer Model Small Enough to Calculate by Hand

I'm not sure how much I can do for you. $V=d=f=2,h=1,N=2,C\ge2$. The project also retains UNK/BOS/EOS, requiring at least three volcab_size, so the two-category configuration is used only for the theoretical verification of the standard library and not directly to the project Config. Enter the two characters A, B, number zero, one; embedded respectively $(1,0)$ 、 $(0,1)$, the position list is both zero. Both the attention output and the forward feed output are set at zero, so the hidden state of the first and second block is identical to the initial embedding. The binalLN's gamma is one, Beta zero, the head mathematical matrix is a 2nd order unit matrix; all settings are specified ad hoc parameters.

The first position is equal to zero-five and the center is converted to $(0.5,-0.5)$ The difference in population is 0.25. $a=0.5/\sqrt{0.25+10^{-5}}\approx0.9999800$, the final logits for each location $(a,-a)$ 、 $(-a,a)$ The target is set to B, A, which is the lower logit in each row, so each line is lost. $\log(1+e^{2a})$, about two o'clock one, two, six. The average is still this value and it will not be divided into words.

Now, head two-column swaps, and the target becomes a higher logit for each row. $\log(1+e^{-2a})$ It's about 00:1269. $2a$ Because... $1+e^{2a}=e^{2a}(1+e^{-2a})$. This gives an independent algebra check of the entire data stream: if the code forgets the final LN, binds the wrong output direction, only exports the last position, or moves the target internally, it does not satisfy the value, shape and difference simultaneously.

import math
embeddings=[[1.0,0.0],[0.0,1.0]]
hidden=[row[:] for row in embeddings]
for _ in range(2):
    hidden=[[v+0.0 for v in row] for row in hidden]
def ln(row):
    mean=sum(row)/len(row)
    var=sum((v-mean)**2 for v in row)/len(row)
    return [(v-mean)/math.sqrt(var+1e-5) for v in row]
logits=[ln(row) for row in hidden]
def ce(row,target):
    top=max(row)
    return -row[target]+top+math.log(sum(math.exp(v-top) for v in row))
high=sum(ce(row,y) for row,y in zip(logits,[1,0]))/2
low=sum(ce(list(reversed(row)),y) for row,y in zip(logits,[1,0]))/2
a=0.5/math.sqrt(0.25+1e-5)
assert abs(high-low-2*a)<1e-12
assert len(logits)==2 and len(logits[0])==2
print('logits',logits,'wrong-direction loss',high,'swapped head loss',low)

Zero branch counts are used to check the connection and not to check that all weights of attention can be learned. When an output project is zero, the value gradient between query and key is zero; this is determined directly by the chapter VI chain law. The next item must be replaced by a general random initialization, checking whether all parameter loads are connected to loss on multiple-position inputes. Giving the two mechanisms to the same group of special zero weights makes it impossible to distinguish “No gradient” from “Normal zero gradient for this case”.

Parameter Registration: Differentiability, Discovery, and Updates

In PyTorch, nn. Module manages submodules and parameters; attributes a module to another module, which involves it in relativating parameters. Multiple pieces should be placed in registered containers such as nn. Modulelist and actually call in loops. The normal Python list can save objects, forward may also run, but the submodule does not automatically appear in the outer layer parameters(), device migration and state_dict().

The wrong example is to create a block first and repeat the same object several times with a list. This is not a number of separate blocks, but the same parameter is repeated; automatic fractions aggregate multiple calls of gradients, but they still have only one weight. The correct way is to register the entire collection every time a new block object is constructed. Chapter V's parameter accounts are able to find fewer numbers, so a layer-by-form object identity check can be directly located and shared; they examine numbers and structures without using training curves.

Normal tensor even requires_grad=True does not amount to a module parameter; self-build weights should be used in nn. Parameter or in the registered module already have Parameter. On the contrary, fixed cause and effect mask, location index, etc. are constants needed to be calculated and should not be counted as trainingable weights in the argument table and can be counted or registered as buffer if required. The registration impact saver, migration and optimizer finds that the meaning of a mathematical formula is not changed.

Before creating an optimist, move the model to the target device and dtype, and create the optimizer with a mode. parameters(). Then check the object's identification pool of parameters held by the optimiser equals the array of parameters for model training. If a Linear is created and replaced, the old optimiser may still point to the old object; the new layer forward and backward are normal but never updated by the old optimizer. Such failures should not be remedied by increasing the number of training steps.

The gradient link check is one by one named_parameters(): for the amount of weight that requires training, Grad is None usually indicates that it was not involved in micropaths or cut off; Grad is the valid result of the current input. The zero line embedded in this batch should also be in the list of terms, rather than requiring that none of the words 'n' rows be zero. Once again, each gradient element is limited, recording the gradient range of the modules; looking at only one head does not prove that the previous block is used.

Checking Each Derivation Against the Implementation

The share item model.py has made these logics specific: token_embeding and position_embedding create initial status, and Blocks constructs the ModeleList of TransformerBlock, final_norm after head. Each one calls in ln_attention, attention, in _feedforward and feedforward; attention is divided and combined and results and consequences are visible softmax. The public interface does not rely on these internal names, but the reading is done by tracking the equation of this chapter along with them without necessarily guessing the hidden black box layers.

Initialization is also an implementation contract: the project resets Linear and Embedding rights to a nominal random number of zeros, with a standard deviation of zero-two, with a linear bias of zero; LayerNorm keeps the frame by default shrunk one, with a bias zero. Random initialization is the conclusion that all layers are artificially selected rather than derived from cross-entropy. If all layers are completely zero, multiple features may remain the same, and the front layer may not receive non-zero signals due to the back layer zero power; so the zero branch limit used cannot be replaced by normal learning tests.

The training session in this chapter must be: content/projects/transformer - practice/README.md Executes the directory of the project and uses the fixed Python and PyTorch environments of the project. It directly calls read Config and TransformerLM and does not rely on non-existent build_decoder or fit interfaces. Here, the four-separations test only the length of learning, and do not serve as natural text for the retention symbols; Chapter VIII constructs the true target through CharTokenizer and the document window.

Overfit a Fixed Batch to Validate the Learning Pipeline

A fixed batch test uses exactly the same tokens and targets at each step, closes dropout and data enhancement, and does not draw validation set conclusions. If a model that expresses the micromap is not sufficient to reduce training losses, priority is given to checking connections, target alignment, gradients and optimists, rather than exchanging more training materials. To avoid random results being miswritten as definitive, the bottom code threshold is a confirmation to be run; only when it works, the ultimate loss can be included in the trial record.

import torch
from hero_transformer.model import Config, TransformerLM
torch.set_num_threads(1)
torch.manual_seed(7)
model=TransformerLM(Config(vocab_size=4,block_size=8,d_model=16,
    n_heads=2,n_layers=2,d_ff=32,dropout=0.0)).double()
optimizer=torch.optim.AdamW(model.parameters(),lr=0.02,weight_decay=0.0)
x=torch.tensor([[0,1,0,1]],dtype=torch.long)
y=torch.tensor([[1,0,1,0]],dtype=torch.long)
model_ids={id(p) for p in model.parameters() if p.requires_grad}
opt_ids={id(p) for g in optimizer.param_groups for p in g['params']}
assert model_ids==opt_ids
model.train()
initial=float(model(x,targets=y)['loss'].detach())
for _ in range(120):
    optimizer.zero_grad(set_to_none=True)
    out=model(x,targets=y,past_key_values=None,use_cache=False)
    assert out['logits'].shape==(1,4,4)
    assert out['loss'].ndim==0
    out['loss'].backward()
    missing=[name for name,p in model.named_parameters() if p.requires_grad and p.grad is None]
    assert not missing, missing
    assert all(torch.isfinite(p.grad).all() for p in model.parameters() if p.grad is not None)
    torch.nn.utils.clip_grad_norm_(model.parameters(),1.0)
    optimizer.step()
model.eval()
with torch.no_grad():
    final=float(model(x,targets=y)['loss'])
    pred=model(x)['logits'].argmax(dim=-1)
assert final<0.1 and final<initial
assert torch.equal(pred,y)
print({'initial':initial,'final':final,'prediction':pred.tolist()})

This small data is alternating rule, but the model must still be embedded in words, position sheets, two blocks, finalLN and head to produce a loss of training. The results should save seeds, configuration, initial/final losses and projections, rather than just the last line of the map. The test does not mean that it learns to depend on long distances or that all parameters carry the same information. The head or disturbance layer can be further frozen for diagnosis, but cannot be used to freeze this chapter without having to do so.

Example 2: Conflicting Labels for One Context Cannot Be Fit to Zero Loss

Construct two identical input sequences, the same location, the same visible prefix; in a given location, the first target is zero and the second is one. Model is implemented with certainty and no additional input such as sample number, so this position must have the same probability for both samples. $p\in(0,1)$, the other category is $1-p$, the average of the two rights cross-entropy is the function below.

The best odds are half and half. Loss. $\log2\approx0.693147$. If the model is in favour of zero to zero nine, the two losses are worse by about one hundred and fifty-three and two thirty-six, respectively, an average of one hundred and twenty-four. Training around one hundred and sixty-nine may therefore be a correct harvest, not a lack of layer numbers. Setting the threshold at random to zero one would make the issue permanently impossible; restoration would require removing conflicting indications, adding information that distinguishes the sample, or recognizing that the task itself is a random distribution of conditions.

import math
def objective(p):
    return -0.5*math.log(p)-0.5*math.log1p(-p)
for p in [0.1,0.5,0.9]: print(p,objective(p))
assert abs(objective(0.5)-math.log(2))<1e-12
assert objective(0.5)<objective(0.1)
assert objective(0.5)<objective(0.9)
# independent normalization check for the matched optimum
assert 0.5+0.5==1.0

System Validation Requires Independent Evidence

The first category is semantic and algebraic: ad hoc hand-counting models, accurate parameters books, non-variable prefixes for cause and effect, repeated input for duplicate output. The second category is learning and status checks: complete loss.backward, gradient connection for each parameter, optimised device identification pool, fixed batching. The former finds structural errors, the latter discovers missing breakcharts and updates; none of these passes replaces the other. Float comparisons are made using defined tolerance differences, without using string lines to replace exact numerical synonyms.

The result can be the same model, the same prefix, which combines two different future suffixes, comparing logits. Checking the pre-evenal and no_grad to avoid random differences in dropout being mistaken for future leaks. Entering longer than black_size should be rejected, not silently looping position numbers; location is a learning table where repeated bug numbers change the meaning of the model. Returning all positions cannot be changed to just the last on the basis of performance, because training contracts require supervision of each valid position.

This chapter item increases the full TransformerLM in motor.py, as well as a direct-run fixed batch acceptance: microconvergence and gradient path checks that include configuration, registration of all modules, forward-return dictionary, loss. Chapter VIII access to document data, right pageding, valid hyphen weight assessment, sampling status and repossibility check points. At that time, fixed batch tests are retained as active controls; if real training fails, run it again first, distinguishing model chain damage from data/restoration problems.

# TransformerLM.forward（替换已提供类的此方法）
def forward(self, tokens, targets=None, past_key_values=None, use_cache=False):
        raise NotImplementedError('decoder')

# TransformerLM.forward（第7章：先完成无缓存整机）
def forward(self, tokens, targets=None, past_key_values=None, use_cache=False):
    if past_key_values is not None or use_cache:
        raise NotImplementedError('第10章再增加缓存路径')
    if tokens.ndim != 2 or tokens.dtype != torch.long or tokens.shape[1] == 0:
        raise ValueError('tokens must be nonempty [B,T] int64')
    batch, length = tokens.shape
    if length > self.config.block_size:
        raise ValueError('input exceeds block_size')
    positions = torch.arange(length, device=tokens.device)
    hidden = self.embedding_dropout(self.token_embedding(tokens) + self.position_embedding(positions))
    for block in self.blocks:
        hidden, _ = block(hidden, past_key_value=None, use_cache=False)
    logits = self.head(self.final_norm(hidden))
    loss = None
    if targets is not None:
        if targets.shape != tokens.shape or targets.dtype != torch.long:
            raise ValueError('targets must match tokens shape and int64 dtype')
        valid = (targets != -100).sum()
        if valid == 0:
            raise ValueError('targets require at least one scored token')
        loss = F.cross_entropy(logits.reshape(-1,self.config.vocab_size), targets.reshape(-1), ignore_index=-100, reduction='sum') / valid
    return {'logits':logits,'loss':loss,'past_key_values':None}