Training, Data and Reproducible Recovery

Objective: Reproduce the Next Update Completely

Chapter VII has demonstrated that the model is connected and prepared to a fixed batch. This chapter connects real documents, different length windows, random sampling, validation and archiving into the same training process. At the end of the session you should be able to explain why a loss denominator, from which document each target comes, and why the next step after an interruption is the same as an uninterrupted training. The premise is to understand the public interface between cross-encolon, AdamW state, fruit mask and model; the precise recovery scope of this chapter is limited to the same CPU environment, the thread and the project locking software environment, and there is no commitment to cross-equipment.

Split Documents Before Building the Vocabulary and Windows

Project data/documents.jsonl contains 24 original short articles, each with a stable id and text; stlits.json specifies 18 trainings, three validations, three tests. Split_documents first check the unique number, three pools, and properly overwrite the original, before encoding and building the window. If you cut an article into multiple overlapping windows and then randomly group it, the same context can easily be entered simultaneously into the training and validation, and scores can be added to the memory effect.

The document ID isolation is a necessary check, but two duplicate text or semantic approximation articles under ID cannot be detected. If the true material is changed, it will also need to examine the source; the project does not achieve universal near-record detection, so numbers cannot be exaggerated as excluding all leaks of information. It is normal that the same Chinese characters appear in multiple pools and do not amount to leaks; what is needed is to prevent answers to the same task or segments of the original text from being seen in advance through improper division.

CharTokenizer.Fit read the training document only and numbered the existing Unicode characters; retained UNK=0, BOS=1, EOS=2. UNK represents the extra-word character of the hyphen, BOS indicates the beginning of the document, and EOS means the end of the document. Validation and testing repeats this hyphen with an unknown map of zero and does not snuck up the hyphenology table during the assessment. The order of words is also part of the semantic meaning of the model: the same weight will change the meaning of the output text even if the size remains constant.

One text length $n$ Document Encoded as $s=(BOS,c_1,\ldots,c_n,EOS)$, length of $n+2$ of which $c_i$ is the character number. Defines the forecast for $(s_i,s_{i+1})$, enter the index $i$ From zero to zero $n$ ♪ So there is ♪ $n+1$ Goal: Once for each real character and once for EOS; BOS for input only. The title does not enter the training body, the label is a character, and the unknown word is still counted as UNK. This denominator determines what the loss can be compared to.

Window Boundaries Are Modeling Choices, Not Just Memory Tricks

Make Block_size a positive integer $C$. Build_windows takes start to zero, $C$ 、 $2C$, stop as start plus $C$ With smaller people who can eventually enter the location; inputs pick up s [start:stop], towers pick up s[start+1:stop+1]. The window does not cross the document, the target area does not overlap, so the trages should be put back together with the document windows exactly to encode the first BOS. This sequence is more powerful than comparing only token.

The new window position is renumbered from zero and the focus is recommenced. The first input to the boundary may be a character that has already been predicted in the previous window, but it is only the starting point for the new context; the goal is not repeated. This score is not intended to be unlimited, nor does it provide a full slide assessment of each target. Changing the length of the window will change the visible context and absolute position, and the two window scores cannot be treated as differences in performance under exactly the same task.

Example 1: Two Documents, Right Padding, and the Correct Denominator

Sets the training list with A, B, C, D, E, Nos. 3 to 7; first entry is “Alpha-ethyl-butyl-methyl-methyl-ethyl-methyl-methyl-ethyl-methyl-methyl-methyl-ethyl-methyl-methyl-methyl-methyl-methyl-methyl-ethyl-methyl-methyl-methyl-methyl-methyl-methyl-methyl-methyl-methyl-methyl-methyl-methyl-methyl-methyl-methyl-methyl-methyl-methyl-methyl-methyl-methyl-methyl-methyl-methyl-methyl-methyl-methyl-methyl---------”, coded $(1,3,4,5,6,7,2)$. The first window is entered $(1,3,4)$ Target $(3,4,5)$;second input $(5,6,7)$ Target $(6,7,2)$. The six targets coincide with the five characters plus one EOS. Second piece of “A.” encoding $(1,3,2)$, Enter $(1,3)$ Objective $(3,2)$ The two volumes combined eight points.

Makes a group of windows one and two short windows, which require a uniform length three. Collate fills EOS on the right side of input and short lines change to $(1,3,2)$;right-hand mark for target $-100$, Change to $(3,2,-100)$. The third position of the short line is manually filled and not involved in the loss; the second position is projected EOS It's a real target. It must be retained. It cannot be equal to the target. EOS Ignore, otherwise the model never learns the end of the document.

Why do you enter EOS without contaminating the real position in front? The causal focus allows the position to read only zero and one; it is impossible to read the right of position two; LayerNom and FFN treat features only in the same location. The result of the first two positions is not affected by the back filling. This conclusion depends on the right fill and the causal structure; if the left fill is changed, the location number and visible prefix are redesigned and cannot be automatically applied.

Assuming that the two assessment instalments contain six valid objectives, zero five mean values and two valid targets, two averages. The total negative logarithms appear to be six times five plus two times two, equal to seven; except for eight:875. An average of one point two and a half directs the averages of the two instalments, resulting in an overweighting of the small batches; adding the filling position to the denominators is a false pressure on the low loss.

# Standard-library window and aggregation reference.
BOS,EOS=1,2
def windows(ids,C):
    return [(ids[a:min(a+C,len(ids)-1)],
             ids[a+1:min(a+C,len(ids)-1)+1])
            for a in range(0,len(ids)-1,C)]
first=windows([BOS,3,4,5,6,7,EOS],3)
second=windows([BOS,3,EOS],3)
assert [y for _,ys in first for y in ys]==[3,4,5,6,7,EOS]
assert sum(len(y) for _,y in first+second)==8
batch=[first[0],second[0]]; T=max(len(x) for x,_ in batch)
x=[a+[EOS]*(T-len(a)) for a,_ in batch]
y=[b+[-100]*(T-len(b)) for _,b in batch]
assert y==[[3,4,5],[3,2,-100]]
weighted=(6*0.5+2*2.0)/(6+2)
assert abs(weighted-0.875)<1e-12
print('inputs',x,'targets',y,'weighted loss',weighted,'wrong batch mean',(0.5+2)/2)

The Shared Boundary of Sampling State and the Training Loop

StatefulBatchSampler does not take an unlimited independent sample from all possible windows, but produces a round of window numbers from his own CPU toch. Generator and takes the next batch in a cursor order; the last batch is shorter than the catch_size. The sequence is exhausted for epoch plus shuffling. One epoch here is all built windows that are visited once and does not represent the same weight for each document: longer text windows and natural contributions to more training positions.

Sampler status includes the number of windows, lot size, current permutation, cursor, epoch and local generator_state. Save only seed will be reset from a random sequence starting point and cannot know how much has been consumed; save only cursor will not be able to re-establish the current rotation. The project does not have Python Ranom, NumPy random or multiprocess data worker, so does not claim to have processed their recovery; if introduced later, it will have to extend the status contract.

Real train_step(model, optimizer, sampler, windows) sets the train mode, numbering from sampler. next_indices(), and then Collate, empty gradients, get lost, Backward, cut global paradigms to one, optimizer.step. Returns the current loss, valid target, pre-cut gradient parameters and window numbers. The outer layer is updated only by step one; when archived, Step indicates how many updates have been completed instead of the next batch of labels.

The training log’s average area is weighted by valid target numbers, and the validation has been carried out throughout the fixed authentication window without random batches of training as an imposter. The training behaviour such as the eval mode closes dropout, no_grad closes the gradient record, which is different and must be used simultaneously. The evaluate_model remembers the pre-entry status and resumes at exit; otherwise, subsequent training may be suspended without error after a validation.

Define average natural log [N] loss as $L$ ♪ Confusion is for $\exp(L)$. Expand the definition $\exp[-\sum_i\log p_i/M]=(\prod_i p_i)^{-1/M}$, i.e. the number of geometrical averages of real target probability; it is not the size of a phrase or the correctness of a subject. The report should contain a valid number, window, vocabulary, and unknown rate. UNK merges more than one non-observed word, which changes the prediction event and cannot be directly compared with the confusion of another large metamorphological model.

Example 2: Restoring Only Weights Produces a Different Update Trajectory

One of AdamW ' s measurable exceptions is the optimist state: initial parameter 1, zero point, two betas at zero point nine, zero point nine, minus zero; for precision calculation, optimise stabilization item zero, the following square is still positive and therefore there is no reduction of zero. In descending order to gradient one, negative one. The first step is zero point one, second step zero point zero one, after correction one, from zero point one to zero nine.

When the second step is reserved, the first step is zero nine times zero one-one minus one-one, which is a negative zero-one-one; the second step is zero-nine times zero one-one plus zero-one-one-nine. After correction, the first step is divided by one-nineteen and the second is reduced by one-nine-nine-nine, which is exactly one. The second-step rule is therefore increased by one-nineteen and becomes about one-nine-five-two-six-three.

If you add only zero nine to the parameter, then you recreate a whole new optimist. Negative one is used as a first step gradient, which is corrected to adjust itself to negative one and increases the parameter directly by zero one to one. Even if the next batch is identical to the gradient, the difference is about zero, nine, four, seven and seven. The actual training gradient also depends on the preceding parameters, sampling and dropout, and the deviation will continue. This is a strict counter-state, not a condensed explanation of the real training recovery results.

import math, copy
def step(state,g):
    state=copy.deepcopy(state)
    state['t']+=1
    state['m']=0.9*state['m']+0.1*g
    state['v']=0.99*state['v']+0.01*g*g
    mh=state['m']/(1-0.9**state['t'])
    vh=state['v']/(1-0.99**state['t'])
    state['theta']-=0.1*mh/math.sqrt(vh)
    return state
start={'theta':1.0,'m':0.0,'v':0.0,'t':0}
saved=step(start,1.0)
continuous=step(saved,-1.0)
restored=step(copy.deepcopy(saved),-1.0)
weights_only=step({'theta':saved['theta'],'m':0.0,'v':0.0,'t':0},-1.0)
assert continuous==restored
assert abs(continuous['theta']-(0.9+0.1/19))<1e-12
assert abs(weights_only['theta']-1.0)<1e-12
print('complete restore',continuous,'weights only',weights_only)

What a Real Checkpoint Saves and the Order of Restoration

save_checkpoint(path, model, optimizer, sampler, step, tokenizer, run_manifest) writes the atomic replacement target to a temporary file and avoids reading only half of the latest document. This does not amount to multiple documentation or complete assurance of post-electricity persistence. Checkpoint_summary of Checkpoint.py's reviewpoint is not a substitute for training recovery; laad_model_checkpoint of evaluate.py only restores reasoning models and does not provide optimizers and sampling progress.

The real training recovery function is restore_training(checkpoint, windows, expected_corpus_sha256=None), returns model, optimizer, sampler, payload. Read the configuration, recreate the model and load the weight, then recreate the optimizer and sampler and load their respective status; ultimately restore the global torch RNG. Because model construction consumes random numbers, if first restored RNG Reconstruct, the new model will be initialized to push randomly forward, and then dropout will no longer be aligned with continuous training.

This project CLI uses saved tokenizer and block_size to re-preparate the data and compare content with Hashi; the same document ID is also rejected if changed to the text or split. Restore_training also requires PyTorch versions to be identical to the number of CPU lines. Resume command -- stepes is the total number of steps targeted, saving them to 160 and writing 200 again means that 40 more steps, not 200 more; train catch_size from checkpoint, new batt-size only impact assessment batches.

Recovery Validation Beyond Loading the File

From the same intermediate state, two branches are divided: A continues three steps without interruption, B continues three steps after archiving, rebuilding and restoring. A step-by-step comparison of window numbers with loss, and then a comparison of all model lengths, optimist status, sampler status with global RNG. It is not enough to load anything; the next batch may be different even if the final model is close. Using non-zero dropout during an examination allows errors from RNG to be actually activated.

import copy, tempfile
from pathlib import Path
import torch
from hero_transformer.data import prepare_data, StatefulBatchSampler
from hero_transformer.model import Config, TransformerLM
from hero_transformer.train import configure_cpu, train_step, save_checkpoint, restore_training
configure_cpu(7,threads=1)
tokenizer,all_windows,manifest=prepare_data(block_size=8)
windows=all_windows['train']
model=TransformerLM(Config(tokenizer.vocab_size,8,16,2,2,32,0.2))
optimizer=torch.optim.AdamW(model.parameters(),lr=0.003)
sampler=StatefulBatchSampler(len(windows),3,seed=19)
for _ in range(4): train_step(model,optimizer,sampler,windows)
def equal(a,b):
    if isinstance(a,torch.Tensor): return isinstance(b,torch.Tensor) and torch.equal(a,b)
    if isinstance(a,dict): return a.keys()==b.keys() and all(equal(a[k],b[k]) for k in a)
    if isinstance(a,(list,tuple)): return len(a)==len(b) and all(equal(x,y) for x,y in zip(a,b))
    return a==b
with tempfile.TemporaryDirectory() as directory:
    path=Path(directory)/'resume.pt'
    save_checkpoint(path,model,optimizer,sampler,4,tokenizer,{'seed':7,'data':manifest})
    before=[train_step(model,optimizer,sampler,windows) for _ in range(3)]
    expected=(copy.deepcopy(model.state_dict()),copy.deepcopy(optimizer.state_dict()),
              copy.deepcopy(sampler.state_dict()),torch.get_rng_state().clone())
    resumed,opt,samp,payload=restore_training(path,windows,manifest['corpus_sha256'])
    after=[train_step(resumed,opt,samp,windows) for _ in range(3)]
    assert payload['step']==4
    assert [x['indices'] for x in before]==[x['indices'] for x in after]
    assert [x['loss'] for x in before]==[x['loss'] for x in after]
    actual=(resumed.state_dict(),opt.state_dict(),samp.state_dict(),torch.get_rng_state())
    assert equal(expected,actual)
print('same-environment continuation matches model/optimizer/sampler/RNG')

Saved by Current Item evidence/smoke - run/metrics.json Records d=32 Windows 16, 2 floors, batch 8 training 160 steps: total training set loss approximately 5.962669 Present. 2.026304, Verify the contract 5.964481 Present. 5.435705. Verifying unknown characters is about 13:14% and testing about 17:54%. These are the results of an already documented document. They are not the new results of this code, let alone the loss of training to introduce a Chinese-language capability.

This chapter item increase is the retraceable window for data.py, complete cycle and restoration for train.py, effective target weight for evaluate.py, and a read-only check portal for checkpoint. Delivery should include split, vocabulary, configuration, data Hashi, paragraph-by-paragraph log, validation results and continuous/restoration comparisons. The subsequent generation section uses the same phrase sheet and checkpoint, but the deduction that the cache is not a training recovery state; Performance chapters must also distinguish between the step of updating, input/output token and the true time range.