Connect SFT and DPO to a Complete Decoder

Task and Dependencies: The Role of Each Implementation

The limited strategy of the first six chapters is to be exhaustively referenced and not to be a final model of the course. This chapter adds a true self-written decoder sequence, which requires readers to have access to Transformer ' s attention, preLN, location and reverse transmission, and can read the PyTorch index. The project root catalogue contains both lab.py and sequence_lab.py: the former tests SFT/BT/DPO formula using an eight-parameter strategy, while the latter trains complete decoder from random rights, with answers generated by several bytes totoken self-regression. Both are offline and do not import brother courses; decoder.py is a local copy of the declared source and does not use nn.Transformer or pre-training weights.

Sequence versions follow the same value x partitions, but the reminder format is fixed to ASCII strings such as "x=-3.0; answer: ". Four responses become a lengthy version of “NEG”, “POS” and each with "because of sign. ". The fixed UTF-8 base byte list is zero to two hundred fifty-five, BOS 256, EOS 257, no UNK and standalone PAD. This only guarantees the encoded coverage, which does not guarantee that the new hint format is semantic. The fifth form probe displays legal codes still fail.

From Raw Strings to Exact Response Targets

Each line of batch(rows) is (x,a). Prefix for BOS plus hint bytes and answer for answer bytes plus EOS, where the link is. Enter the last item and target the first one. Effective mask drops len(prefix) off a False and len(answer) entry True. This length seems to be reduced precisely because the last input position forecast for prefix answers totoken; If you put len(prefix) Falses, the answer will be left out. The target is negative 100 and the right fills EOS, which is causal to ensure that future right fills do not affect effective left-hand queries.

response_logps(model,rows) I've had it before. B×T × 258 Logits, logs log_sftmax for the heading of the vocabulary; uses the target minus one hundred temporary clamp for a safe gather index only, and multiplys the effective mask to make its contribution and gradient zero. The logprobs that complete each answer are sought and returned along time, as well as the true number of target answers per line. Real EOS is included in answer, the target minus one hundred is not entered into the denominator. The model does not appear to be a random left-pading interface, nor is it valid as EOS by input.

Example 1: End-to-End Accounting for a Negative-Number Prompt

X=- 3.0, hint string 'x=- 3.0;answer: "There are 14 bytes, prefix plus BOS is 15. Short answer NG byte plus EOS with four targets. Enter 18 in total, the first 14 target positions are not rated, index 14 first predicts N, then E, E, E, E, E, E, E, E. G 、 EOS, with four effective targets. Long answer: NEG You can also enter a 35-point total under prefix. The two responses will be filled in a 17-digit right, and the two lines will be filled without extra targets.

The first check begins with a valid position mask. valid = (y!= - 100, and remove y in order [valid]. Every target number obtained must be with "Assured" UTF-8 Byte numbering, then one. EOS The second check compares the probability of a full answer with the same short answer, separately and in the long run. The third check takes the right type of log_softmax, and seeks a sum of the correct type, and compares it to the quantitative gath. The number corresponds precisely to the member relationship; the floating point is compared to the stated accuracy and tolerance difference.

The Complete Model and All Its Parameters

Sequential model configuration V=258, d=16, four head, one layer, front-broadness 32, maximum position 64, dropout zero. Attention Q/K/V/O no bias, previous beas, learned absolute location, preLN with precise GELU, input output matrix not shared. The parameter is a matrix of two words, 8256, position 1024, attention 1024 at this level, pre-bean matrix 1024, prior bias 48, two times LayerNormom64, and ultimately LayerNorm 32, total 11472. This account is closely aligned with the actual parameter Numel and cannot be described as a mere output head.

Take_step to accept the calculation of lost, clear gradients,backward, check grad existence and all limited by name parameters, then do a global paradigm threshold one crop, and end up AdamW Step. If los or gradients are not limited, make a direct error; nan cannot be replaced with zero to continue producing the result. Seventeen sheets include embedding, location, attention four projection, two layers forward, and all homogenization. The receipts save the number of units and the maximum parameter changes in the whole phase; Tests have examined these links with a group of SFTs and a group of DPOs, rather than replacing the evidence of complete training with a total loss.

The APIs for SFT, the Reference Model, and DPO

Sets_sft_loss takes a negative view of each answer to logprob, with an average of 18 models and chapter II is consistent with the overall NLL goal. Staple SFT 60 steps, AdamW learning rate 0. 006, decay 0. 01. Deepcopy out of reference, event and closes all requires_grad; calculates ash with its full parameters bytes. DPO builds AdamW with learning rate 0. 002, decay 0. 01, $\beta$ 0.1, training 40 steps. Rebuilding the optimizer is a clear phase border and does not claim to optimize the same goal from SFT retention.

Seq_dpo_loss split 18 pairs into two groups: Winner and Loser, each counting the complete reposition logprob; reference scores the same two groups in no_grad. Four sets get margin by chapter VI differential log, and softplus takes it evenly. There are limited gradients and physical changes in the 17 steps of policy. The code does not copy the simplified gradient of the limited strategy to decoder, nor does it divide the length of the answer logprob.

The independent selected parameter has a limited difference using the Double model, which adds one hundred thousandth to the first parameter of the output to answer the SFT loss range in its entirety, compared to seven decimals. It only verifies a sequence chain and cannot be said to have all elements of the complete network validated by item; the existence, limitations and updates of the full volume are covered by other tests. The limited reference does distinguish between SFT and the DPO centre for all eight of its parameters, and the two achieves complementary verification costs.

Independent Experiment: Finite Differences Across All Seventeen Parameter Types

The old test only gives a SFT score for the first coordinates of the output, and cannot claim that all network branches have been verified. New gradient_audit.py created seed=919 The random float64 decoder, which closes the dropout, does not read the frozen training weights or executes the optimist. It uses two SFT answers of length and two preference comparisons; references are copied and frozen in depth. For each target, each parameter is rounded to select a non-zero coordinates with the absolute maximum value of the parsing gradient, and then calculates the positive or perturbation loss independently. This avoids a zero-to-zero empty check that is taken with the unused position.

Only one coordinates are checked for each volume: 17 times two targets, 34 points; not a list of 11472 parameters by coordinates. The absolute error threshold is fixed at 10-6, the actual maximum error of SFT 2.450343×10-8, and the DPO is 2.420345×10-9. The table below shows the “resolution gradient/ variance absolute error” of the selected point; the specific flat index, the numerical differential result, configuration and source code summaries are stored in runs/gradient-audit-reference.json. Different size selection points should not be compared to table values to rank the hierarchy in importance.

from gradient_audit import audit
r = audit()
assert r['random_model_restored_exactly'] and not r['training_performed']
for name, phase in r['phases'].items():
    assert len(phase['coordinates']) == 17
    print(name, phase['max_absolute_error'])

.venv/bin/python gradient_audit.py --output runs/gradient-audit.json

The point difference is only wrong to give a random model and input, and it cannot substitute for the general chain rule to prove that the freezing model is generalized. The freeze, the precise members of the integer mask, the right filler are consistent, the source file identity remains the responsibility of other checks. At the end of the script, all parameters of the random model are checked for consistency with the initial values; it is an independent acceptance, not a new round of training.

Example 2: A Recorded Case of Probability Escaping the Candidate Set

Fixed test x=-3.5, SFT correct short answer to NEG's logprob about − 0.707623 with absolute probability of 0 492814; total path probability for the four specified candidates is about 0 575782. The correct short answer after DPO is about − 2.265495 with probability down to 0 103779; the other three candidates are reduced faster to very small and the total quality of the four candidates is reduced to about 0 103779. There are still a large number of possible strings outside them, and the probability does not disappear but moves to an unstated result.

If you do softmax for only four candidates, the post-DPO condition is almost equal to one, because the other three are lower. This is not the probability of generating it in its entirety and cannot replace 0.103779. The independent check is that each original logprob first gets an extra copy of the total quality of the candidate, then an absolute probability of a candidate gets the condition; the two steps must be separated. The serial version therefore reports Candidate_tal_path_mass and Candidate_convention_probabilities and clearly limits the candidate KL to this re-alignment space.

This is the sixth counter-measure in a complete model: the preference for margin improvement does not guarantee an increase in the absolute probability of Winner. It also suggests that greedy is not correct enough to judge distribution quality; as long as a path progresses towards argmax, the whole range may still be low, and random sampling falls into many untrained strings. The course does not run large-scale random sampling success rates, so that four right samples are not used as strategic accuracy expectations, much less as a true sample success.

Saving, Independent Evaluation, and State Boundaries

Save format version, serial data contract summary, full configuration, policy and reference weights and $\beta$;load check the summary to recreate and load the two models. The summary contains a reminder format, response string, BOS/EOS, mask and SFT/DPO request for peace, avoiding only checking values x without ignoring changes in format. This document is used to freeze reasoning and independent evaluation, without preserving Adam's state, phase cursor or RNG, and therefore does not provide precise recovery commitments for serial training.

CLI operates on the project root directory: evidence_lab.py Train generates a true training receipt, information_lab.py evaluation-split test independently of freezing weights; probe_formats.py keeps all output on five fixed format changes. The two sets of outcome documents have been separated to avoid mixing the precise KL of the eight parameters with the DEcoder candidate diagnosis. The compressed package includes decoder, two realizations, data rules, testing and status, and no other curriculum catalogue is required.

This chapter item adds a complete post-sequence training chain. Chapter VIII will explain the failure of training Loses, setting aside model NLLs, hidden Pair indicators, greedy behaviors and formats, indicating which target conflicts do occur. Before entering an assessment, it should be possible to recreate each step from catch members, path fractions and reference hash, rather than simply trust a “Training successful.” logbook.

Read-only style diagnostics further unplugged SFT distribution changes. The right answer for the correct short answer average NLL ranged from 0.690142 to 2. 658300. The correct answer ranged from 2.472356 to 91. 301003; the combination of two short and one lengths restored about 1.28413 and 32. 205868, respectively. The long answer is one of the desired directions that clearly inhibits the length of time and cannot be said to be “All correct answers are degraded.” on the basis of the combined mean. However, the average NLL for short answers also rose, and evidence of a single path for an absolute decline in NEG suggests a more specific qualitative outflow than the change in the mix of targets.

The greedy function also imposes a constraint that is not contained in the training rating: to set BOS candidate logit on a negative basis and then take an argmax. Response_logps always rate unshielded 258 classes of softmax. Hence the greedy correctness is a separate decoding strategy, neither random sampling from original pi nor distributed selection criteria. If changed to restricted random sampling, the token probability will be allowed to double-check with one less BOS quality and the full path probability will vary accordingly; This course does not present this new probability as an original distribution of training.

Two types of verification require individual tolerance: the discrete target member, response count and reference status identity requirements are accurate and consistent; successive floating point losses and limited differences are given a precise and statutory range of tolerance. All checks are uniform to “Something like that.”, leaving the mask in the wrong position, and all floating point statutes require the same numbering and misreporting the legal order as algorithms fail.

import math
sft=[-.7076233625411987,-9.331679344177246,-2.491082191467285,-9.736905097961426]
dpo=[-2.2654953002929688,-39.93329620361328,-89.37799835205078,-123.78866577148438]
for label,lp in [('SFT',sft),('DPO',dpo)]:
    absolute=[math.exp(v) for v in lp];mass=sum(absolute)
    conditioned=[p/mass for p in absolute]
    assert abs(sum(conditioned)-1)<1e-12 and mass<1
    print(label,'winner absolute',absolute[0],'candidate mass',mass,'winner conditioned',conditioned[0])
assert math.exp(dpo[0])<math.exp(sft[0])
prefix=15;short_answer=4;long_answer=21
assert prefix+short_answer-1==18 and prefix+long_answer-1==35
print('first response target index',prefix-1,'counts',short_answer,long_answer)

python sequence_lab.py train
python sequence_lab.py evaluate --split test
python probe_formats.py
python -m unittest discover -v -s tests

import json
from sequence_lab import load,evaluate,ROOT
from lab import SPLITS
model,reference,beta=load()
report=evaluate(model,reference,SPLITS['test'],beta)
assert sum(p.numel() for p in model.parameters())==11472
print('response targets',report['response_targets_with_eos'],'mean NLL',report['mean_response_nll'])
print('short and long diagnostics',json.loads((ROOT/'evidence'/'sequence-path-diagnostics.json').read_text())['by_phase']['dpo']['short_mean_response_nll'])