Evaluation: What Has the Model Actually Learned?

Turn Learning Claims into Testable Propositions

The previous chapter has trained and preserved complete models, but the reduction in training losses only means that the optimist has improved its goal on the data that are used repeatedly. This chapter answers three more stringent questions: Is it possible to give a higher probability on documents that are not involved in updating? Which component is the change from? Which model version and range of evaluation are these conclusions dependent on? First, chapter II, pseudo-crossing entropy, and chapter VIII document splitting and inspection points. Model training, generation of a text, one point, three different observations, cannot be replaced.

We fix the question as "The real past character in the given document, predicts the next character." The evaluation is fed with a true prefix called teacher coercion; this is not about feeding back errors generated by the model itself. The former examines the probability distribution of conditions, the latter examines long rolling behaviour. A model may have a better probability of conditions, but it repeats them when freely generated; it may also occasionally generate a flow of text, but gives the vast majority of documents a very low probability of retaining them. Both checks are done, but both inputs and outputs are written separately.

The training group participates in the gradient update; the certification team allows the selection of training steps, learning rates, etc.; the test group only makes a final evaluation after these options have been frozen. If you have seen the test results and then change the settings, report again the best number of the same test set, it has assumed the validation role. This selection history should be retained at this time, as well as to prepare the unused test information, which cannot continue to be called independent tests.

Deriving Total Loss from Every Valid Target

Set a fixed model parameter as $\theta$, the document is assembled as $D$. Document $j$ Include after encoding $n_j$ A target that needs to be predicted, which does not include the input-end BOS, which includes the course 's agreed EOS; the right-hand position has no target. $t$ The goal is... $y_{j,t}$, the true prefix available in the current rating window is $x_{j,<t}$. Defines the negative logarithm probability of the target $\ell_{j,t}=-\ln p_\theta(y_{j,t}\mid x_{j,<t})$, the logarithmic bottom is taken from the natural base, and therefore the unit of loss is the nat of each projection word.

This conclusion requires the addition of molecules and denominators before dividing them. The average loss per batch is directly averaged, usually at the end of a batch with only two active targets, with the same weight as a batch with 200 valid targets. Only when the number of valid targets is exactly equal in each batch are the two routes equalized. Filling the length, the last size, and different length files will undermine this condition. The project 's evaluate function thus accumulates loss_sum and token_count instead of building an unweighted list of catch_loss.

Set the first instalment with only one valid target and loss is $0.1$; the second instalment had nine valid targets, with an average loss of $2$. The wrong batch averages $(0.1+2)/2=1.05$. The correct molecule is $0.1+9\times2=18.1$ The denominator is ten, so $L=1.81$. This difference is not due to the accuracy of the display, but rather to the evaluation problem being changed to “Draw even batch”, which is no longer “Draw a valid word evenly”. Changes in the weights of the sample must be clearly named.

Each input lost can also be independently recalculated by fractions. The model is a real fraction of the length of a given location 's output sheet, with the probability of first reducing the maximum value and then re-evaluating the index. Negative log-sum-exp is equal to a stable target score for real targets. This allows us to take the average value of the frame returned, compared to the sum of the points counted by location, and to find errors in label deviations, neglects, and the axis of return. Just look at the final number at one. “Reasonable range” They can't be found.

import math
def nll(logits, target):
    maximum = max(logits)
    return math.log(sum(math.exp(v-maximum) for v in logits))+maximum-logits[target]
assert abs(nll([0.0, 0.0], 1)-math.log(2)) < 1e-12
assert abs(nll([1000.0, 1000.0], 1)-math.log(2)) < 1e-12
batches = [(0.1, 1), (18.0, 9)]  # 每批的损失和、有效目标数
weighted = sum(s for s,n in batches)/sum(n for s,n in batches)
wrong = sum(s/n for s,n in batches)/len(batches)
assert abs(weighted-1.81) < 1e-12
assert abs(wrong-1.05) < 1e-12
print({'token_mean':weighted,'batch_mean_wrong':wrong,'ppl':math.exp(weighted)})

This item divides the document into a limited window with no overlapping objectives, rather than recalculating the entire history for each character. The adjacent window repeats an input character at the boundary, while the target is counted separately. The opening position of the latter window reads shorter prefixes, and the location number is again starting from zero. Therefore, “Each target is counted just once.” is not the same as the " Maximum history available for each object ". Changing the length of the window or changing to a sliding window rating may change the score, even if the check point remains constant; the evaluation report must preserve the window rule as part of the protocol.

Another place where the protocol can be changed quietly is by ignoring the label. The project ignores only the goal of a negative 100 number, while UNK is the valid target for a zero number. If the score is good enough, the unknown character is ignored, the hardest information disappears from the denominator. The “Known Subset of Characters” indicator can be defined in advance, but the overall number, the unknown rate and the original indicator should be retained at the same time, and the expected quality of the subset cannot be renamed as a whole.

The evaluation should also be calculated in conjunction with a change in the assessment model and a closing of the gradient record. The former affects the behaviour of the depopout class, while the latter affects the computational chart and memory; they are not the same switch. The retention of the training model would only close the gradient, which would change the scores of fixed check points by random masking. In turn, only the evaluation pattern would be evaluated, and the gradient could still be recorded and wasted.

Perplexity and Consistent Units for Comparison

Define Confusion $\mathrm{PPL}=e^L$. Replace the loss definition with a geometrical average of the number of real targets in each location. $V$ The probability of every real target is evenly predicted on a word. $1/V$, $L=\ln V$ ♪ And the confusion equals ♪ $V$. It's not the number of candidates actually listed at each step of the model; it's not evenly distributed, and it gets the same value. The index changes the logarithm for a scale and does not add new observations.

If the average loss is 1:81, the confusion is about 6:11; if it is 1:05, the confusion is about 2:86. The error of attribution in the previous section would make the model much better. Consider two probability scenarios of one quarter: loss is a natural logarithm four and confusion four. By combining the two positions with one “Characters”, the combined probability becomes one in sixteen, and the confusion of each new word becomes 16. The model does not change from the original text and the unit changes so that the confusion of the different words is not directly comparable to size.

The course uses a character vocabulary, but this does not automatically equal the length of each character encoding in the original text. BOS does not score, EOS scores, unknown characters are mapped to UNK; these engagements change the target space. By dividing nat/token with natural logarithm two, only bit/target characters are obtained. To report the bit/character or bit/UTF-8 by bits, it is necessary to specify the original denominator, termination cost and lossless coding of unknown characters. Turning multiple unknown words into a number would lose the information, and the low loss after folding could not claim the original compression capability.

So the project reports both an unknown ratio of authentication to test characters. Assuming that two of the ten test characters are not in the training list, they become UNK; even if the model puts the two goals in balance, it does not distinguish between the original two words. This is not a software anomaly, but rather a boundary. The fix can be done using codes covering all bytes or subwording in other courses; in the current experiment in this course, the honest approach is to retain the unknown rate and limit the conclusion about character language proficiency.

Ablations: What Exactly Changes in Each Experiment?

Decomposition refers to the controlled removal or replacement of a mechanism to observe predefined output changes. Here it is necessary to distinguish between the temporary closure of a branch for an already trained model, which is examined at the extent to which the model currently relies on this branch; the initialization training of a new model without the branch, and what this structure can learn under a given training agreement. The time for intervention is different and the damage in reasoning cannot be interpreted directly as “Another structure is unlikely to succeed.”.

For example, test learning locations are embedded. Protocol A sets the position vector zero at a fixed check point and the other parameters are not moved; protocol B recommends the engine and trains an unpositioned version from its head. The A-positioning model depends on the location channel, but creates an internal state outside of the training distribution. B compares the two training systems, but optimizes the trajectory, fitability and number of parameters may also change. The report must give the name of the protocol instead of synthesizing both experiments to “Take off your position and drop it.”.

Consider deleting the branch of attention: the amount of each step will decline. If both editions are trained for a thousand steps, with the same number of updates, they do not guarantee the same amount of budget calculations; if all are trained for one minute, a faster version may see more samples. Which is fair depends on the questions to be answered. Our first round is divided by fixed files, by words, by seeds, by batch order and by number of updates, and the number of parameters to be reported is time-consuming.

The same text may still be kept under different numbers. ID segregation and overall material can fix this input without automatically indicating duplicate information, let alone excluding close repetition of several words. Independent audits running in the project directory in the following paragraph are first standardised for Unicode NFC, which converts successive blanks into a space, and then regularizes the text by volume; it can detect precise repetitions under this clear rule.

import json,hashlib,unicodedata
from pathlib import Path
documents = [json.loads(line) for line in Path('data/documents.jsonl').read_text(encoding='utf-8').splitlines() if line.strip()]
seen = {}
for document in documents:
    normalized = ' '.join(unicodedata.normalize('NFC',document['text']).split())
    fingerprint = hashlib.sha256(normalized.encode('utf-8')).hexdigest()
    assert fingerprint not in seen, (seen.get(fingerprint), document['id'])
    seen[fingerprint] = document['id']
print({'documents':len(documents),'normalized_exact_duplicates':0,'near_duplicate_check':'not implemented'})

When the digestion is received by the project, the original check point is retained and the ablate module is called on: the default is only to copy the current model deep enough, the table of position is evaluated zero, in the same validation window; it does not cover the original weight. After the 40-step pairing has been opened, the arms start with the same initial state reset the whole random state and use separate samplers with seeds, window number tracks. No position arm permanently zero and freezes the table, eliminating confusion between "initial zeroing but subsequent re-learning positions". The arm has fewer training parameters, and reports both general and trainable parameters.

.venv/bin/python -m hero_transformer.ablate   --checkpoint runs/my-model/checkpoint.pt --batch-size 8 --threads 1   --seed 7 --retrain-steps 40 --lr 0.003   --output evidence/my-position-ablation.json

The report answers the reliance of the current model on location, and calls for initialization, data trajectory and updated budget training comparisons. Examination of the data trajectory is equal, zero position sheets are trained, original model weights are zero, and no intervention has been made to rewrite them, and then read the initial loss, effective target and actual time-consuming effects of each arm. Locational decomposition may increase, reduce or almost change points in this small data set; procedures are not subject to preset success. If one option is selected on this basis, then the missing information will have to be used after the option to freeze and the same validation cannot be reused as an independent certificate.

This check-point intervention only led to an increase in the validation of loss by about 0 001217, while the 40-step re-training zero-position arm was even lower. They were not contradictory: one asked whether the current parameters were dependent on location and the other asked what different training tracks received after a limited budget. The two re-tracking tracks dealt with 4676 valid optimized targets, window numbers, but with different training capacity, with only one seed and very small test set. Additional fixed sequence sensitive tasks, expanded document coverage and multi-twinning subexposures are needed to continue the testing of reasons.

Describing Variation Across a Finite Document Set

The average loss in the test set is the determined amount of statistics on this group of documents, but we are usually concerned with new documents from similar sources. Using adjacent characters as a completely independent sample, it ignores the subject and duplicate structure within a paragraph. This course uses the document as a sampling unit: each time an equal number of documents are extracted from an existing test file, recalculating the total loss by dividing the overall word number and repeating an empirical distribution. The long document still enters the final conclusion according to its target.

This is known as document level bootstrap, which is a approximation for assessing limited data fluctuations and not an unconditional guarantee of confidence. It assumes that the documents held represent the source of the target in the relevant sense and replace the unknown population with an empirical distribution; it is clear that three test files cannot cover a wide range of Chinese subjects. The percentage interval below describes only the extent to which the current sample is produced. A digital compartment is narrow enough to fix the source bias, content leakage or unknown folding.

import random
documents = [(2.0, 2), (18.0, 9), (8.0, 4)]
def token_mean(records):
    return sum(s for s,n in records)/sum(n for s,n in records)
rng = random.Random(17)
draws = sorted(token_mean([rng.choice(documents) for _ in documents]) for _ in range(2000))
lower, upper = draws[int(.025*(len(draws)-1))], draws[int(.975*(len(draws)-1))]
assert lower <= token_mean(documents) <= upper
assert abs(token_mean(documents)-28/15) < 1e-12
print({'observed':token_mean(documents),'document_resampling_range':[lower,upper]})

Connect Numerical Results to the Same Deliverable

The final result should be traced back to a freeze check point rather than to the most beautiful line in a log. Generating samples, confusion, gradients, and recovery checks must indicate whether they are the same code, configuration and data.

This chapter presents an evaluation list of document IDs, valid targets, losses and unknown characters, gives total loss and confusion, and at least one example of failure. Another pre-designed digestion, recording controls and changes. The next chapter will allow the same checkpoint to be freely generated, and verify whether the cache optimization is kept logits; that is the equivalence of the reasoning algorithms, which are two separate acceptance lines compared with the quality forecast for this chapter model.