From Prediction Tasks to Data Contracts

Lesson Task: Give Each Training Position One Clear Target

This lesson starts before network training. Turn a set of source texts into a traceable document manifest, a character vocabulary fitted only on training text, and three aligned tables: inputs, targets, and valid positions. Every later model must follow this contract: predict the first character from the start marker, predict each subsequent character from the available prefix, and finally predict the end. A model can optimize the wrong task successfully—for example, by copying its input.

You should be comfortable with Python lists, dictionaries, and loops; multiplication and division of fractions; and matrix rows and columns. Probability, tokens, batches, and conditioning are defined when first used. The final project implements and trains every parameter of a custom decoder-only Transformer. This lesson builds its data interface. A lookup table or an isolated output-layer update is not complete Transformer training.

1. Derive Next-character Prediction from Joint Probability

Begin with a finite alphabet: the set of symbols allowed to appear. A token is a discrete unit produced by an encoder. This lesson uses character-level encoding, so an ordinary token corresponds to one Unicode character rather than a complete Chinese word. Let the observed characters be $x_1,\ldots,x_n$, preceded by the fixed start marker $x_0=\mathrm{BOS}$ and followed by $x_{n+1}=\mathrm{EOS}$. BOS represents the empty prefix and is not a random event the model must predict. EOS is the end event that the model must predict.

For events A and B with P(A)>0, conditional probability is defined by P(B|A)=P(A∩B)/P(A). Multiplying by P(A) gives P(A∩B)=P(A)P(B|A). Let A mean “the first character is 春” and B mean “the second character is 风.” This does not assume the characters are independent. If P(A)=0, the division is undefined. Every observed path in this example has positive probability; the next lesson discusses zero-probability cases through the loss.

Apply the same definition to a third event C: P(A∩B∩C)=P(A∩B)P(C|A∩B). Substituting the previous identity gives P(A)P(B|A)P(C|A∩B). Each new character is conditioned on all preceding events occurring together, and the expansion continues through EOS. This is the probability chain rule, an exact identity. It does not imply that the next character depends only on the most recent character.

The parameters θ represent the complete collection of numbers to be learned. Replacing the true distribution p with a computable qθ selects a model family; the chain rule does not guarantee that this model is correct. If the model reads only the latest W tokens, it uses a truncated conditional distribution and discards information from the full prefix. W is a resource and architecture choice, not evidence that natural language obeys a finite-memory law. Later lessons test the consequences on independent texts.

Worked Example 1: Sequence Probability and One-step Probability

Construct a small distribution for hand-counting purposes only, with output supported by “春、风、EOS”. Each line in the table gives the given three conditions probabilities under the prefix, both non-negative and one. The number here is a teaching set, not a measurement of a training model. We calculate only the event that “春风” ends immediately after the wind, so three factors are required, rather than just the probability of first or last characters.

The probability of getting “春” is 0:06, multiplied by the probability of having a wind at the time of spring, zero:05, the probability of having a spring wind at the time of spring, zero:3 for the first two characters; and the probability of having a complete event at 0:24, which is zero:3 if EOS is missing, which means “At least start with the spring wind.”, and also includes events that continue to be generated. If you add up to 0:06, 00:05, 00:08, it is 1:9 and more, which is also contrary to the rules for calculating joint events.

There are two independent inspections. The first one, line by line, confirms that each row is indeed one, avoiding three random scores as probabilities. The second comparison involves a relationship: it happens to be spring wind and ends with a sub-event starting with spring wind, so zero-two-four cannot exceed zero-three. This variable check does not depend on reprinting the same serial number.

2. Why the Vocabulary Must Come from Training Text Only

The datasets are divided into three parts: training, validation, testing. The training text is used to compile parameters and the vocabulary, validating the text for decision-making such as learning rates, training rotations, etc. The test text will not be counted until after these selections have been completed. The fraction units are first full original documents, and documents with identical content or rewrite should also be placed in the same group.

We use original text. “春风” “春雨” “雨” Training. “秋雨” It's a test. “雪风” Test. Same as the real project CharTokenizer.fit, fixed first. UNK=0 、 BOS=1 、 EOS=2, the characters that appear only in the training text are weighed and sorted by Unicode, with spring three, rain four, and wind five. The number is just an address, and the sorting rule is not available. “Rain is bigger than spring.” The semantics. Autumn and snow don't participate in charting. They enter what they already have. UNK. This sorting and special items must be aligned with the project file data.py and the check point.

Authentication text “秋雨” coded as [1,0,4,2]. Unknown ratio is counted only on the original ordinary character: an unknown divided by two characters equals one in two; BOS and EOS are not mixed into denominators. UNK combines different unseen characters and the information is irreversible, so a decoding of UNK cannot be described as a complete restoration of the original text. This limitation of the character-level scheme can be improved through by by bit coding, but the prediction unit must be recorded simultaneously when comparing losses.

The split rules and the vocabulary rules guard against two different leaks. The whole repeat entry training and testing will make the test more like a memory; from the validation set, a character table is created that uses the information of the distribution for model design. The latter does not necessarily provide a direct answer to each question, but does not correspond to the previously fixed Train-Only protocol. We need to state precisely which boundary was breached, rather than call all the questions “Cheating.”.

Worked Example 2: Inputs, Targets and Valid Positions

“春风” is fully numbered as [1,3,5,2]. Take the sequence other than the last entry for input X, with the exception of the first sequence as a target Y So... X = [1,3,5] ， Y = [3,5,2]. The zero position reads: BOS Forecast spring; first position readable BOS And spring, forecast wind; second position readable BOS Spring, wind, forecast. EOS. The whole training. Y Stored, does not allow the model to read it. The model’s readable prefix continues to be limited by the cause mask at the attention.

Take “雨”, full number [1,4,2], without filling [1,4], target [4,2]. The project collate is filled to three positions only and entered [1,4,2] with the objective [4,2,−100], validly marked from target not equal to − 100 obtained [1,1,0]; the long text is effectively marked [1,1,1]. Together, two items are valid projections. Enter the two values on the right to fill only, without adding the PAD category; target two remains the true EOS that must be counted. The position is not valid based on input==EOS.

If Y=X is wrongly ordered, the first location target becomes BOS, the second location target becomes spring, and the network learns to copy the current location, reducing its loss. The way to detect it is not by looking at the length of the array, because the error array is still three; the “Next item after the prefix.” should be checked by location. The other failure is to link two pieces directly to “Spring rain.”, allowing the rain to be predicted after the wind and the original document to be wiped out. Cross-document connections can be another training agreement, but not by removing boundaries while claiming that learning to go through documents.

3. Give Every Matrix Dimension a Meaning

Decrees B is the number of documents in the batch, T This post is part of our special coverage Syria Elections 2011. V This is the number of entries in the list. D is the width of the future embedded vector. X 、 Y, the shapes of the effective markings B×T ； X The elements are legal token numbers, Y The valid element is the target number, filled with elements other than the vocabulary - 100, validly marked as a boolean value. E Yes. V×D Press X I can get a check. B×T × D. Can't. Y where - 100 Check the chart. The final dimension of the future output layer is mapd as V, the shape of a logits that are not downgraded to the projected fractions is B×T × V 。

This batch of B=2, T=3, V=6 with 24 numbers for the provisional D=4 and 24 for the activation of E and thirty-six for the output score after the check sheet. The values are equal only by coincidence, and the parameters and activations are still different objects. The same embedded line can be read in multiple locations and only one parameter is stored. Adding a character will change the number of V and output categories and will not automatically increase the projected location of the original document; − 100 should not be classified as a seventh category.

The parameters are the number shared across samples, which will be modified by training; this batch input, target and valid tags are sample data and should not be updated by the optimiser. The numbering replacement provides a semantic check: exchange spring and wind numbers, and synchronizes the exchange of words, embedded lines and output categories, indicating that the task remains unchanged. If the program directly uses the number as a continuous size to participate in the regression, the value will be undermined. This check will be translated into an embedded checklist in the next chapter.

There is also a probability boundary that must be clarified: each condition is consolidated and does not alone guarantee that the unlimited generation will eventually encounter EOS. A process that always gives one share of the probability will never end. This leads to the probability of a limited path being assigned, and the probability that all limited documents will be added together has not yet been proved. The actual generation is set with an end-to-length cap; if the upper limit is imposed, it should be recorded as a cut, and the model cannot be falsely predicted as the end of the model.

A window with a limited growth degree is needed after long text, but the end of the window does not necessarily mean the end of the document. If there is another character in the original text, EOS should not be forged at the end of each window; BOS should also mean only the beginning of a genuine document and cannot be empty on an arbitrary pre-discrete prefix. The goal within the window is still given by the next item of the original sequence, and the model's read range is limited by the length of the context. This short text does not need to be cut off, the boundary synonyms are fixed, and the subsequent extension loader will check if it has changed the task.

4. Standalone Program: Build and Check the Contract

Only standard libraries are used below, and the re-engine-only sorting of entries and the right-filling synonyms of the collate are not imported into PyTorch. The content fingerprint and peer check are additional pre-data audits: SHA-256 is used to check whether the byte contents change and does not prove the permission or statement to be true; the duplicate copy of the example belongs to the same training group, to retain a copy after the re-entry. The true split_documents check document ID division, with additional inspection teams in this chapter not gathering together completely duplicated content; Don't think of semantic repetitious as exact hash has been solved.

import hashlib
import json

records = [
    {"id":"t1", "group":"g1", "split":"train", "text":"春风"},
    {"id":"t2", "group":"g2", "split":"train", "text":"春雨"},
    {"id":"t3", "group":"g3", "split":"train", "text":"雨"},
    {"id":"v1", "group":"g4", "split":"validation", "text":"秋雨"},
    {"id":"s1", "group":"g5", "split":"test", "text":"雪风"},
    {"id":"copy", "group":"g1", "split":"train", "text":"春风"},
]
for row in records:
    row["source"] = "课程原创短句"
    row["text"] = row["text"].replace("\r\n", "\n")
    row["sha256"] = hashlib.sha256(row["text"].encode("utf-8")).hexdigest()

group_split = {}; digest_split = {}; unique = []; seen = set()
for row in records:
    for key, registry in ((row["group"], group_split), (row["sha256"], digest_split)):
        if key in registry and registry[key] != row["split"]:
            raise ValueError("同组或相同文本跨越数据切分")
        registry[key] = row["split"]
    if row["sha256"] not in seen:
        unique.append(row); seen.add(row["sha256"])

UNK, BOS, EOS, IGNORE = 0, 1, 2, -100
characters = sorted({ch for row in unique if row["split"] == "train" for ch in row["text"]})
vocab = ["<unk>", "<bos>", "<eos>"] + characters
stoi = {ch:i for i,ch in enumerate(vocab)}
def encode(text):
    return [BOS] + [stoi.get(ch, UNK) for ch in text] + [EOS]
def make_batch(texts):
    if not texts: raise ValueError("不能拼接空批次")
    sequences = [encode(text) for text in texts]
    width = max(len(seq)-1 for seq in sequences)
    x=[]; y=[]
    for seq in sequences:
        n = len(seq)-1
        x.append(seq[:-1]+[EOS]*(width-n))
        y.append(seq[1:]+[IGNORE]*(width-n))
    valid = [[int(target != IGNORE) for target in row] for row in y]
    return x,y,valid

x,y,valid = make_batch(["春风", "雨"])
assert vocab == ["<unk>","<bos>","<eos>","春","雨","风"]
assert x == [[1,3,5],[1,4,2]] and y == [[3,5,2],[4,2,-100]]
assert valid == [[1,1,1],[1,1,0]] and sum(map(sum,valid)) == 5
assert encode("秋雨") == [1,0,4,2]
assert sum(target == EOS for row in y for target in row) == 2
for text, xr, yr, mr in zip(["春风","雨"],x,y,valid):
    seq=encode(text)
    for t,flag in enumerate(mr):
        assert flag == int(yr[t] != IGNORE)
        if flag: assert xr[t] == seq[t] and yr[t] == seq[t+1]
empty_x,empty_y,empty_valid=make_batch(["雨", ""])
assert empty_x == [[1,4],[1,2]] and empty_y == [[4,2],[2,-100]]
assert sum(map(sum,empty_valid)) == 3

rows = [[0.6,0.3,0.1],[0.2,0.5,0.3],[0.1,0.1,0.8]]
assert all(abs(sum(row)-1) < 1e-12 for row in rows)
prefix_probability = rows[0][0] * rows[1][1]
complete_probability = prefix_probability * rows[2][2]
assert abs(complete_probability-0.24) < 1e-12
assert complete_probability <= prefix_probability
vocab_hash=hashlib.sha256(json.dumps(vocab,ensure_ascii=False).encode()).hexdigest()
print("joint_probability", round(complete_probability, 6))
print("vocab", vocab)
print("X", x, "Y", y, "valid", valid)
print("valid_targets", sum(map(sum, valid)))
print("validation_unknown_rate", sum(ch not in stoi for ch in "秋雨") / len("秋雨"))
print("vocab_sha256", vocab_hash)
print("all_contract_checks_passed")

The procedure does not mean that the future model is cause-and-effect, or that all recent repetitions are eliminated by indicating the severing, dictionaries and move-by-separation contracts of the examples given in this chapter. To verify that the checker is indeed sensitive, the copy of the standard could be changed to test, which should be reported wrong before the vocabulary is created; and the Y should be changed to X, which should fail on a location-by-place basis. These known errors are positive contrasts, avoiding a false sense of security given by an always returned examiner.

5. Progressive Exercises and Worked Solutions

Project Increment and Interface to the Next Lesson

The list of documents with source, group and content fingerprints is kept, the token numeric table that has been frozen, and the synonyms for X, Y, valid tags. No neural network parameters have been updated or quality indicators trained. The next chapter will receive the numbering and target of B×T, defining how to turn each location 's V fraction into a probability, and then extrapolate the target to appear, cross-entropy and fraction gradient; and can learn to embed E, which will replace this chapter only. The latter attention still needs to be given to ensuring that “Only read prefixes” is correctly labelled as a substitute for causality.