Task: Add Information and Position Contracts to Content Retrieval
Each query in Chapter III can be viewed in its entirety and used directly for the next token training to read the target ahead of time. This chapter provides for a confluence of causes and effects, disassembly of multiple heads, and adds an absolute studyable location, eventually comparing the whole section with the cache forward. The data protocol is based on the real item data.py: UNK zero, BOS I, EOS II, training character Unicode, without independent PAD number; Enter the right-loaded EOS, with the target right filled − 100. The matrix index, softmax and local reverses that require pre-chapters are still not equal to the complete training model containing the normalized, forward feed and optimizers.
A batch size B, sequence location T, model width D, header H, each header width d=D/H. The position index starts from zero and the text for “PArticle t Okay.” refers to index t. D must be able to be removed by H. The input location t vector in this chapter corresponds to the already observed item t and is used to predict the next entry, so it is allowed to read the current position itself. All causal conclusions are discussed in one frontal direction of the fixed parameters; training to update shared parameters may change in another, of course, and this is not a future leakage of information along the calculation chart.
1. Define the Allowed Set Before Computing Softmax
A common sequence system can mix ji with “Keys j Validity”. This project has a narrower and clearer agreement: fill only right, and if the document is valid in length i <l, the fill key jl must have j>i, which has been excluded by cause. Therefore, the project forward does not receive any left fill or visible padding-key mask. Allowed of the reference function in this chapter really allows; Project causal_attention’s future really means forbidding the two to complement each other and cannot directly swap the Boolean array. Illegal entries are added to the list of negatives and then are re-established, calculating the equivalent of the legal scores first selected.
Take three places and the full number of zeroes: row zero permits [True,False,False], with the weight of [1,0,0]; row one permits [True,True,False], with the weight of [1/2,1/2,0]; row two, with the weight of one-third each. If you first clear all positions softmax, then line one becomes [1/3,0,0], with the weight and not the same definition.
If you change to < by mistake, there are no valid keys for position zero, and softmax has no denominator. The zero input for this task is BOS I, which allows for reading it without leaking the first character; the first character still exists only for the target. Effective query is readable at least, and the full mask line of the extra configuration is wrong. Right fill query itself will still be produced in project forward and not forced to zero; its target is -100, so do not participate in loss. This unsettled output cannot be confused with an effective empty line.
PAD is only filled here, not the fourth special token. Collate adds “雨” to input [1,4,2], target [4,2,−100]. The first two queries are valid and the third is not counted; the true EOS target 2 must be scored in its second position. Entering the last two does not trigger the global shield with its value, and the project does not treat EOS as padding_idx. Causation does not affect effective search by right filling, and target -100 does not allow these fill queries to be optimized. This reasoning is no longer applicable if left fills are replaced with a new mask-position contract; the current forward does not claim support.
2. Multiple Heads Change Subspaces, Not Time Order
Q 、 K 、 V Full-wide projection to output first B×T × D. Split the last dimension into H a continuous d-dimensional segment, and then move the header to the timeline, get B×H × T×d. The h header a coordinates corresponding to the original feature number h. d+a. The change of shape without the exchange of axes may result in a cross-reference of data over time to the same head, and the numerical shape may still be legal, and therefore be subject to index-by-index verification.
Each head is divided by its own d-dimensional point, and the legal key position is re-allocated. Get B×H × T×d. Then move the header back and spell it in its original sequence. B×T × D and by the output matrix W_O shape D×D 。 W_O No bias, responsible for mixing the results of different headings; it is not an oblivable demonstration. The number increases do not automatically teach each head a certain semantic meaning, and the division of labour requires observation and experimentation.
When a fixed model is wide-scale, four full-wide projectors have square parameters of the model ' s width. The first number does not automatically increase the total parameters of these matrices; what changes is that the subspaces per head read are united with independence. If the attention tables for each head are kept in a visible way, the head increase will increase that part of the memory. Therefore, head numbers, total width, and calculation costs must be accounted for separately, so that more heads alone cannot be seen to assume that the model is larger or faster.
Example 1: Two-Head Retrieval and Output Mixing
Close the position item temporarily to facilitate hand-counting. In this small matrix. BOS Embedded from Spring 3 as [1,0,0,1] 、 [0,1,1,0] ， D=4 、 H=2 、 d=2, all three are four-story units matrix. The first two-dimensional reading, the second two-dimensional reading. The zero query can only read itself. The collage results are still the same. [1,0,0,1]. Both keys are given in the first query. [0,1/√2] Set a=1/(1+exp(1/ √ 2 ))、 b=1−a. These manual vectors are used for arithmetical checks and are not embedded in projects that have been trained.
Value a is about 0.330238, b is about 0.669762. The first header is [1,0] and [0,1], the second header is [a,b]; the second header is [0,1] and [1,0] is the output [b,a], so it is combined with [a,b,b,a]. Note that both headcounts happen to have the same weight and do not represent multiple structure requirements for the same weight; they read different value coordinates.
Select the output map to turn [cnil,cOne.,cTwo.,cIII] into [cnil+cTwo.,cOne.+cIII,cnil−cTwo.,cOne.−cIII]. So the zero output is [1,1,1,−1], the first output is [1,1,a−b,b−a], and the last two are about [−0.339523,0.339523]. The weighted contour boundary is created only at each head output and can be negative or even larger after W_O, and is no longer “Probability vector”. This calculation also tests that the integration axis and output projection are actually used.
3. Adding Learned Absolute Positions to the Model
Use position sheet for main items P, shape as C×D ， C is the maximum number of places to express; it is embedded with token E Separate storage. If you enter the number x_bt, the absolute position is o_b+t, o is the length of which the prefix is valid before this input is entered E [x_bt] + P [o_b+t]. The two talents are equal to those of the width. “Seventh position.” It is not a token that requires model predictions and is not included in the list objectives.
This chapter program manually sets a small value for the position sheet, which only validates additions and indexes, and does not train it. When you are really in reverse, dX goes to two tables simultaneously: the character table is added to the token number and the position table is added to the absolute position. For example, the upper gradient of the first position of the two series in a batch is the same. [1,2] 、 [−3,1] Same P-0-line. Copy that. [−2,3]; the two lines of the character table receive their gradients if the two characters are different. Position sharing and character sharing are different index rules.
The absolute position table does not define the location beyond C and must either reject or adopt a declared extension strategy; modeling a duplicate location and cutting a transborder position into the last line will change the model. Some positions in the table, if never used in training, cannot be said to have learned a long text because of initialization of parameters. Learning is the choice of the project, not the conclusion that the relative location method is always worse.
Example 2: A Mask Does Not Automatically Resolve Every Ordering Problem
Consider the unplaced attention of a single layer, comparing [A.,B,c)] to the last position of [B,A.,c)]. The final query is the same, and the final line 's causal permission contains all three positions; the first two keys and values are only exchanged together. Softmax weights will be exchanged with the key, and the same values will be re-consumed by the same exchange, and the output will be identical. This particular structure cannot distinguish the order of the first two in the last row, even if it does not read out the future.
After adding the position, A-B exchanges carry different P lines, and the keys and values are usually not simply exchanged, so there is an opportunity to distinguish between the two prefixes. The program validates the difference in the values given. Do not expand this single inverse to "all mask no-position models are completely invisible": the causal permission itself is not symmetrical and multiple settings may have other sequential leads. What we need is a clear location with a verifiable modelling capability that cannot be declared complete by a triangle mask alone.
RoPE is another location scheme, mainly by rotating queries and keys to match the built-in volume, instead of adding this chapter P directly to the input. It requires additional fixed rotation pairs, frequencies, and relative dots. This chapter provides only a comparative reading portal, does not mix the second set of positions in the main project, nor does it substitute for the results of the original paper. Subsequent controlled digestion allows comparison between the two options under the same training conditions.
4. Causality Must Hold Across the Entire Computation
Under fixed parameters, initial X Line I relies only on this location token and the location number. The causal focus line i is calculated only from zero to i; the output projection is only mixed with the characteristics of the same location and therefore does not add a future path. The later location is re-allocated, the exterior and the residue can be retained in a layered manner if not read over time. This is the structure reasoning, and the future disturbance is the error tool in terms of actual realization, and both are mutually supportive but not the same evidence.
The future disturbance check fixed lengths and prefixes, replacing the suffix with a clearly different token or continuous vector, comparing all prefix output. Legitimate realization should remain constant in the float space. A version of the login permission is also run, confirming that changes will be observed in the same test; otherwise it may be falsely passed because input degradation, output overlayed by a constant or test location is wrong. Only check the upper right corner of the weight is zero, and no other leakages, such as upstream time-to-time reunification, can be detected.
5. Why Token-by-Token Caching Requires a Position Offset
K, V, where each head has a location, can be saved. New queries, keys and values are calculated for the new token only. For a fixed model that only results in reading, adding future token will not change the old position, so the old K, V, which is cached can be reused; multiple layers of models need to be saved separately at each level. The new location still needs to read all valid history instead of just new token. Cache is a calculation of recovery and cannot omit the information that the model would otherwise require.
If the cache has three valid tokens, the new token is three, not zero, and the location vector of index III should be added and the new key and value projected. Each time you enter one but reset the position zero, the location is misspelled and the price cannot be considered equal because the array length is one. If you move the same whole sequence forward and then enter the cache item by item, a comparison of each output will make an error. This chapter cache only deals with a single entry without filling the input stream; The project cache length is the stored key length and does not have a sample-by-sampled left-filled interface, nor should the right-filling suffix in the training be treated as a true prefix that needs to be renewed.
6. Independent Program: Separating Four Types of Evidence
import math
D=4; HEADS=2; d=D//HEADS
UNK, BOS, EOS, IGNORE = 0, 1, 2, -100
TOKENS=["<unk>","<bos>","<eos>","春","雨","风"]
# 索引与工程一致，数值是仅用于手算的固定嵌入。
E=[[40.,-30.,20.,-10.],[1.,0.,0.,1.],[.2,.1,.3,-.1],
[0.,1.,1.,0.],[.5,.5,-.5,.5],[1.,0.,0.,1.]]
P=[[.1*(t+1),.03*t,.07*(t%2),-.02*t] for t in range(8)]
WO=[[1.,0.,1.,0.],[0.,1.,0.,1.],[1.,0.,-1.,0.],[0.,1.,0.,-1.]]
def linear(x,W): return [sum(x[i]*W[i][j] for i in range(len(x))) for j in range(len(W[0]))]
def split(x): return [x[h*d:(h+1)*d] for h in range(HEADS)]
def masked_softmax(scores,allowed):
legal=[v for v,ok in zip(scores,allowed) if ok]
if not legal: raise ValueError("有效查询没有合法键")
m=max(legal); e=[math.exp(v-m) if ok else 0. for v,ok in zip(scores,allowed)]
return [v/sum(e) for v in e]
def read(q,keys,values,allowed):
scores=[sum(a*b for a,b in zip(q,k))/math.sqrt(d) for k in keys]
weights=masked_softmax(scores,allowed)
return [sum(w*v[j] for w,v in zip(weights,values)) for j in range(d)]
def embed(token,position,use_positions):
if not 0<=position<len(P): raise ValueError("位置越界")
return [a+(b if use_positions else 0.) for a,b in zip(E[token],P[position])]
def full(ids,use_positions=True,causal=True):
if not ids: raise ValueError("空输入")
X=[embed(token,t,use_positions) for t,token in enumerate(ids)]
# Q/K/V 为无偏置单位投影；无 padding mask 参数，所有查询均计算。
parts=[split(x) for x in X]; outputs=[]
for t in range(len(ids)):
allowed=[j<=t or not causal for j in range(len(ids))]
concat=[]
for h in range(HEADS):
keys=[row[h] for row in parts]
concat+=read(parts[t][h],keys,keys,allowed)
outputs.append(linear(concat,WO))
return outputs
def difference(A,B): return max(abs(a-b) for ra,rb in zip(A,B) for a,b in zip(ra,rb))
a=1/(1+math.exp(1/math.sqrt(2))); b=1-a
manual=[[1.,1.,1.,-1.],[1.,1.,a-b,b-a]]
assert difference(full([BOS,3],False),manual)<1e-12
base=full([BOS,3,4]); perturbed=full([BOS,3,UNK])
assert difference(base[:2],perturbed[:2])<1e-12
unmasked_difference=difference(full([BOS,3,4],causal=False)[:2],full([BOS,3,UNK],causal=False)[:2])
assert unmasked_difference>1e-6
# collate 的短文“雨”输入与目标；不能按 input==EOS 判断。
short_inputs=[BOS,4,EOS]; short_targets=[4,EOS,IGNORE]
short_valid=[target!=IGNORE for target in short_targets]
assert short_valid==[True,True,False]
assert [target for target in short_targets if target!=IGNORE]==[4,EOS]
assert difference(full(short_inputs)[:2],full([BOS,4]))<1e-12
assert difference(full(short_inputs)[:2],full([BOS,4,UNK])[:2])<1e-12
assert len(full(short_inputs))==3 # 填充查询输出存在，只是不计分。
try: masked_softmax([0.,1.],[False,False])
except ValueError: pass
else: raise AssertionError("全遮罩未被拒绝")
no_position_difference=difference([full([3,5,4],False)[-1]],[full([5,3,4],False)[-1]])
with_position_difference=difference([full([3,5,4])[-1]],[full([5,3,4])[-1]])
assert no_position_difference<1e-12 and with_position_difference>1e-6
def new_cache(): return {"keys":[[] for _ in range(HEADS)],"values":[[] for _ in range(HEADS)]}
def step(token,cache,position_override=None):
offset=len(cache["keys"][0])
position=offset if position_override is None else position_override
parts=split(embed(token,position,True)); concat=[]
for h in range(HEADS):
cache["keys"][h].append(parts[h][:]); cache["values"][h].append(parts[h][:])
concat+=read(parts[h],cache["keys"][h],cache["values"][h],[True]*(offset+1))
return linear(concat,WO)
cache=new_cache(); incremental=[step(token,cache) for token in [BOS,3,4]]
cache_error=difference(incremental,base)
assert cache_error<1e-12
bad_cache=new_cache(); step(BOS,bad_cache); step(3,bad_cache)
bad=[step(4,bad_cache,position_override=0)]
reset_position_error=difference(bad,[base[-1]])
assert reset_position_error>1e-6
print("two_head_manual_output",manual)
print("future_perturbation_error",difference(base[:2],perturbed[:2]))
print("unmasked_negative_control_change",unmasked_difference)
print("right_padding_valid",short_valid,"scored_targets",[4,EOS])
print("permutation_no_position",no_position_difference,"with_position",with_position_difference)
print("cache_equivalence_error",cache_error,"reset_position_error",reset_position_error)
print("all_causality_position_checks_passed")The program does not update the location table, project or word sheet parameters, which are proof of the structural check of the reference algorithm. It completes six examples of the list, right fill and target — 100, but does not add an unsupported valid parameter or left fill capability. Unit Q/K/V is a manual calculation, while chapter III retains a non-unit projection reverse reference. The actual master project uses self-written PyTorch decoder-only, preLN, GELU pre-feeds, absolute places for study and all projection training without nn. Transformer as an alternative to teaching.
7. Progressive Exercises and Solutions
TRY IT YOURSELF
Practice a single resize and output projection
After two counts, the first query is counted for two fractions, weights, reading vectors and final four-dimensional output at the first point. If D is used to replace d, in which direction is the weight changed?
Hint 1
Hint 2
Compare with the Solution
TRY IT YOURSELF
Practice two-bit fixes two confusing border errors.
Attaining the first position is only seen at j<i and using P-zero lines every time a new token is generated. Explains why it does not comply with this chapter contract separately. There is already four prefixes, which line table should be read by the new token. How can a second error be detected with independent reference?
Hint 1
Hint 2
Compare with the Solution
TRY IT YOURSELF
Practice three-packed design to distinguish between causality and sequencing.
Construct two sets of tests, the first one to test future visibility, and the second to test the degradation of the order of a single-storey no-position model in this chapter. Each group writes what it maintains, what it changes, what it should observe, and explains why one group cannot substitute for another.
Hint 1
Hint 2
Compare with the Solution
This Lesson's Contribution and the Next Lesson's Dependencies
The project now adds a valid readable, right-only fill agreement, multiheading, W_O, absolute location index, and prefix disturbance and cache equals. The fill check is determined by the trages100, the real EOS is retained. The next chapter puts the same calculation into the preLN debris and the biased GELU feed before allowing cross ellipse to enter all parameters. The whole model still has to be rerun for cause and effect check: the single layer does not introduce errors by not ensuring that later-created reclassifications, data windows or caches are introduced, and the length of the table of position, head count and vocabulary versions must be jointly saved.