From Probabilities to Vector Representations

Task: Trace Scores Back to Shared-Parameter Gradients

The previous chapter has fixed the X, moving target Y and valid tags. This chapter answers two connected things: how to obtain an optimal probabilities target after a position is exported; how to learn how to jointly modify the same vector when a character is repeated in different locations. We will personally deduce the fractional gradient of cross-entropy and then the inverse accumulation of the checklist. There is no complete model training here, and the final procedure is to test the independent numerical references of these local mechanisms.

The first step includes the condition probability connection of the previous chapter, the word sheet address and matrix shape, and the rule for the metric orientation: the index is still an index, the natural logarithm is the last number of its own variables, and the composite function is multiplied by the chain rule. A vector gradient is created by item on the coordinates. V Can you imagine the number of tokens? z=(z Zero,..., zend) is a location of logits, i.e., no actual fractions were assimilated; y is the number of the real next token, not the probability.

1. Why Output Scores Need Normalization

The fractions can be negative, greater than one, and there is no constraint on the sum of the fractions. Defining each category 's non-constrained weight is exp(z_i), divided by the weight of ownership and Z, gives a softmax probability. Because the index is positive and the number of candidates is limited, Z>0 has a probability of zero to one and is combined. This is a parameterization selected, not all taxonomic models can be constructed in this way. It allows models to redistribute probabilities only by changing relative scores.

Add constant to all fractions c, molecule becomes exp(c) exp(z_i) The denominators are the same. exp(c), so the probability of withdrawal is constant. m=max(z) I'll change the score. z−m The biggest index is one, and the rest is not more than one, and avoids directly looking for a large positive index. The equalisation is the exact algebra equivalent; the rounding on the machine is still in place. We use the original when we extrapolate the guides and avoid misusing the branches of the max in the achievement of the values as a new model mechanism.

Example 1: Matching Probability, Likelihood, and Stable Loss Numerically

For manual purposes, a local distribution of three types is used, with categories zero, one and two merely an index for this calculation, which does not correspond to UNK/BOS/EOS in the project; the true output width is still determined by the frozen phrases. Take three fractions [ln 2,0,−ln 2], the index being [2,1,1/2] and Z=7/2, so the probability is [4/7,2/7,1/7]. The real target is set to type 2, number y=1. This observation appears to be 2/7 under the model. The probability of maximizing this real goal is more detailed than whether the highest probability categories are correct to provide feedback; when the probability rises from one tenth to two tenths, the argmax may not change, but the model has already allocated more quality to real events.

Defines unit loss I=-ln p_y. Negative numbers give a pari passu value for minimizing loss, because the ind is strictly incremental. In this case l=-ln(2/7) = ln(7/2), about 1.252763. If the conditions of the complete document are multiplied by the probability of negative logarithms being added to the loss of position; this is obtained by the log number multiplier and the constant equation. Multiplication of documents is also based on a model that separates the document under parameters, not automatically derived from the character chain mode.

This difference is important in extreme scores. If the score is [1000,−1000] and the goal is the second, the stabilization index is [1,exp(−2000)], which may spill zero in normal double accuracy; the zero logarithm will fail. But the direct calculation on the right has lost about 2,000, which is still limited. A probability becomes zero in machine expression and does not mean that mathematical softmax actually gives it zero under a limited fraction.

2. Deriving the Cross-Entropy Gradient Step by Step

Set a fraction coordinates z_k first, and the other fractions remain static for a while. Z Only one. exp(z_k) This one changes with it, so... Z /∂ z_k=exp(z_k). By a logarithmic law, Z /∂ z_k=(1/Z)exp(z_k) =p_k. Again =z_y: k=y is negative. If you add two, you get p_k minus “k Is it the real target?” is the amount of instruction.

Unique heat target e_y is a V vector: item y is one, and the rest is zero. So the gradient vector can be written as p−e_y. This example has been obtained from [4/7,−5/7,1/7], which is zero. When the implementation gradient drops z new = z old gradients, the second score increases and the rest decreases. If 0.1 changes to [−2/35,1/14,−1/70]. This is only a step to temporarily treating the fractions as independent variables and does not represent all fractions in the actual network that can be independently updated; the network parameters also need to be followed by chain styles.

The gradient and the first independent check for zero: equalizing the whole number of parts without a loss, so the lead must be zero along the direction of the whole party. The second check is a limited difference in the centre: only item k is added to or reduced by h, divided by two loss lines of 2h, approximating the coordinates. It has a cut-off error and a round-up error, h cannot be taken lightly. Multiple h observations of errors are more credible than just one approach; the limited calculation is no substitute for the above decomposition.

If the distribution target q is used instead of the individual correct category, cross-entropy is defined as i q_i in i, requiring q_i to be non-negative and one. The monoheat q corresponds to the degradation cost chapter loss. Z Reasons q=1 Reservation once, with the result that p−q That's a sign. p−y This proverb must first be explained that y refers to monoheat or distributed vectors, and that integer categories cannot be numbered directly from each probabilistic vector.

3. From a Single-Position Loss to the Batch in Lesson One

Mark the position as a batch documentb and time t. Real items define valid tags m_bt by y_bt 100, take zero or one, N = bt m_bt>0. -100 is a missing mark, not a word sheet category, must be screened before loss can be calculated by category index. The batch uses an equal number of valid token rights. m and N are fixed data, so the valid position gradient is (p_bt-single heat target)/ N, while the missing location target loss corresponds to the logits gradient zero; the true target EOS II is still involved in the score.

If the total loss is fixed at ten, the correct average is two, divided by six to five six times two, or about one to six to six to seven. If the weights of each token in the short are equal to the length of the last chapter, each token weight is different from the length. Both training objectives can be defined and cannot be separated by using the same “Average loss” name on a log. The effective collection must be reported wrongly and the denominator must not be changed to one.

Example 2: Why One Character Has Only One Parameter Row

Embedded Table E Yes. V×D The first line of the learning matrix is the token number I D vector. For three input addresses [3,5,3] Take it D=2, and set E line 3 = [1,0] E, line 5 = [0,2] After checking the chart, H Three behaviors. [1,0] 、 [0,2] 、 [1,0]. Two number three uses the same parameter line, and does not reproduce two separate weights because they appear at different times. The input address is a discrete data, and no continuous guide is required for the address itself.

To see the reverse increment accurately, select a linear probe loss. J == sync, corrected by elderman == @elder_man C Three behaviors. [1,2] 、 [−1,1] 、 [3,−1]. Count three places first: one, two, two, three, total. J=6. Guided by a scale multiplied, each H The upper gradient is the equivalent. C Line E is used in line 0 and 2 and the gradient is [1,2] + [3,−1] = [4,1] and E-5 [−1,1]; other lines are not read, with a gradient of zero.

Defines the monother selection matrix S, with three rows of one (all indexed from 0) and the other zero, respectively, at 3, 5 and 3. The coordinates for H=SE are H_td=i S_ti E_id. For E_kd guidance, only i=k entries were reserved and obtained t S_tk C_td, so the overall gradient is S-subscribe to C. The gradient for the index 3 is therefore [4,1], the index 5 is [−1,1], and the unread index 4 is [0,0]. The search here is for shared parameter reverse rules, not empirical techniques. If the program uses attribute overlays, the index line of 3 retains only the last [3,−1], leaving out the contribution of the first location.

Independent inspection one: Direct check forms and monochrome arrays multiplied by the same H. Check two: Only row E zero-dimensional is disturbed, with a central difference of four for probe losses; if three is obtained, the reverse overlay is indicated. If number III is changed to four without changing the table, the task changes; reordering is only another expression of the same model if the word sheet is exchanged at the same time, the line is embedded and the output category is exported.

A chain relationship of the linear layer is also required for sharing embedded how to receive a true forecast loss. D;output matrices W Yes. D×V, score z_k=d h_d W_dk. The derived fraction gradient is g_k. A h_d is fixed, guided by a fraction, and obtained z_k/ h_d=W_dk, add all the paths passing through z_k and get l/ h_d= k g_k W_dk. The fixed parameter W_dk only corresponds to the number of components, so l/ W_dk=h_d g_k. The path is to reconcile multivariant issues to multiple metrics, and does not use the matrix as an unexplained black box.

Make a double sum: h=[1,2], W two acts [1,0,−1] and [0,1,0], so z=[1,2,−1]. For target group 2, the index is [exp(−1),1,exp(−3)], which is standardized to approximately [0.259496,0.705385,0.035119]. The fractional gradient is [0.259496,−0.294615,0.035119]. So the first-dimensional gradient is the third less, about 0.224377; the second-dimensional is the middle item, about −0.294615. The first and second lines of the W gradient are equal to g. If h comes from an embedded line, this vector gradient is added to that line; the contribution of the other positions of the same character will continue to be added.

The prediction of the distribution of its own entropy is weighted by the probability of negative logarithms in the model itself; this monitoring of cross-entropy is weighted by the single heat distribution of the true target. A model is extremely confident in selecting the wrong category, predicting that entropy can be small, but monitoring losses are large. Reducing uncertainty does not automatically mean improving correctness. This chapter is optimized on the basis of observing objectives rather than simply encouraging distribution variability; follow-up temperature sampling does not allow distribution to claim better training.

4. Why Vector Representations Still Need Position Information

Check the matrix to keep the rows in order, but the same token has the same vector before entering the context module. If the follow-up is only a sum or mean value for the line. [春,风] and [风,春] Get the same summary: both at the top case and below the vector [1,2]. Loss from symmetrical aggregation rules, no “Vector does not indicate order”. For self-direction without a location, no causal mask, there is also a corresponding sequencing of attention. Chapter III will calculate it directly, not by intuition.

The main project will use an absolute list of available learning places P, write the sign of position t E [Numbering_t] + P [t] Both of them. D The vector of a dimension to be added to the element. P Share it by location, E Shares by character; the gradients they acquire are added to the place where they are used. Chapter IV then achieves this path. The causal mask itself has asymmetric information and cannot be inferred from the above-mentioned equations as "all masks with no location are completely insensitive"; location mechanisms and information visibility are studied separately.

5. Independent Numerical Verification

import math

def softmax(z):
    m=max(z); e=[math.exp(v-m) for v in z]
    return [v/sum(e) for v in e]
def cross_entropy(z,y):
    m=max(z)
    return math.log(sum(math.exp(v-m) for v in z)) + (m-z[y])
def finite_difference(f,x,h):
    out=[]
    for k in range(len(x)):
        plus=x[:]; minus=x[:]
        plus[k]+=h; minus[k]-=h
        out.append((f(plus)-f(minus))/(2*h))
    return out
z=[math.log(2),0.,-math.log(2)]; y=1
prob=softmax(z); grad=[v-(i==y) for i,v in enumerate(prob)]
assert all(abs(a-b)<1e-12 for a,b in zip(prob,[4/7,2/7,1/7]))
assert abs(sum(grad))<1e-12
assert abs(cross_entropy(z,y)-math.log(3.5))<1e-12
assert abs(cross_entropy([1000.,-1000.],1)-2000.)<1e-12
assert max(abs(a-b) for a,b in zip(softmax(z),softmax([v+1000 for v in z])))<1e-12
errors=[]
for h in (1e-4,1e-5,1e-6):
    numeric=finite_difference(lambda v:cross_entropy(v,y),z,h)
    errors.append(max(abs(a-b) for a,b in zip(numeric,grad)))
assert max(errors)<1e-7

ids=[3,5,3]; V=6; D=2
E=[[0.,0.] for _ in range(V)]; E[3]=[1.,0.]; E[5]=[0.,2.]
C=[[1.,2.],[-1.,1.],[3.,-1.]]
H=[E[i][:] for i in ids]
S=[[float(i==k) for k in range(V)] for i in ids]
matmul=[[sum(S[t][i]*E[i][d] for i in range(V)) for d in range(D)] for t in range(3)]
assert H==matmul
analytic=[[0.,0.] for _ in range(V)]
for i,g in zip(ids,C):
    for d in range(D): analytic[i][d]+=g[d]
assert analytic[3]==[4.,1.] and analytic[5]==[-1.,1.]
def probe(value):
    table=[row[:] for row in E]; table[3][0]=value
    return sum(table[i][d]*C[t][d] for t,i in enumerate(ids) for d in range(D))
h=1e-6; embedding_fd=(probe(1+h)-probe(1-h))/(2*h)
assert abs(embedding_fd-4.)<1e-8
# 与 data.py 的 collate 相同：输入右填 EOS，目标右填 -100。
UNK, BOS, EOS, IGNORE = 0, 1, 2, -100
batch_inputs=[[1,3,5],[1,4,2]]
batch_targets=[[3,5,2],[4,2,-100]]
flat_targets=[target for row in batch_targets for target in row]
flat_logits=[[0.]*6 for _ in flat_targets]
valid_indices=[i for i,target in enumerate(flat_targets) if target != IGNORE]
assert len(valid_indices)==5
assert sum(flat_targets[i]==EOS for i in valid_indices)==2
batch_loss=sum(cross_entropy(flat_logits[i],flat_targets[i]) for i in valid_indices)/len(valid_indices)
assert abs(batch_loss-math.log(6))<1e-12
batch_gradient=[]
for logits,target in zip(flat_logits,flat_targets):
    if target == IGNORE:
        batch_gradient.append([0.]*6)
    else:
        batch_gradient.append([(p-(k==target))/len(valid_indices) for k,p in enumerate(softmax(logits))])
assert batch_gradient[-1]==[0.]*6
assert batch_gradient[2][EOS]<0 and batch_gradient[4][EOS]<0
print("project_batch_valid_targets",len(valid_indices),"scored_EOS",2,"loss",batch_loss)
print("probabilities",[round(v,8) for v in prob])
print("loss",round(cross_entropy(z,y),8))
print("logit_gradient",[round(v,8) for v in grad])
print("finite_difference_max_errors",errors)
print("embedding_row3_gradient",analytic[3],"numeric",embedding_fd)
print("all_probability_and_embedding_checks_passed")

These checks cover normal fractions, extreme scores, public flattening and shared indexes, which touch on different failure mechanisms. If the limited differential fails, first check the dissecting, disturbance location, recovery of the original parameter, and the loss denominator; do not immediately make the difference larger. If all formulas are correct but training is not improving, data, model connections, and optimizers are checked. This chapter has not yet attracted attention, and local values do not prove that the entire network is working.

Do not replace the loss of stability with a small constant that you want to add to probability. If the target is changed to “Probability plus constant”, when the real target is very likely to be very high in hours, the gradient will be significantly altered by this constant, and the optimised is no longer as if it were supposed to be. A stable algebra rewrite to keep the mathematical goal, and additional cut-off or smoothing requires a different definition of the target and the lead. The result of the error can be a limited number and does not mean that it has been achieved correctly.

6. Progressive Exercises and Solutions

This Lesson's Contribution and the Next Lesson's Dependencies

This chapter provides the project with stable softmax, cross-entropy with valid token scores, deconstruct fractional gradients and share-inverted rules. The next chapter generates Q, K, V from H, allows different locations to read each other according to content, and genuinely returns the loss gradient to these projections. Chapter IV combines the causal, multi-heading and location into the same structure. Prior to that, a mean vector categorizer could not be described as a complete sequence model, nor could it have been trained with pre-predictable loss proof of an imposter model.