SFT: From Response Likelihood to Full-parameter Gradients

Task and Prerequisites

This chapter provides a fine-tuning of monitoring, SFT in English. The input is a reminder and a model answer to y. The task is to increase the probability of a demonstration. Readers need to have access to the four candidate strategies of the previous chapter, as well as to the one-digit index, logarithm and chain-style rules. We first push out the gradients for all eight parameters in a limited number of candidates, and then explain the demands and masks of the language model with many token answers. The chapter does not rename the four classification references as language models; it only tests the same condition in a minimal compute form.

For hint x, actiona logit is $z_a=b_a+w_ax$. Define probability $p_a=\exp(z_a)/\sum_j\exp(z_j)$. As long as the logit is limited, all probability is strict and one. Logit itself can be negative, larger than one, not credible; adding the same constant to all categories does not change softmax. This gradient describes parameters that are not the only direction to be determined by observation, but does not affect distribution and target definition. Values are achieved by subtracting the maximum logit, then calculating indexes and avoiding large positive spillovers.

From Likelihood to Cross-Entropy

A model category k appears to be $p_k$, negative log [N] $\ell=-z_k+\log\sum_j\exp(z_j)$. This is the precise algebraic variable: the softmax molecule denominators are logged, and the logit difference is given to the homogenization item. For jlogit, the guide to the homogenization item is $\exp(z_j)/\sum_i\exp(z_i)=p_j$ negative. $z_k$ The indicator of negative contribution, i.e. j equals k minus one. $\partial\ell/\partial z_j=p_j-\mathbf1[j=k]$. The sum of the gradients is zero, which corresponds to the common equalisation does not change the loss.

Then you can guide the actual parameters. $z_j$ Yeah. $b_j$ One, yes. $w_j$ The function is x, so the gradient is. $p_j$ Deflates the target indicator. The gradient is multiplied by x. The parameters of other categories also have gradients, which do not add points to the correct category alone, regardless of the denominator. The N model is averaged and the gradients are added by strip, with the N divided. The project achieves direct accumulation of the eight scales and then the learning rate is subtracted from the parameter; there is no automatic mini-black box or freezing of a layer of parameters.

Example 1: One Update for One Prompt

Sets x=2, all four logits are zero, and the true demonstration is No. 1 “Positive，Short”. Each probability is one quarter and loss log4. The gradient of preference is [0.25,−0.75,0.25,0.25]; the gradient is [0.5,−1.5,0.5,0.5]. If the initial all parameters are zero, the learning rate is 0 1, the updated deviation is [−0.025,0.075,−0.025,−0.025], the tilt rate is [−0.05,0.15,−0.05,−0.05]. The new logit under the same x is [−0.125,0.375,−0.125,−0.125].

The correct category is 0. 5 higher than the rest, so the new probability is $e^{0.5}/(e^{0.5}+3)$ About. 354661, greater than 0. 25; new loss approximately 1. 036592 This post is part of our special coverage Global Voices Online. 386294. The independent check is that four bias gradients are summated to zero each of the four slope gradients; the independent check is that either parameter is given a central limit difference, which is recalculated twice and should be close to the decomposition number. The limited difference is only valid for the point value and is not a substitute for the preceding decomposition.

Language Model Answer Conditions and Masks

If the answer consists of multiple tokens, the reminder is x, and the answer is y1 to yT in order to include the final EOS as the end goal. Conditional probability is multiplied by step by chain method: step read only to totoken; negative log is turned into T summation. This is not a block of four, but a vocabulary for each output location is sent to the same condition sequence.

Response-only refers to a reference to the answer and its EOS rating, which is used for conditionality and not loss. It does not delete the hint from the input, or change the task. Nor does it rate the position of “Enter appears to belong to answer” alone, because the first answer totoken is predicted by the location at the end of the reminder. The correct mask must be determined by the segment of the target, where input target is wronged and non-response target is set as a value. Sequence filling should also be excluded from the Los denominator, but true EOS does not rule out.

Example 2: Locating the First Response Token

The whole toy string is a reminder [10,11], separator [12], and answers [20,21], EOS [2]. Enter [10,11,12,20,21] with the target [11,12,20,21,2]. Response-only should be changed to [−100,−100,20,21,2]; valid mask is [0,0,1,1,1]. The third input position is 12 separator, but it is responsible for predicting the first answer 20, so it must count. If “Input is answer.token” is used to determine mask, 20 will be missed and an additional wrong location may be counted elsewhere.

Sets three true probabilities for responding to the target at 0.5, 0.25, 0.5, with a total NLL of four times log2, three negative logs at log2, two times log2, and four times log2, with an additional three targets, and approximately 0.924196. Directly averaged a five-point series with a lower average of about 0.554518, but simply placed two unrated points in the denominator, which is not a better model. EOS is number three, removing it will change the termination of learning.

The second independent inspection is an illustrative projection of: 12 forecasts 20, 20 projections 21, 21 projections 2, which covers and closes; 10 projections 11, 11 projections 12 with no indication and no rating. The first answer totoken and the final token are in the counter account. If you fill the batch right, the target negative 100 is completely unchecked; the valid count is given by mask or length and cannot be extrapolated by input number. These principles can be directly transferred to the decoder training, but only once per demonstration of the limited four candidate projects has a complete response to decision-making and does not contain a forged token mask interface.

Empirical Distributions, Duplicates, and Weights

The project 's example of the same x is two correct short, one long. If you ignore the sharing of parameters, the free distribution of the x is q and the corresponding experience frequency is 2/3 and 1/3. The average cross-collateral is q. $-(2/3)\log q_{brief}-(1/3)\log q_{verbose}$. A short q plus a long q with binding q is an extremely small one: two thirds of the negative q is short, and one third is short q less q, leaving a zero q short 2/3. This is the best that can be built only at the ideal border that excludes the wrong category or allows its probability to be zero, and limited softmax training is only close.

This example explains why the demonstration does not amount to preference. To push the correct shortness to 0.99 would make the correct long demonstration a very negative log, SFT might oppose further simplification; and the preferred data would clearly require that it be shorter than longer. The course would keep the two objectives separately and not press “Post-training effects” into the same Loss. If the true demonstration contains conflicting answers, SFT would combine the experience rather than automatically know which is the truth.

The answer of Dotoken also offers a different weighting option. If the answer length is one and nine, all token total NLLs are divided by the total length, which increases the overall weight of the answer; if the answer is averaged first, then the answer is equal. You can also give a larger typical gradient measure for each reply.

What Gradient Checks Can Establish

The project test adds one hundred thousandth to each of the eight parameters, recalculating the same fixed demonstration pool of losses and verifying the resolution gradient with a central difference, with a maximum error requirement of less than one hundred thousandth. Checks that all eight parameters change from random base to SFT. The first location deduction or realization error, while the second position gradient is calculated as an issue that has not been updated. All changes in the eight parameters do not prove the best for the global picture, nor does they guarantee that any arbitrary data will allow each parameter to be zero per step; certain gradients may be accurately offset when this task is symmetrically identified, and it is necessary to distinguish between reasonable zero values and break chains.

The SFT uses full-blue gradient decline, fixed learning rates of 0 15, 200 updates, no motion and random sampling, and therefore continues to train only on current parameters and fixed data, without the need to restore Adam's recalibration or data cursor. Randomity is only initiated evenly at small scales of the initial eight parameters; saved Seed and the initial parameters can recreate the starting point. This chapter does not assign this light state to the full Checkpoint needs for real large model training, which are also optimizers, schedulers, random sources and distribution status.

Validation of SFTs should see the full probability of leaving the same context, rather than just an argmax. A model may be correct at every time, but gives a lot of probability to answer; the preferred stage is about distribution quality. Conversely, a new context close to zero may not be confident enough because of sharing linear slopes, even if the distance from zero is low.

The delivery of this chapter is the SFT goal, the parsing gradient, the real full parameter update and answer mask rules. The next chapter no longer assumes “Label gives the only demonstration”, but uses winner/loser as a new data object at the end of the same hint. Reference has to be copied and frozen deep at the end of the SFT because it will be the fixed baseline for the latter DPO ratio; if it is updated with the strategy, the copy distribution obtained in this chapter will no longer assume the original reference role.

The mean denominator also determines the gradient scale. The two answers are three and twelve, respectively. If the local gradient of each token is the same, the answer total NLL averages four times the total range; the answer averages the same for each token. If the reply averages 15 tokens, each token, etc., but the total weight is one to four. All three objectives can be clearly implemented and “To avoid the weighting of long samples” cannot be taken as an unwritten default. The series of items selects a combined and even-handed response, allowing the full answer event definition of the DPO to correspond directly to the SFT account.

The positive item of the monitoring gradient for the wrong category on a single sample does not amount to “punishment of all undemonstrated answers forever”. The next demonstration of the same reminder may include another category as the correct entry, with an average experience frequency balance. If all undemonstrated responses are marked as absolute bad, then a stronger comparative monitoring than SFT is introduced. This course solves this difference through three models, and the winner/loser relationship favours data is provided separately in the next chapter.

The sequence SFTs respond to training along the real past, so they can fail on their error prefix at the time of generation. This is not a mask failure: the correct mask is still only used in the past, but the training was used as a demonstration and produced a strategy. Free generation and cessation should be retained as independent checks, and the low level of NTL per target cannot be directly translated into the whole section.

import math
x=2.; label=1; lr=.1
W=[[0.,0.] for _ in range(4)]
def loss(w):
    z=[b+s*x for b,s in w];a=max(z)
    return a+math.log(sum(math.exp(v-a) for v in z))-z[label]
grad=[[.25-float(i==label),x*(.25-float(i==label))] for i in range(4)]
errors=[]
for i in range(4):
    for j in range(2):
        a=[r.copy() for r in W];b=[r.copy() for r in W];a[i][j]+=1e-5;b[i][j]-=1e-5
        errors.append(abs((loss(a)-loss(b))/2e-5-grad[i][j]))
assert max(errors)<1e-9
new=[[v-lr*g for v,g in zip(row,gr)] for row,gr in zip(W,grad)]
assert loss(new)<loss(W)
target=[11,12,20,21,2]; response_target=[-100,-100,20,21,2]
assert [v for v in response_target if v!=-100]==[20,21,2]
print('before',loss(W),'after',loss(new),'max gradient error',max(errors),'response targets',response_target)