Scores, Denoising Identities and Probability-flow ODEs

Task: Distinguish the Density Gradient, Mean Denoising, and Stochastic Reverse Drift

This chapter has a continuous time $r$ A positive process from data to noise, and a flow from noise to data from the next chapter $t$ Different. For a strict and moderate density $p_r(x)$, define score as $s_r(x)=\nabla_x\log p_r(x)$, that is, a logarithmic density gradient for space coordinates. It gives the fastest-growing direction for local densities, not for sample mass ratings, nor for probability itself. Different densities vary at different times, so score in the same position also depends on time.

Let's just think about the noise. $X_r=a_r Z+\sigma_r\epsilon$ of which $Z$ For clean data, $\epsilon$ For stand-alone standard normal, bar $a_r$ and $\sigma_r$ Known and Current $\sigma_r>0$. The fixed clean value $z$ After which the conditions density is mean $a_rz$ The co-op. $\sigma_r^2I$ Gaussian. For the second item of its logarithm, the condition is negative score divided by the variance, i.e. $-(x-a_rz)/\sigma_r^2$ and equals $-\epsilon/\sigma_r$ 。

1. Why Conditional Supervision Can Learn the Marginal Score

The margin density is a fraction of the distribution of the conditions density to the clean data. Assuming that the fractions and the fractions are exchanged, then the gradient is drawn on the fraction and divided by the marginal density. By multiplying each by the same condition density, clean data is observed. $x$ The later weights. The edge score is then equal to the after-check average of the condition score, rather than the non-entitlement of all clean sample conditions score. This reasoning explains both denomination and indicates that estimation at low density points may be influenced by the lack of a sample.

The second option simply takes the original noise-relations to the current point and replaces the noise mean with the first. If a function is trained by square error to approach the score label, the desired orthographic projection nature ensures the ideal optimal score. The actual network may have capacity and optimization errors, and the average training error is small and does not guarantee the accuracy of all space locations; in particular, sampling tracks are accumulated when they enter an area where training coverage is insufficient.

Divide when using noise network arguments score $\sigma_r$ It'll make score MSE equal to press $1/\sigma_r^2$ Weighted noise MSE. This weight may be large near the zero noise level, even if it is undefined at the end. The weighting of different times is not an insignificant detail of realization. We use these constant equations only at internal moments of the sound; if endpoint values exist, they should be negotiated by proven limits or individual models and cannot be directed to the optimizer.

Example 1: Mean Denoising Does Not Identify the Hidden Answer

Set a moment $a_r=0.6$ The difference in noise is 0:64 and the observation value is 0:06; a known accurate score here equals negative 05. Then the mean of noise conditions is negative times negative + zero + eight times negative + zero five, which is zero + four times negative + zero five. Clean conditions are equal to zero + six and zero six equal to about zero + four, six, six, six and six.

This check only proves that the average after-check meets the mean ratio, and does not prove that the clean point actually extracted is zero point four, six, six, six. At the centre of the data for two peaks, the later average may also be located in a low density area between two modes. Direct output of this mean usually causes excessive contraction in distribution; multi-step generation requires updating the total random distribution, rather than crushing each moment into one of the most likely answers.

2. Deriving Density Evolution from a Stochastic Differential Equation

Satisfied to random process $dX_r=f_r(X_r)dr+g_r dW_r$ Of which... $f_r$ It's a drift vector. $g_r$ It's a time-only metric diffusion factor. $W_r$ The typical measure of a random item is the square root of the time interval, not the time spacing itself. This does not involve a state-related diffusion matrix, which requires additional dispersion items and cannot directly follow the formula in this chapter.

From Random Displacement to the Density Equation, Term by Term

The space coordinates are x, the current time is r, and short intervals are recorded as r. The diffusion factor g_r is based on time; drifting is only possible. f_r(x) You can rely on position. Assuming that the coefficient is smooth enough, that there are limited local rectangulations in the process, and that the rest of Taylor is expected to be zero after r. Brown increment meets E. [ΔW] = 0 、 E [(ΔW)²] = r; this is not to replace random items with zero because the square items will be retained. X=f_r(x) Δ r+g_r The X-W plus the high rank.

Select a secondary micrometric function that is not density, but is used to observe any smoothness of random locations, such as location or position square. Expanding (x+X): a constant item (x), a regular item (x) and a second item of two parts (x) (x)2. A first expectation of the current x condition distribution, then a reduction (x), divided by r, and the limit. This limit definition is working on the function platitude of generating calculator L_r; it transforms an observation into a transient rate of change in the condition of the observation.

Now move the guide from the twig to the density. [f p φ] The boundary item is zero, which can be supported by a tight or require functions to decline long enough. One segment results in f(fp); the diffuse item makes two consecutive (p), two negative numbers offset. The definition is accompanied by a density of (f_r p)(g_r2/ 2 The asterisk is right here in relation to the accumulation, not the compound.

Finally, s=p'/p, or p'=ps, writes the diffuse item as (g2/2) (ps). So the density equation becomes r p=− [p(f−g²s/2)]', which is a continuous equation for the determinative speed of removal v=f−g 2 s/2. The original random process is not the same path as ODE; they have the same one-time margin under the same conditions of initial density, de-existence and density evolution as required. The conditions of existence and singularity are explicitly imported here, rather than proving all random process theorem at a time by Taylor.

Glossy Test Functions $\phi(x)$, second-tier Taylor starts with a condition expectation. A drift gives. $f\cdot\nabla\phi$ The second order of the Brown Increment is given in half. $g^2\Delta\phi$; the higher order disappears after the interval of time has become zero. For the current densities, and for the fractions when the boundary item has disappeared, the guide is moved from the test function to the density, and the Fokker-Planck equation is obtained. $\nabla\cdot$ is the dispersion of the vector field. $\Delta$ is the sum of the second steps of the coordinates.

So the certainty ODE speed. $v_r=f_r-\tfrac12g_r^2s_r$ The same density continuity equation is met, with the same single moment edge in the same starting distribution and sufficient regular conditions. This is called the probabilities stream ODE. It does not mean that a single ODE track is equal to the original random trajectory or that the combined time distribution of the two processes is identical. It is possible to construct different random paths to match the edges; this course evaluates the sample distribution with concern for the margin, but it is not possible to interpret the path without any difference.

3. Reverse Time and the One-Half Factor

Expanding the Reverse Generator: Where the Full Score Comes From

The reverse time u=R−r is increased, and short-step reordered under short-step Bayesian conditions is assigned to the current position. In a positive density and sufficient regularity, the local generation of this inverse process can be written as a companion set below. This combination formula is the result of the back-of-the-time analysis imported in this section; we then complete the implementation of the fractional algebra in it, instead of treating the normal inversion of time as proven by the basic guides.

The drifting part begins. -(fp)'=-(fp)-fp fp-fp}'. Then the spreading part: (p)=2; after subtracting, the constant density second leg disappears, leaving 2 pp' and p. Multiplies the original g2/2 factor of two is offset by half with Taylor, so that the g2p.

The reverse migration of SDE is — f+g 2s and retains proliferation g; the reverse probabilities stream ODE directly reverses to the original ODE time, drifting to — f+g 2 s/2 without the Brown item. Both can give back density, but the compensation for proliferation is different. If g relies on space, the first steps to move g2 out of range are ineffective, it adds a number of references, and the set cannot be applied.

To write the process from noise echo data, make new time $u=R-r$ It increases with the creation. Reverse proliferation retains its proliferation intensity and drifts to $-f_r+g_r^2s_r$. The calculator can be moved short by the Bayesian: the reverse test function produces a meta-metric equal to the associated generation of a meta-metric effect $p_r\phi$ Less $\phi$ Take it. $p_r$ I'm not going to be able to do that. $p_r$. The drifting item gives a negative effect when the multiplier lead is expanded $f_r$, La Plass cross-direction gives a whole piece $g_r^2\nabla\log p_r$ The remaining proliferation generation is the same. That explains why it's not half score.

This reasoning requires a positive density, a shift in available conditions, sufficient smoothness and no-problem boundaries. If you multiply the positive to random incremental numbers by negative numbers, you cannot get the correct conditions for the reverse Brown path; the new random noise is not the reverse of the original noise at the time of training.

Example 2: One Incorrect Coefficient Changes the Probability Flow

The current parameter for the extraction variance to maintain a positive path is $f(x)=-x$ 、 $g^2=2$; setting the current edge is a one-dimensional Gossic with an average value of zero, a difference of two and five points, so in $x=1$ The negative four is the negative four. The positive-probability flow is a negative one minus one times a negative four and a negative three; the inverse ODE speed is a negative three. The reverse drift is a negative seven to one plus two times a negative four, plus a new noise from root two of the diffusion coefficient. Negative three is not inconsistent with negative seven. They describe different processes.

The independent examination of the deviation guide: The difference rate is two times two times two plus five plus two to one five to one five for SDE. The probability flow is three times its own in this instant, with a variation of six times two plus five, which is also one point five. Half of the score compensates for the removal of the random diffusion item in ODE. If the whole score were used on the positive path to ODE, the variance would not be matched, and this check would be more perceptive than just observing a trajectory direction.

Another very positive comparison is the standard normal smoothing process: drifting to a negative current position, spreading square two, and currently score to a negative current. The positive-probability flow is therefore zero, and density is still constant; reverse drifting to a negative current position, which is based on proliferation and contraction balances, maintains the same normal state. If half of score is misdirected to a reverse SD, drift to zero and spread still exists, with the result that it has been degraded to the Brown movement. This counter-case does not require complex data to detect a mix of ODE and SDE coefficients.

If the new coordinates are equal to a positive sign, the density is replaced by an aberration constant because of the variable, and the logarithmic gradient is divided by the score size. It is not an arbitrary vector that can be moved directly from the data unit. The original coordinates ' score gets 10 times the magnified coordinates directly used, which makes noise correction inconsistent with the variable probabilities model; standardized parameters must be saved together with networks and samplers.

The theoretical relationship and procedure check should still be divided. The project checked the score formula for this task by automatically requesting a space gradient by parsing mixed logarithm density and comparing it with the responsibility line that was independently written. It did not train the score network, did not randomly reverse the spread of images, and did not prove that any neural vector field is a gradient of a density. By writing these more powerful features, one test would go beyond the evidence provided by the actual call chain.

from math import log,pi
def log_density(x):return -0.5*log(2*pi*0.25)-x*x/(2*0.25)
h=1e-5
score=(log_density(1+h)-log_density(1-h))/(2*h)
assert abs(score+4)<1e-8
f=-1;g2=2
forward_ode=f-0.5*g2*score
reverse_sde=-f+g2*score
assert abs(forward_ode-3)<1e-8
assert abs(reverse_sde+7)<1e-8
assert abs((-2*0.25+2)-6*0.25)<1e-12
print(score,forward_ode,reverse_sde)

# 确定性计算一阶短步矩；不靠有限Monte Carlo噪声掩盖系数错误。
variance=0.25;g2=2.0;h=1e-6
# 此刻f=-x、score=-x/variance=-4x。
forward_ode=-1+g2/(2*variance)  # 3
reverse_ode=-forward_ode       # -3
reverse_sde=1-g2/variance      # -7
rate_forward=((1+h*forward_ode)**2*variance-variance)/h
rate_ode=((1+h*reverse_ode)**2*variance-variance)/h
rate_sde=((1+h*reverse_sde)**2*variance+g2*h-variance)/h
wrong_reverse_sde=1-g2/(2*variance)  # 错把一整份score减成一半
rate_wrong=((1+h*wrong_reverse_sde)**2*variance+g2*h-variance)/h
assert abs(rate_forward-1.5)<2e-5
assert abs(rate_ode+1.5)<2e-5 and abs(rate_sde+1.5)<2e-5
assert abs(rate_wrong+1.5)>1.9
print(dict(forward=rate_forward,reverse_ode=rate_ode,reverse_sde=rate_sde,wrong=rate_wrong))

Exercises and Complete Solutions