Train a 2D Velocity Network and Verify Numerical Gradients

Task: Train a Neural Velocity Field Beyond the Analytic Teacher

This chapter brings the first six chapters interfaces to the full project projects/diffusion-models. The network input is a 2D current position and continuous time, the output is 2D speed; the training label is reduced directly by an independent sampling endpoint. The dissecting teacher does not engage in reverse transmission, but only measures field errors in validation. This difference determines whether we actually did machine learning: if the sampler actually uses the manual solution formula, the result cannot be called a neuronetwork training generation. This project also saves the output of both fields, but records clearly distinguish between learned and analytic_oracle.

The operating environment is fixed to Python at least 3:10, PyTorch at 2:60, and CPU one-way. Training is done to update all trainingable parameters using Adam, with no pre-training weights and no freeze on the hidden layer only for the final output layer. Since the task has only two-dimensional coordinates and synthetic distribution, the standard library and PyTorch are sufficient to download images, image encoders or remote services.

1. Tracking Tensors and Parameters

Set Batch Size As $B$. Both initial noise and target end shape $B\times2$, Time Shapes $B\times1$. The item spells two space coordinates, time itself, time sine and cosine into five-dimensional features; the angle of the sine and cosine is used $\pi t$. They provide non-linear time coordinates, but they are not semantic features to be learned. Enter the linear layers with a width of 64, SiLU activated, second homogeneous linear layer, SiLU, and then output two coordinates through the linear layer; three layers of linearity are biased.

The first weight is 64 times 5; the second layer 64 times 64; the output layer 2 times 64; and the number of active volumes varies according to the size of the batch. The number of parameters does not vary according to the size of the batch. Confusion between the two will lead to an erroneous estimate of the size of the model. The project unit tests a single parameter to measure the existence of a gradient, which is limited and non-zero.

2. Information Flow Through One Training Update

Trainers are first drawn from a separate local random number generator. $X_0,X_1,t$, based on the actual linear bridge construction of the current position and margin labels. The network forecast is averaged with the label's elemental square difference on the sample and coordinates, resulting in a mark loss. Clearing the last gradient and then transmitting it backwards, and then executing Adam's update. Verifying the use of fixed and different seed generation without participating in the updating of these parameters; this is not a document cut-off problem, but rather an independent synthesis of random samples and isolation of random ends.

Example 1: Calculating Loss and Output Gradients by Hand

Take two samples at a rate of minus one and minus two, with projections of zero and minus one, respectively. Four coordinates have negative ones, positive one, positive one, negative one, square one and four, divided by two times the size of the batch two, with loss of one. The output gradient is negative five, positive five, positive zero five, negative zero five, and negative zero five. If taken by coordinates and not average, the loss and gradient are doubled; it may still fall, so “Training can run.” cannot independently prove that it is consistent with the agreement.

If the last level is $v=Wh+b$, where hidden vectors $h$ From the previous layer, the weight gradient of a single sample is the output gradient and the external build-up of hidden vectors, which is the output gradient; the hidden gradient is the weight shift by the output gradient, which continues to spread through activation and the front layer. Stop hiding the gradient, pre-routing the hidden vector to normal values, or placing large segments forward into no_grad would undermine the full parameter learning. The limited difference should be directed at parameters that are genuinely involved in the loss and avoid default error dominance by using a two-precision minor disturbance.

3. Why Correct Conditional Training Does Not Drive MSE to Zero

The projection breakdown in chapter V tells us that the speed differentials of hidden endpoints constitute an undesirable part of the equation. Trainers see a single draw, but the network can only read the current position and time. Even if they know the edges precisely, it is not possible to guess all the different labels on each repeated input.

Example 2: A Positive Theoretical Loss Floor

Consider the standard normal of the initial endpoint of one dimension, average of two for the target end point, average of one for the end point, independent of the end point. At half the midpoint, the mean of the difference is two for the speed difference and two for the speed difference; it is zero for the current position. Because the single Gauss example combines the normal of the zero-coordinate difference, which means that the two measures are independent, the speed difference remains two for the current position. The best forecast is two for the location and the minimum MSE is two for each coordinates; wrongly expects it to be zero and the right field will fail as training.

This single Gaussian example does not directly treat the value two as an accurate general floor for a four-package mix. The project has 400096 fixed validation endpoints, and the label MSE, calculated by means of the solution field, is approximately 2:416,644; this is a measure of a limited sample. The ultimate web label MSE is about 2:4640, compared to the teacher ' s field MSE is about 00:05932. The cross-over in the limited sample is not necessarily zero, so it is not hard to require that the "net tag MSE minus teacher's label" be small enough to equal the field MSE.

4. Actual Run Records and Checkpoints

Frozen examples of training use 1,500 updates, lot sizes 2506, hidden width 64, learning rate zero zero-two, model initialization seed seven, training endpoint versus generator seed eight. Fixed validation end point for seeds is 10.07. The web label MSE falls from 5:455285 to 2:464000; the field MSE of a dissecting teacher falls from 2:624,367 to 0:5932. The total number of parameters and the number of training can be 406,744. These are the actual results of Project JSON, not indicative expectations.

Checkpoints save models, Adam status, update steps, model configuration, global torch random state, local training to generator status and training reports. Keeping this information helps review operations, but CLI does not currently provide precise training orders, so we do not claim to have achieved and tested the recovery interface. Evaluation commands read only the check points and check the previous and forward file fingerprints unchanged.

The validation losses in the training report are derived from synthetic endpoints that were not involved in the update. It does not prove that generalization of real world data is missing, nor is it equal to the correct distribution of samples. Sampling also encounters errors in solvers; local sites may be more biased in regions with less distribution; and individual training seeds do not show fluctuations in different initializations. Chapter VIII will lock this model and evaluate end distribution using multiple sampling seeds against failure, rather than continue selecting training versions to match indicators.

The limited differential test must freeze the same endpoint and time. If the parameter is re-smoked after minor disturbance, and another batch when reduced, the difference in loss mainly reflects a random sample rather than a guide. The item fixes a double sample, calculates the reverse transmission guide, then adds a weight to a negative five-point reduction of ten, using the same scale loss to make the central difference. Finally restores the weight to its original value. By verifying only the number of lead checked and the full parameter gradient path, the test does not prove that the training system has no other error.

SILU activates directly by multiplying rules to check local gradients: the sigmoid is divided by a negative input of an index and the guide is equal to sigmoid plus input multiplied by a sigmoid minus sigmoid. The activation also does not cut the gradient completely between negative areas, but deep weight combinations may still leave the gradient small. Actual values are limited and parameters are indeed updated, and they cannot be avoided because a smooth activation is selected. CPU line and fixed random seeds help to rerun. This course does not promise that cross-hardware versions will be sequentially aligned.

import torch
from flow2d.core import Config,VelocityNet,generator,pairs,interpolate
torch.set_num_threads(1);torch.manual_seed(7)
model=VelocityNet(Config(16))
optimizer=torch.optim.Adam(model.parameters(),lr=0.002)
rng=generator(8)
for step in range(5):
    x0,x1,t=pairs(32,rng);x,target=interpolate(x0,x1,t)
    loss=(model(x,t)-target).square().mean()
    optimizer.zero_grad(set_to_none=True);loss.backward()
    assert all(p.grad is not None for p in model.parameters())
    optimizer.step()
print("updates",5,"loss",float(loss.detach()))

Exercises and Complete Solutions