Dual Encoders, Contrastive Objectives and Parameter Training

Image–Text Encoding and a Bidirectional Contrastive Objective

Chapter II completes the no-training terminology baseline. The question now is: Can graphic pixels and text phrases enter the same comparable space by learning? Here, “Same.” coders jointly accept target bindings, rather than just equal output dimensions. This chapter calculates loss and gradients on two pairs of samples, then connects the actual volume image encoder to the text encoder. The main project does update all parameters, and there is no hand vector or category check sheet disguised as a learning sign.

Readers have met the matrix multiplying method, vector point accumulation, and basic guides. The chain-style approach is the calculus rule that is imported here: a composite function will lead to a local number of parameters multiplied by a dependent path and collected. PyTorch is responsible for performing this automatic fraction, and the course does not recreate the calculus points from the real number of publics, but will independently calculate the logit gradient in two pairs of samples and perform a central difference in a true network parameter and check the chart connection. A numerical check can only verify the selection point and cannot prove all input.

The input image is 32 times 32 RGB float-point arrays, the channel values are obtained by two hundred and fifty-five different original eight pixels, and the channel order is converted to a channel, high and wide. The image network is two steps two, three times three volumes of one filled, three times three volumes of 16 passages in three to eight, each using ReLU, which is the correct volume, and lastly flat and linearly mapd to 32 dimensions. The output size rules for filling and step bands are 32 to 16 to 8, so that the final linear layer is entered into 16 times eight, instead of being extrapolated by operation.

Understanding Convolution from One Output Pixel

Frame Conv2d uses interconnectivity: takes the input window to its current position, and then tries to comb the elements by multiplying them, without first turning them over. The mathematical material sometimes calls the first-to-rotation operation a volume, so you have to fix the agreement here. Set a single channel as 4×4, line 1 to 16; n [[1,2], [0,−1], no fill, no step one, no bias. Top left window [[1,2], [5,6] ♪ Give 1×1+2 × 2+0 × 5−1 × 6=−1; one cell to the right, window [[2,3], [6,7] Here. 2+6−7 = 1. The two numbers already indicate that the nuclear weight is shared and the window is still located differently.

Each output channel has a set of cores that overlay all input channels. For output channel o, position (i, j), the input channel c and nuclear coordinates (a, b) are combined, and the output channel is biased. If you fill p, step s, the input value is () si+a−p, sj+b−p), cross-border zero fills the model. The different channels of output have different cores, but the same core is shared in all space locations.

Into the first layer. H=32 、 p=1 、 k=3 、 s=2 ： floor((32+2−3) / 2 )+ 1=floor(15.5)+1 = 16 The second level will be H for 16. Got it. floor(7.5) + 1=8;width is the same. The first parameter is 8×3 × 3×3+8 = 224 Second floor. 16×8 × 3×3+16 = 1168, display linear layer as 32×(16 × 8×8)+32 = 32800. The image branch totals 34192, the text branch 1408, totals 35600. 8×8, the display layer input is also adjusted to the new size.

See how a weight is influenced by pixels. The output of one cell above the right equals one, and the reLU local guide number is one. A nuclear upper right corner is guided by W01, leaving only the window top right input 3 so that the output of the flow to W01 is three. A local gradient of the same weight is zero for the left output − 1, after ReLU. If the loss depends on multiple outputs, the gradient of the same share weight is added to the entire window; it is not a pixel of just one window. ReLU is innocuous at zero, the frame selects a sub-gradiental length that cannot be used as a smoothing guide for the location.

import torch
import torch.nn.functional as F
torch.set_num_threads(1)
X=torch.arange(1,17,dtype=torch.float64).reshape(1,1,4,4)
W=torch.tensor([[[[1.,2.],[0.,-1.]]]],dtype=torch.float64,requires_grad=True)
Y=F.conv2d(X,W)
expected=torch.tensor([[-1.,1.,3.],[7.,9.,11.],[15.,17.,19.]],dtype=torch.float64)
assert torch.equal(Y[0,0],expected)
loss=Y.relu()[0,0,0,1];loss.backward()
assert W.grad[0,0,0,1].item()==3.0
plus,minus=W.detach().clone(),W.detach().clone();h=1e-6
plus[0,0,0,1]+=h;minus[0,0,0,1]-=h
finite=(F.conv2d(X,plus).relu()[0,0,0,1]-F.conv2d(X,minus).relu()[0,0,0,1])/(2*h)
assert abs(finite.item()-3)<1e-8
# 不运行网络即可计算输出尺寸；用框架只做最后交叉检查。
size=lambda H,k,s,p:(H+2*p-k)//s+1
assert size(32,3,2,1)==16 and size(16,3,2,1)==8
print('output',Y.tolist(),'weight derivative',W.grad.tolist(),'finite',finite.item())

First layer one output overlay 3×3 Enter a window; the second layer combines three first tiers of two pixels, so that the effective sense of a second layer position is 3+(3−1) × 2=7, the second adjacent layer of output is in four pixels at a distance from the original map. The first two layers of the volume are shared in local cores, without ensuring that the later flat layers are constant: the first and second grids correspond to different linear weights. Chapter 7 flattency failure can be used as a mechanism assumption, but the sole reason for which the flats are to be reduced to controlled structures, which cannot be proved by the current failure alone.

Text input uses a strictly restricted vocabulary that only matches the four coloured terms: red, green, blue, yellow, three geometrical shapes, and small, large and Latin letters; unmatched plain Chinese is not automatically understood. Each phrase becomes an average word bag vector and is then mapped to 32 dimensions through both linear, hyperbolic and linear layers. Average means that duplicate and word sequence information may be lost, for example, by exchanging information. “Red” and “Small” The order does not change the vector. This is a model assumption, and not all natural language expressions have nothing to do with the word sequence.

. By dividing it by the amount of u and v of each o'C, the non-zero vector length is one; the library function avoids a threshold of zero at very small lengths, and the following contour deductions are established only in areas where the length is greater than the threshold. The two vector points are cosine similarities, negative one to one. Temperatures are fixed positive zero one, which is used to zoom in similarity, and the current code does not have any temperature parameters to learn.

The calculation also requires a stabilization of the value. Direct indexation of large fractions may spill, with one line of maximum value. $m$, reduce each $m$ And at the end. $m$. This is an algebra constant variant, not close to softmax; it avoids many spills with limited precision, but still has a default error. If there is an NN in a row or there is no limit to the amount of it, it needs to be rejected upstream or explicitly addressed, and it cannot be assumed that stable forms will repair bad input.

All training parameters of the network constitute vectors $\theta$ _, update target average loss of current 20 pairs of fixed training samples $L(\theta)$. The gradient drops only to ensure a partial explanation of the direction of the decline, which can be carried out at one step in the appropriate length and smooth adjacent area; ReLU networks are uncorroborated, and Adam is not here to get the best proof of the global picture. Real loss falls are observations of this limited trajectory and cannot be directly rolled out by the goal definition to learn all the attributes that are necessarily combined.

The batch contains B only graphic pairs. For image i, softmax indexes the text fractions to that line and selects them in order to obtain a distribution of the conditions for selecting text j from the list. The training label assigns the correct column to i, and the negative logarithm probability is punished for cases where the true pair is not selected. The line is used again as a text selection image. Double-direct loss is an average of two average negative log probabilities; this is just the goal of the closed matching task, and output is not the probability that a fact is established in reality.

Example 1: Loss and Every Logit Gradient for Two Sample Pairs

B is equal to two, and all four points are zero. The probability of a one-by-two for each row and column is one-half, with a single-way loss of log2 and a two-way average of log2 and not double because of double counting. Row losses on the score S are guided by negative-opportunity labels, softmax probability for logs and index entries; and lot sizes are eliminated. Columns are identical. The diagonal gradient of the two-way even point is a negative quarter, and the non-record is a positive quarter, so that the gradient drops by an increasing correct matching and reducing the mismatch points.

Checking the gradient per row and adding a constant to the even symmetric example indicates that the whole score table does not change any probability of belonging to one. The general situation is only full scale gradients and necessarily zero, and it is not possible to extrapolate from this special symmetric example that each row is always zero, because the verbs softmax combine with the other lines. The training tests check log2 and four gradients by item, so that only a comparison of loss decreases will miss the wrong direction.

import math
import torch
from rag_lab.encoder import symmetric_contrastive_loss
s=torch.zeros(2,2,dtype=torch.float64,requires_grad=True)
loss=symmetric_contrastive_loss(s);loss.backward()
assert abs(float(loss)-math.log(2))<1e-14
expected=torch.tensor([[-.25,.25],[.25,-.25]],dtype=torch.float64)
assert torch.allclose(s.grad,expected,atol=1e-14)
print(float(loss),s.grad.tolist())

Example 2: Why Gradients on Normalized Inputs Do More Than Increase Length

If u is equal to a, minor changes to a are also variable. The length gradient for a is a except r, i.e. u; the commercial law sets the number of guides in a unit matrix apart r, minus the sum of a and u plus r squared, and reduces uuu by r. The matrix removes the vector fraction. The goal in the single space is therefore mainly reoriented; a simple magnification does not increase the number of cosine fractions. Here r must be non-zero, and the library in the threshold area achieves another function.

Take two-dimensionala of three to four, r equals five, u of zero-six to zero-eight, text direction of v of one to zero, temperature of one. A point-six. Minus the weight of v in the direction of u and get zero-64 to negative-zero-48; with a further five-point gradient of zero-eight to negative-zero-nine-six. It is independent of a point zero.

The training set consists of 20 of the 24 colours, shapes, size combinations, and four combinations are set aside. Each basic word appears in the training, so it tests a combination of properties of limited synthetic domains that migrate without encountering a new word, a new background, or an open semantic understanding of the new object. Each step uses all 20 pairs as batches, the only label; if the two maps are identical, cross-entropy still forces them to compete for the only label on the corner, there will be false negatives and more positive targets than claims that training data is naturally free of problems.

Fixed random seed XXIII, CPU single-line, Adam learning rate zero point 033, with 220 steps updated. The model has 35,600 training parameters, and the first end of the training is about 3.0864 and 0.16733 cross-entropy. Independent tests examine the existence of a gradient for each parameter, which is limited and does not mean that the total absolute value is zero, and then double-accurate central points for a weight in the real text linear layer. The non-zero length of the ladder indicates that this image and text branch is connected to the target, and does not mean that each individual coordinate is changing at every step.

The results of the study must be saved together with a word-base. Of the fixed 24 candidate libraries, four have set aside two-way search for the whole package, but only four samples are trained and tested for shared colour prototypes, fixed background and centrifugal structures; this is not evidence of cross-distribution. Chapter VII places the same freezing model on a real-generated flat, dark, greyscale picture, observing an obvious failure. It cannot be replaced by a low level of training cross-causts, nor can it report only the best disturbance branch.