Quantization: From Rounding Errors to Model-quality Limits

Declare the Approximation Target Before Changing the Function

The previous cache requirement is the same function as the reference model, and the quantification replaces part of the real weight with a approximation on a limited grid. Its acceptance cannot be written only for “The program can run.” or “The output shape remains the same.”. This chapter defines a defined round-up map in the standard library, and then observes logits, distributions and loss changes on a copy of the freezing model. The project does not achieve an integer matrix multiplier, does not store the integer after wrapping, nor does it reduce the storage of actual float-point parameters. It is a pseudo-qual experiment used to isolate the source of error and cannot be derived from it to decipher the speed of the true quantification of the inner core.

Widen to k, assemble by symmetric integer $[-q_{max},q_{max}]$ of which $q_{max}=2^{k-1}-1$. This intentionally does not have a code with the most negative integer symbols so that there is a positive or negative symmetry. Give a limited weight to a group $w_i$, peak $a=\max_i|w_i|$, if a is not zero scale $s=a/q_{max}$ Integer $q_i$ Yes. $w_i/s$ the most recent integer and limit to range, restoring value to $\hat w_i=sq_i$. The zero-size group agrees that the scale is zero and zero. Python round and PyTorch round follow even-digit direction for the exact semi-integer, and both ends of this item are subject to the same agreement.

The recent rounding means that without the hyperbolic cut-off, $|w_i/s-q_i|\leq1/2$, by the right scale $|w_i-\hat w_i|\leq s/2$. The scale of the peak definition leaves the original limited value within the range, so that the static weight after training meets the boundary. If the scale comes from the calibration range and future activation of the transboundary boundary, clip introduces an extra error, this half-step length will no longer cover all inputs. It must be clear at the inference point that the boundary commitment to the fixed range of weights is safe.

Worked Example 1: Rounding and Outliers

For a group of rights with zero, half, and one, three symmetric ranges are negative three to three scales, one third. Zero maps are zero, one map is three and precise recovery; half is divided by a scale of one and five, and rounded to two by even number rules, and returns to two and six by six, which is exactly half a step long. The maximum error margin here is the real number near boundary, while the binary floating point is achieved with a slight indication of error, and the test should allow the machine ' s precision level to be measured. This is a partial error, which is not directly equal to the full model loss change.

The maximum absolute error is only zero two, and if one looks at a hundred relative errors, it will be all right. Small weights may be crucial in the direction in which they should be, but cannot be said to be insignificant because they are absolutely small. The separation of different range of output channels from the same scale would avoid the large value of a channel that drags down the resolution of all channels, but the discrete values within the same channel still exist.

def quantize(xs,bits):
    qmax=2**(bits-1)-1
    scale=max(map(abs,xs))/qmax if any(xs) else 1.
    qs=[max(-qmax,min(qmax,round(x/scale))) for x in xs]
    ys=[q*scale for q in qs]
    assert max(abs(x-y) for x,y in zip(xs,ys))<=scale/2+1e-12
    return scale,qs,ys
s,q,y=quantize([0.,.5,1.],3)
assert q==[0,2,3]
assert quantize([.1,.2,100.],8)[1]==[0,0,127]
assert quantize([0.,0.],4)==(1.,[0,0],[0.,0.])
print(s,y)

From Weight Error to Linear Output Error

Set a linear layer output as $y_j=\sum_i w_{ji}x_i+b_j$, enter fixed, non-quantified, the output error is $\Delta y_j=\sum_i\Delta w_{ji}x_i$. From a triangulation $|\Delta y_j|\leq\sum_i|\Delta w_{ji}||x_i|$. If the output line uses a common scale $s_j$ And it's unstoppable. There's more. $|\Delta y_j|\leq(s_j/2)\|x\|_1$. The input range magnifies the weight error; after multiple layers of non-linearity, reclassification and residue, the local boundary cannot be added directly to the end without conditions.

For example, an output channel scale of zero point zero two, with three and four input points, is zero point one times seven, i.e. zero seven. If the weight error is zero point zero five, negative zero point zero one, the actual output error is zero one-five plus zero zero zero four, i.e. zero five. This complete small example also shows that the symbol can be accumulated or offset; the average weight error is close to zero, which does not guarantee a given input output error. The calibration data should cover the true input range, while the random token of this experiment only validates the current fixed-value function.

Worked Example 2: Comparing Actual Four-Bit and Eight-Bit Models

The project quantifys all Linear weights by output channel, including Q 、 K 、 V 、 O, pre-dips and headers; embedding, normalization, bias. The restoration value still returns a copy of the double-precision model. The fixed 24-word, batch 2 proximate predictions have forty-six active positions. The eight largest logits are about zero, three, three, five, five, on average. KL NYO $3.78\times10^{-7}$ Forty-six positions are all the same. The four largest differences are about 0:053,344, the average KL is around 0:01,163 and the rate of consistency is 40 divided by 46, and about 86:96 per cent.

The same negative logarithm for the same experiment appears to be three points, four points, six, two, two, and four points, three points, four, four, four, five, eight, nine. The losses on these random targets are slightly lower, not indicating that the model is more linguistic or broader. Random initialization has no semantics, and random targets may occasionally favour arbitrary disturbances. These figures should be interpreted as proximate function remunerative differences, requiring the freezing of trained checkpoints, independent real files, same vocabulary and rating protocols, and reporting uncertainties.

from lab import make_model,tokens,quant_report,state_hash
model=make_model();before=state_hash(model)
for bits in [8,4]:
    report=quant_report(model,tokens(24,2),bits)
    assert report['ledger']['actual_storage_unchanged']
    print(bits,report['maximum_logit_error'],report['argmax_agreement'])
assert state_hash(model)==before

Ideal Bit Counts Are Not Measured Savings

This experiment is approximately 25,568 Linear weights, with a scale of 600,07 output channels. The desired weight load is 25,000,568 bytes if you wrap it up eight-bit, and 12,784 bytes if you wrap it closely. Assuming that each scale is saved in four bits, there are still 24,228 bytes, and no quantitative parameters, alignment or container are included. The current code is not packaged: the actual weight is still eight byte and the scale is used only to generate restoration values. So the report says that the condition book is a conditional account, and it's only the actual fact that it was achieved.

Real low cores must address integer, scale applications, possible zero points, clustering and alignment, and demonstrate that there is no overflow or misbroadcast. The weight quantification, activation quantification, KV quantification is a different location; this project covers only type one round-up recovery. SmoothQuant and GPTQ provide, respectively, more sophisticated calibration or compensation mechanisms, an independent method discussed in the original paper, which have not been achieved, and do not include simple line-by-line additions to their names. Quoted papers describe in-depth directions and institutional origins rather than their experimental results on other models and hardware.

The first type of validation is a local mathematical nature: zero recovery, integer range, uninterrupted half-step length and local linear error margin. The second category is end-to-end function examination: the original model Hashi remains constant, the candidate does change, the distribution differences with input, target losses and changes in decision-making. The local dimension, by not replacing the end-to-end check; anagmax consistency does not prove the same probability calibration. The next chapter will establish how these types of evidence together form the receiving and inspection threshold, and use smaller decision intervals to show that “It looks the same.” is still insufficient.

The symmetrical map also implies a semantic definition of zero points. If the integer multiplier is to directly replace the float-point multiplier, the cumulative result of each output channel needs to be restored on the appropriate scale when only weight is quantified and inputted; if input is quantified, the scale becomes the multiplier of the input scale versus weight scale. Non-zero points require correction of the deviations in the build-up medium. The theoretical load decrease does not automatically cause a low delay because different algorithms and hardware are able to integrate these steps. The current experiment is intended to restore floating-point values forward so that error studies do not mix kernel differences and the cost is that there is no real storage or calculation savings.

It can also be achieved by error checking the axis: PyTorch Linear shapes are output width multiplied by input width, so that each output channel should look for a peak value along the input axis and scale sizes at output width by one. If you seek a peak value along the output axis, it is still broadcastable at the scale of each input channel, but becomes another algorithm. A small matrix of clearly different lines should be checked, such as first and second row peaks by 100, to confirm that two steps are different. The test shape is not successful enough to prove particle size; These axes should be coded with formulae.

Exercises and Project Contributions

This chapter adds a full-scale error account with eight and four places in delivery. No empty operation has been submitted that actually equals “Quantification”, nor has the speed of the floating point restored to the program been considered as an integer nuclear advantage. Subsequent sections discuss the correctness threshold and performance statistics separately, using the same model and the same input.