Generation and Caching with the Same Model

From Ground-Truth Prefixes to Self-Generated Prefixes

This chapter uses the same model that has been retained in Chapter VIII and evaluated in Chapter IX. Each step before the rating can read the true past; only the user provides the beginning, and then only the words it has just selected. The goal is to achieve a stop-and-recoverable, verifiable generation cycle and reduce double counting. First, fix the causal focus, absolute position, softmax, and save recovery. Cache does not change the parameters of the model, nor should it replace the fractions it gives to the same prefix; the choice of strategy changes the output distribution and two things have to be tested separately.

First encodes the entry starting with the word sheet saved during training and puts a BOS in front of it. Sends the current input to the model. Logits at the last position indicates the next fraction of the word immediately following this prefix. Select a new number, add it to the sequence, then calculate the next step. The EOS is stopped; if there is no EOS at all times, the preset cap for the new word number stops. This upper limit is resource constraints, not the “The end of nature.” that the model predicts. The output record should distinguish between two causes of cessation.

Prefix numbering as $s_{1:t}$, the next word character numbered as $v\in\{0,\ldots,V-1\}$ model output. $z(s_{1:t})\in\mathbb R^V$, default probability is $p(v\mid s_{1:t})=\operatorname{softmax}(z)_v$. The circular state includes at least sequence, cache, amount generated and sample random status. If you enter a word that starts with an extra training term, the encoder can only use UNK; the interface displays the original word does not mean that the independent expression of that word is read inside the model.

Token Selection Rules Are Explicit Algorithms

Greed chooses to return the number of the maximum score. The same fractions may appear in parallel, requiring fixed rules such as the selection of the smallest number; otherwise, the two sets may succeed in a different sequence even if they are the same. The probability that greed maximizes only this step at the moment does not guarantee the greatest combined probability of the whole sequence, much less the correctness of the facts. The probability of subsequent conditions depends on the words already selected, so the question of progressive maximum partial search is different.

temperature $\tau$ is a strict scale. Separate all logits and make softmax, define sampling distribution $q_\tau$ ♪ When... $\tau=1$, the distribution is the same as the original distribution of the model; when the temperature drops, the fraction difference is magnified. The probability ratio for both numbers changes. $\exp((z_i-z_j)/\tau)$, which means that the change is due to a difference in relative scores, rather than adding factual evidence to a word. The project specifically defines temperature=0 as a branch of greed and cannot really be divided by zero.

At temperature equals two, the non-alignment weight equals the square of the three possibilities, divided by the sum of the square roots; the distribution is evener. When the temperature equals one-half, the weight equals the square of the probability: one-fourth, one-hundred-nine, one-zero-four, plus one-three-eight, so the new probability is about zero-five-five-seven, one-two-six-six, one-five-three-three-three-three. This hand count changes only the sample, and the original check point and its teacher's forced loss remains the target.

Top-k Keep the highest score again $k$ Numbers, form a collection. $A$. The other numbers are zero in probability when re-aligned on the reserved collection. If the original distribution is 0:05, 00:03, 00:02, get the first two. $0.5/0.8=0.625$ and $0.3/0.8=0.375$ Forget about the fragmented distribution of the re-alignment. If the correct answer is just outside the pool, the opportunity to create it becomes zero; cutbacks control randomity and do not serve as a guarantee of authenticity.

import math
def truncated_distribution(logits, temperature, k):
    assert temperature > 0 and 1 <= k <= len(logits)
    keep = sorted(range(len(logits)), key=lambda i:(-logits[i], i))[:k]
    shifted = [v/temperature for v in logits]
    maximum = max(shifted[i] for i in keep)
    weights = [math.exp(shifted[i]-maximum) if i in keep else 0.0 for i in range(len(logits))]
    total = sum(weights)
    return [v/total for v in weights]
logits = [math.log(.5), math.log(.3), math.log(.2)]
q = truncated_distribution(logits, 1.0, 2)
assert all(abs(a-b)<1e-12 for a,b in zip(q,[.625,.375,0.0]))
assert abs(sum(q)-1.0)<1e-12
print(q, truncated_distribution(logits,.5,3))

Turning a Causal Property into Reusable Intermediate Results

When you don't use the cache, we recalculate the entire window on each floor after adding a word. Q 、 K 、 V, and the middle state. But the calculation of a past position does not allow reading of the future: its input characters, location numbers and legal prefixes remain unchanged. The previous position is expressed in the first tier, with the same prefixing and assessment mode, fixed parameters, and the second layer readings are also read in the same form, and this argument can be repeated on a layer by layer. This is the re-use of the past. K 、 V The reason is not to guess them from experience. “I think it's something like that.” 。

Yeah. $l$ layer, the key and value of the cache that have been processed, in a shape of each $[B,h,t,d_h]$. The length of the new entry is $r$ . The same value is applied to the old key and the new key in a time frame. New query and length $t+r$ . The output is still only for new segments. $r$ Positions, then enter the next layer. Only at the end of each layer will K and V be returned to the caller.

Location number is the premise of this equivalence argument. If the old cache length is five, a new character should be used instead of starting from zero. If you enter two new characters, they are five and six: the first new query cannot read a second key, the second one can only read both. A partial two-by-two triangle mask does not correctly indicate five old keys at the same time. The project builds the query position with past_length add a new segment index and then compares it to all key positions.

First, the length of all layers of caches is the same; otherwise the first steps have to be conflicted; lots, head count, width per head and equipment must be identical. Second, values are assessed against value and dropout is closed. In training, the intermediate value of different random masks is used directly, destroying both forward parity and changing the gradient map. This course only uses caches as an optimist and does not subtly insert them into the training cycle imposed by teachers.

The first call to send the entire hint into the model, and the cache returned has covered these hints. When a new number is selected, only the new number is sent in and the old cache is carried; do not repeat the whole hint; or history is repeated. Selecting a number, and using it as the next input, is two moments before and after. If selection to EOS stops, there is no need to calculate the hidden state entered with EOS. With each step "Input length, Old Cache length, New Cache Length" records, you can independently detect the length error caused by repeated feeding.

The project returns to the stop_reason field: EOS is allowed to stop only and the current round is set to EOS, while other cases where the cap of the new word ends are Budget. Zero budget is also a Budget; if it is explicitly forbidden to stop EOS, even if the last number is just EOS, the reason cannot be recorded as eos. This allows the stop rule to be a testable state shift, rather than an after-action decoded text guessing why the procedure ends.

import math
vectors = [[1.,0.],[0.,1.],[1.,1.],[-1.,.5]]
def attend(query, keys, values):
    score = [sum(a*b for a,b in zip(query,key))/math.sqrt(2) for key in keys]
    maximum = max(score)
    weights = [math.exp(s-maximum) for s in score]
    weights = [w/sum(weights) for w in weights]
    return [sum(w*value[d] for w,value in zip(weights,values)) for d in range(2)]
cached_keys, cached_values = [], []
for end, vector in enumerate(vectors, 1):
    cached_keys.append(vector[:]); cached_values.append(vector[:])
    cached = attend(vector,cached_keys,cached_values)
    reference = attend(vectors[end-1],vectors[:end],vectors[:end])
    assert all(abs(a-b)<1e-12 for a,b in zip(cached,reference))
print('single-head identity-projection cache check passed')

Cache Memory Cost and the Computation It Saves

Defines each mark store $b$ Bytes, Layers $N$ Bulk $B$ Current Cache Length $t$ Total width of attention $d=h d_h$ One K and one V each, shared. $2Bhtd_h$ markers;multilayers and bytes, obtained $2NBtdb$. It does not include model weights, current queries, temporary fractions, distributor balances or frame objects. Using this formula as a memory of the entire process would systematically understate the need.

With a small configuration of two layers, lot 1, window 16, width 32 and floor 32, the cache is 81,000,922 bytes per mark, i.e. eight KiB. If eight samples are produced in parallel, 64 KiB will be created. If you change to 32 floors, width 4,966, length 40,966, lot 1 and volume 2, the normal multi-head cache is two GiBs. KiB and GiB convert to two and 30 dimensions, respectively, without directly reducing the number of inconsistent units.

The calculation also limits the model. Only the statistical matrix is multiplied by a fixed batch and the prefix for discussion is increased from one to the next. $m$ Never exceeded the upper limit. $C$ ;softmax, normalization, non-linearity, and visitation at other costs. $f$, for lengths $t$ The full forward, the projection and the amount of the forward feed is... $Nt(d^2+df)$, focus two matrix times the scale $Nt^2d$. Only fixed $f$ and $d$ proportion, for example, the four-fold width that this project is commonly used for can be summarized as $Ntd^2$. The length of all prefixes is required to be accumulated without delay and calculated from scratch: the sum of the attention items is $O(Nm^3d)$. Only one query is added to each step after the cache, but all past keys and values are still to be read, so this step is focused on $O(Ntd)$ sum $O(Nm^2d)$ 。

This is not “After the cache, attention becomes constant time.”. The longer the context, the more K, V will have to be addressed at each step; hardware may still be subject to bandwidth restrictions. The prefilling of first-time tips still has to be calculated for the whole period and does not mix or move the prefilling time into or out of the statistics, claiming that some cache strategy is all faster. The project values test whether the output is equal; the actual first word delay, word spacing and total time are performance issues that need to be measured separately.

A Full Window: The Critical Boundary for Lossless Optimization

The project uses a limited learning absolute position table. When the cache is full, then the words are added to the available location. The oldest K, V cannot be cut off directly, and the remaining left can be moved to a position where they are still valid: the position is embedded, the past hidden state may also contain information about the deleted characters. Even if a key 's location label is changed, no mixing has been removed from the front layer.

This project uses a clear limited window protocol: the window can be added incrementally; when the oldest term is phased out, the cache is empty and the final C number is fully recalculated from zero to continue. The same rules for window access are used without a cache reference. The two are the same algorithmic objectives. It cannot retain memories outside the window or claim unlimited context through the cache. If different location systems or loop memory are to be used, alternative models and training methods need to be defined, and not merely replace the container.

This also limits the application of the previous complexity conclusion: once this project is full, each step will have to be recalculated. $C$ ♪ Window of the attention of about $O(NC^2d)$ Linear and forward feed around $O(NC(d^2+df))$. Therefore, the cache income under the window cannot be extended to any longer than is necessary. $C$ Fixed, with only an increase in the number of steps generated, the total cost of the full window is almost linear, but the constant that is recalculated with the complete window; it is two different compartments from “Only one new query per step”. The performance report should include both the length of the hint, the full window moment and the number of recalculations.

The creation also shows that the probability is similar to the numbering. The calculation of the array of floating points may result in a very small logits error; if the two maximum values are almost equal, the error can change the result of greed. The item compares the maximum logits error of each step and the sequence of generation under fixed conditions; not all intermediate values can be considered equivalent simply because the final text is incidentally identical.

Sample recovery is also an additional state: Random Number Generator. Only integer seeds are recorded, and the number of extras is drawn between them, and the subsequent sequences are still different. The project is to generate local generator, so that no other code is used to steal the same random stream; while saving or re-selling samples, specific strategies and states are to be retained. The fixed feed does not fix the wrong position offset, the fixed output string is not proven to be distributed correctly, and the two should be examined differently.

Stops judging using special character numbers instead of searching for the letter EOS in normal text that has been decoded. The entered literal string, retention number, and readable display are three levels. The project generation interface processes a hint at once; while the model itself supports batch loads, it also generates requests of multiple lengths that require a line-by-line maintenance of survival status, cessation of cause and cache locations, and cannot end a line-by-line broadcast. The queue and successive batch processing of the distribution service is reserved for an efficient reasoning course, and does not mix the single request here with the correct conclusion.

This Lesson's Deliverable: Establish a Reference Before Enabling Optimizations

Maintains a reference branch, an incremental cache branch, which is recalculated at the time of submission; both read the same check point, vocabulary, input and selection rules. Check empty hints for BOS input, a new word, multiple new words, window boundaries, super-long hints and EOS stop. Record cache sizes, recalculations and fractional differences, before talking about speed. The next chapter analyses mechanisms to reduce KV head counts and avoid overt storage of large focus matrices; they cannot replace the location of this chapter with causal contracts.

# 沿用同一starter.py；第3/7章方法已经完成，现增加以下分支。
# causal_attention: query_positions = arange(Q) + query_offset
# TransformerLM.forward: 验证past各层形状/长度，positions从L开始
# 保留原无缓存路径与有效目标损失。

def causal_attention(query, key, value, query_offset=0, dropout=None):
    """[B,H,Q,D] × [B,H,K,D]; row i can see keys j <= offset+i.

    This intentionally uses matmul/softmax rather than a fused attention API.
    Probabilities are returned before any dropout so rows sum to one.
    """
    scores = query @ key.transpose(-2, -1) / math.sqrt(query.shape[-1])
    query_positions = torch.arange(query.shape[-2], device=query.device) + query_offset
    key_positions = torch.arange(key.shape[-2], device=key.device)
    future = key_positions.unsqueeze(0) > query_positions.unsqueeze(1)
    probabilities = torch.softmax(scores.masked_fill(future, float("-inf")), dim=-1)
    mixed_probabilities = probabilities if dropout is None else dropout(probabilities)
    return mixed_probabilities @ value, probabilities

# TransformerLM.forward（替换第7章的无缓存方法）
def forward(self, tokens, targets=None, past_key_values=None, use_cache=False):
        if tokens.ndim != 2 or tokens.dtype != torch.long or tokens.shape[1] == 0:
            raise ValueError("tokens must be a nonempty [batch,time] int64 tensor")
        batch, length = tokens.shape
        offset = 0
        if past_key_values is not None:
            if len(past_key_values) != self.config.n_layers:
                raise ValueError("past_key_values must contain one (K,V) pair per layer")
            lengths = []
            for key, value in past_key_values:
                expected = (batch, self.config.n_heads, key.shape[-2], self.config.d_model // self.config.n_heads)
                if tuple(key.shape) != expected or tuple(value.shape) != expected:
                    raise ValueError("invalid cached K/V shape")
                if key.device != tokens.device or value.device != tokens.device:
                    raise ValueError("cached K/V and tokens must be on the same device")
                lengths.append(key.shape[-2])
            if len(set(lengths)) != 1:
                raise ValueError("all layers must have the same cache length")
            offset = lengths[0]
        if offset + length > self.config.block_size:
            raise ValueError(
                f"cached length {offset} + input length {length} exceeds block_size "
                f"{self.config.block_size}; recompute a sliding window from position zero"
            )
        positions = torch.arange(offset, offset + length, device=tokens.device)
        hidden = self.embedding_dropout(
            self.token_embedding(tokens) + self.position_embedding(positions)
        )
        presents = []
        for index, block in enumerate(self.blocks):
            past = None if past_key_values is None else past_key_values[index]
            hidden, present = block(hidden, past, use_cache)
            if use_cache:
                presents.append(present)
        logits = self.head(self.final_norm(hidden))
        loss = None
        if targets is not None:
            if targets.shape != tokens.shape or targets.dtype != torch.long:
                raise ValueError("targets must match tokens shape and int64 dtype")
            valid_tokens = (targets != -100).sum()
            if valid_tokens == 0:
                raise ValueError("targets must contain at least one scored token")
            loss = F.cross_entropy(
                logits.reshape(-1, self.config.vocab_size), targets.reshape(-1),
                ignore_index=-100, reduction="sum",
            ) / valid_tokens
        return {"logits": logits, "loss": loss,
                "past_key_values": tuple(presents) if use_cache else None}