From Byte-pair Statistics to Reversible BPE

Task and Prerequisites: Learn Representations Only from Training Text

The base byte code is fixed and complete, but allows a Han word to normally take more than one place, so that a limited window can accommodate shorter visible text. This chapter achieves a byte-to-code that is easy to verify item by item. Readers need to have the byte number, list scan and frequency count of the previous chapter. BPE This item does not use regular pre-cuts, does not set word boundaries, and does not duplicate any of the complete acts that are readily available tokenizer; the merger can pass through spaces or points, but must not cross the document boundary. This range is sufficient to study compressed, overlayed and experimental calibres.

Starts with two hundred and fifty-six bytes and two boundary instruments. $r$ The rule will be counted from zero and will be numbered $258+r$, save binary groups $(a_r,b_r)$. Its bytes are defined as two sequential links of existing segments. Special symbols do not appear in the training consolidation sequence and are therefore not merged; a rule can only refer to the original byte or to earlier-generated numbers, and cannot cite itself, future numbering or boundary numbering. The actual size of the vocabulary is 250 plus the actual rule number, not the user ' s request limit.

Four Deterministic Choices in Each Iteration

First, separate coding of each training document into the text bytes list. The second step counts all current adjacent pairs, each internal compound, but not matching the end of the previous article with the next beginning. The third step selects the largest frequency. If a column is added, selects the smallest numbering of the binary dictionary sequence. The fourth step scans each sequence from the left to the right, and the pair produces a new number and moves two cells, otherwise it produces the original number and moves one cell. The scan is performed on the old series, and the new generation of token is not matched again by the same wheel. Then the next round of recounting began.

“Count includes overlaps，Replace no overlap” is easy to confuse. The sequence [a,a,a] has two adjacent (a, a) but a legal replacement can only take the first two a, get [new,a], and cannot consume two intermediate elements at the same time and get [new,new]. Therefore, the maximum pair count is not the universal equivalent of the reduction in the length of the current cycle. Our plain achieves full proximity to the entire series and re-scanning it over a single round, at about the time of the consolidation times the current total length; It is clear, but not suitable for large-scale materials and cannot be called an industrial tokenizer training device.

Example 1: A Complete Calculation with Ties and Multiple Rounds

Training documents are... “abab” and “acac”, bytes a, b, c, 97, 98, 99. The first pair counts ab twice, b once, and the second ac twice, ca once. The maximum frequency ab and ac are both two, selected by dictionary order. 97, 98 Replace the two last two with [258,258] and [97,99,97,99]. The new count is 258, 258 One, ac twice, ca once, second round ac, new number 2509. [258,258] and [259,259] 。

The next round will only appear once, and the project will stop, even if the upper limit is thirty-two. So the actual vocabulary is 260, with eight original text bytes turned into four text token, with an average of two token bytes. When recoding “abac”, two orderly rules will be applied in turn. [258,259] will be obtained and added to the border to [256,258,259,257]. The two token bytes sequenced connection will be restored to aac. The independent inspection first is that the order of the manual rules is fully consistent with the realization, and the independent inspection two is to conduct round-trips for new combinations other than training, avoiding merely validating the memory mapping of the training chain.

Example 2: A Shorter but Irreversible Replacement

For “aaa”, if you export all the duplicates token aa, you get [258,258], decoded by “aaaa”, decoded by an additional one. The correct scan output [258,97], with a three-digit length, remains “aaa”. Another error is to take the new token immediately after a round of replacements and continue the unregistered merger with the next byte, which will make the encoding dependent on the control process rather than the saved rules. Tokenizer's summary, even if identical, would not save it if training and reasoning were different; it validated the subject of the rule and required a code version and a back-to-back test of the interpretation of the binding rules.

Reversibility can be explained. Each initial number represents a original byte, and the collage is the original document. Assuming that a wheel is right; replacing two adjacent segments as their connection, without changing the summation, or changing the order of other segments, the wheel is still correct. A limited number of rounds are repeated with the same bytes, and then the UTF-8 recovery text. This proof does not require consolidation as the highest frequency; the maximum frequency is simply a learning strategy. Inverted also: Token sequences that decode can be decoded are not necessarily the standard sequences selected by this wordbox encode.

from collections import Counter

def merge(seq,pair,new):
    out=[]; i=0
    while i<len(seq):
        if i+1<len(seq) and tuple(seq[i:i+2])==pair:
            out.append(new); i+=2
        else: out.append(seq[i]); i+=1
    return out
seqs=[list(b'abab'),list(b'acac')]; pieces={i:bytes([i]) for i in range(256)}; rules=[]
for rank in range(32):
    counts=Counter(p for s in seqs for p in zip(s,s[1:]))
    pair=min(counts,key=lambda p:(-counts[p],p))
    if counts[pair]<2: break
    new=258+rank; rules.append(pair); pieces[new]=pieces[pair[0]]+pieces[pair[1]]
    seqs=[merge(s,pair,new) for s in seqs]
assert rules==[(97,98),(97,99)]
assert seqs==[[258,258],[259,259]]
correct=merge([97,97,97],(97,97),258)
assert correct==[258,97]
assert b''.join((b'aa' if i==258 else bytes([i])) for i in correct)==b'aaa'
print('rules',rules,'encoded training',seqs,'overlap example',correct)

Why Rules Must Use Only the Training Set

Phrase learning does not have a visible predictive label, but it still fits. Assuming that training is repeated ab, testing is repeated only by xy; adding the test to the pair count will allow tokenizer to assign a short number in advance to xy, changing the length of the test, the amount of the parameter and the coverage of the context. Even if the weight is randomly initialized, the system has used the test distribution. [train] Gives it to fit, saves orderly rules and summaries; authenticates and tests calls encode, does not update statistics. “No test training model used” Must include tokenizer, not just optimizer.step without looking at the test loss.

If a merger learned during training does not appear in the test, it cannot be deleted because the number, output array and checkpoint are tied. If a new wordbox is needed, it should be considered a new experiment, re-initialized or clearly designed map migration; this course does not migrate. All base bytes remain, so that no characters are changed to unknown symbols, but more token may be required. The same data compression rate can be used as an indication of the volume of statistics, but it cannot be declared easier to predict on the basis of a high compression rate: it may also result in larger groups with more low frequency, making it difficult for limited data to estimate the probability of their conditions.

A Canonical Path Does Not Sum Over Every Encoding of the Text

BPE Saves a, b, and adds ab. Models may generate token [a,b] It can also be generated. [ab], the two decoding is the same bytes string. Training and evaluation only calculates the probability of the normal path selected by encoded. If the code code is for text ab [ab], the path probability is multiplied EOS Probability does not automatically include [a,b,EOS] probability. To get the probability of decoding the edge of the text, all legitimate token paths to decoding the text need to be found, usually at great cost. We failed to achieve that.

The cross-coding indicator in chapter VII therefore formally refers to “every original bytes of the normative path numbers containing EOS”. It is a useful experiment under a fixed protocol, but it cannot be called without any qualification as the true compressed file size and does not take into account the costs of tokenizer, models, metadata and specific encoder. This name is not a word of caution: if both paths give a higher probability to the same text, the margin probability will be greater than the single path probability, and the negative logarithm will vary. The definition of the subject for evaluation determines the effective scope of the conclusion, which must be retained in the subsequent chart.

This chapter item increment is sequenced ByteBPE: orderly rules, snippet tables, determination of Tie rules, encode/decode The next chapter will use these real lengths to build windows; the same short text may vary according to the name, so token numbers. “It's about a long story.” Can not justify discarding tail windows or equivalents on average.

The order of rules is not optional. Learn ab first, and then learn “abandc”'s merger abc. When encoded abc, the number of abb is changed to an intermediate number before the number of ab is identified in the second. If the rule is implemented in turn, the number of the intermediate is not visible and only the number of ab and c is eventually obtained. Decode can recover the original, so no code can be found to drift separately; and it should be asserted that the exact numbering sequence of the sample is specified. The project saves the rules in order of creation, restoring a different algorithm that is not sorted by the length of a segment or “Current maximum string priority”.

Token may contain only the first half of a UTF-8 character in the schematic table, because the pair statistics are organized by bits rather than code points. For example, combining two consecutive bytes can get a segment that cannot be strictly decoded by itself, but it is perfectly legal when connected to subsequent segments. So debuging the vocabulary does not require that each piece of pie be independently decoded; hexadecimal bytes should be read before being verified on the full document layer. If you decoded the bytes with an alternative character for printing convenience, you cannot rewrite the bytes with a replacement string, otherwise reversibility is destroyed.

The sequence shortens also change the model ' s valid text view. Set 24 locations, with base byte models looking at the history near the text bytes at best; BPE combines the same number of positions to cover more bytes, but the coverage varies according to content and cannot be referred to as a fixed byte by twenty-four characters. When comparing the wordboxes, the original text context from the model is also changed. To isolate the purely indicative factor, additional matching bytes or explicitly using the extended bytes as part of the word-splitting benefit; the small items are currently fixed only token windows and recognize this confusion in the report.

The consolidated budget also affects model parameters. For each additional token, unshared input tables have two more lines d in width with the output head, the added parameter is doubled. If the two combinations are fixed, d the actual parameter budget is different. The project does not interpret the thirty-two combination as “the word algorithm is better under strict same parameters”; here only demonstrates that training rules are recreated, coded and interfaces can be integrated into the complete model.

The sample units of frequency statistics also influence the rules. Two identical documents, if re-registered in training, double all of them; BPE does not know whether they should be weighed. The reader in chapter I only bans precise cross-assembly repetitions and does not delete internal duplications within the same training pools, so that language extensions must proactively define internal de-weighting policies. The current original text is different from text, clearly grouped and sufficiently authenticated algorithms; and cannot misread “I'll just watch.train” as having a reasonable source weight.

python -m unittest -v tests.test_pipeline.PipelineTests.test_bpe_exact_rules_and_tie tests.test_pipeline.PipelineTests.test_train_only_and_digest

# merge_pair
def merge_pair(ids,pair,new_id):
    raise NotImplementedError('bpe')

# fit_merges
def fit_merges(documents,max_merges):
    raise NotImplementedError('bpe')

# merge_pair
def merge_pair(ids,pair,new_id):
 out=[];i=0
 while i<len(ids):
  if i+1<len(ids) and tuple(ids[i:i+2])==tuple(pair):out.append(new_id);i+=2
  else:out.append(ids[i]);i+=1
 return out

# fit_merges
def fit_merges(documents,max_merges):
 if type(max_merges)is not int or max_merges<0:raise ValueError('nonnegative integer max_merges')
 rows=[list(text.encode('utf-8'))for text in documents];merges=[]
 for rank in range(max_merges):
  counts={}
  for row in rows:
   for i in range(len(row)-1):
    pair=tuple(row[i:i+2]);counts[pair]=counts.get(pair,0)+1
  if not counts or max(counts.values())<2:break
  pair=min(counts,key=lambda p:(-counts[p],p));new_id=258+rank;merges.append((pair,new_id));rows=[merge_pair(row,pair,new_id)for row in rows]
 return merges