Bytes, Unicode and Prediction Units

Task and Prerequisites: Encoding Every New Character

Chapter I has fixed the document and division. This chapter is to convert text into a clear integer sequence and to ensure that a new Chinese word or expression that does not appear in the training can still be encoded. Readers need an integer division, a binary shift and a hexadecimal; they do not require familiarity with Unicode. We only handle Python strings. UTF-8 coding text, which does not accept the isolated proxy point. The basic alphabet of this item is 250 possible bytes, numbers zero to two hundred fifty-five; BOS No. 256. EOS No. 257. Zero is a real byte. “Empty” Skipped. At this time the scale of the statement is two hundred fifty-eight, there are no unknown symbols, and there are no separate fill numbers.

The Unicode code is an abstract integer; UTF-8 is a code that converts the code point into one to four bytes. A visible font on the screen may also consist of multiple code points, such as letters added to accents. Thus, len(text), len(text.encode("utf-8")) is three different amounts from the length after word breaks. We keep the original text without automatic formalization, spaceing or re-branding; these operations change the byte identity and assessment denominators. If formalization is explicitly adopted in the future, it must be simultaneously edited and it must be acknowledged that the task has been transformed into a predictive formalized text. Encoding covers not “Understood.” for visual fonts, but only the reversible nature.

From Code Points to Bytes

In numbers. $u$ Represents a valid Unicode markup. ASCII range $0\le u\le127$ Directly by a byte; the larger value is assigned to several loads, with the first byte prefix indicating the total length of the first byte and the subsequent byte prefix fixed to the binary ten. For three bytes, the number of loads is four, six and six, for a total of sixteen. Place the high byte, the middle six in the second byte, the lower six in the third byte, and remove the prefix when decoding, and then recombine it; do not decode the coded ten-bit string into a number of numbers. The legal range also excludes proxy areas, over-coding and values that exceed the maximum number.

Example 1: Fully Decomposing the Character 春

“春” has a hexadecimal 6625, or twenty-six thousand one hundred and forty-nine. Four of them are six, and six of them are twenty-four and six of thirty-seven. The first byte is therefore 224 plus six equals 230; the second byte 128 plus twenty-four equals one hundred and fifty-two; the third byte 128 plus thirty-seven equals one hundred and sixty-five, and the hexadecimal system is E6, 98, A5. $6\times4096+24\times64+37=26149$, restores the original numbering point. The base token sequence for “春” is [230,152,165].

After Project Plus Border [256,230,152,165,257]. The input is the first four, the target is the last four: first from BOS Two hundred and thirty. One hundred and fifty-two, one hundred and sixty-five, final. EOS. The text has only one Chinese word, but there are four projection targets, because the end of the event also counts. The independent check is by Python. UTF-8 The code must produce the same three integer numbers, and the independent check two must restore one after the three bytes have been combined. “春”. The two examinations cover hand count and round trip, not just the length.

Example 2: Coverage Does Not Guarantee Every Generation Is Valid

Any validity UTF-8 Text can be broken into bytes, so it's not in the training. “🙂” It can still be encoded by four bytes without having to map it as an unknown symbol. But bytes models can produce any of the 250 or 56 bytes at each step. BOS Generate [255,257] ， EOS Previous FF Bytes on UTF-8 There is no legal explanation. A strict decoding will make a mistake. [230,257], it is an incomplete tribyte sequence, and it is also wrong to report. The two failures are illegal bytes and early terminations, respectively, and the word sheet covers no guarantee that none of them will happen.

The project reasoning saves the generated_ids as they are and returns to valid_utf8. If a strict decoding fails, it can be shown with an alternative character, but it has to be reported that an alternative text has occurred and cannot be used as a byte string actually generated by the model. For UTF-8 to be legally generated, additional limited state constraints are required to track several consecutive bytes and the first byte range, so that illegitimate candidates are hidden; the project does not achieve this constraint. Training gives a low probability of illegal retention in legal texts is an empirical effect and not a guarantee provided by a type of system.

Changing Probability Units Changes the Meaning of the Number

Set the same test text in common $B$ Original byte, coded in text $n$ One token, border EOS. $N=n+1$ a target. The total negative logarithm is $S$, in the amount of natural logarithmic information called nats. Average of each target $S/N$ With perplexity $\exp(S/N)$ relying on word-sharing;number of digits per original byte $S/(B\log2)$ . The text unit is the sine qua non for comparison, and then the window, boundary and sample must be unified.

If a fully even base model allocates the same probability to all two hundred and fifty-eight categories, each target NLL is $\log258$, about 5:553. For documents containing only “春”, the total NLL is four times that number divided by three original bytes of text. $\log2$ Then you get about 10:68 a bytes. It can be larger than eight, because the model also assigns two special categories the probability and pays for the end of an event.

Conversely, compressing a high-repeated phrase can make the text much less token, and even more likely for each token loss of the even model, but even less for each original byte. This is a trade-off between the size of the vocabulary and the length of the sequence. The actual model can also change the distribution of conditions, the amount of parameters and the amount of calculation, so that it cannot be predicted directly by the even baseline. BPE Merge to measure this trade-off and retain the base bytes model as a comparable point of departure for expression, rather than treating the wordbox as a black box without interpretation at the outset.

import math
u=ord('春')
parts=[224+(u>>12),128+((u>>6)&63),128+(u&63)]
assert parts==[230,152,165]==list('春'.encode('utf-8'))
assert ((parts[0]&15)<<12)+((parts[1]&63)<<6)+(parts[2]&63)==u
for text in ['春','A🙂','é','é']:
    raw=text.encode('utf-8')
    assert raw.decode('utf-8')==text
    print(repr(text),'codepoints',len(text),'bytes',len(raw))
for bad in [bytes([255]),bytes([230])]:
    try: bad.decode('utf-8')
    except UnicodeDecodeError: print('invalid bytes rejected',list(bad))
    else: raise AssertionError('should fail')
s=4*math.log(258)
print('uniform bits/raw-byte with EOS',round(s/(3*math.log(2)),6))

Strict Conditions for Boundaries, Padding, and Reversibility

BOS and EOS are not byte values and therefore do not collide with a control character in the text. Project decode skips these two special numbers, links the text totoken bytes sequentially, then strictly decoding. The function is a display and return tool: even if an error between sequences is inserted in EOS, it will skip and cannot claim that the syntax of the sequence boundary is valid on this basis. Generates cycles stop separately at the first EOS, where the text code is only once at the beginning and once at the end. In order to position a mistake as a specific responsibility, separate the expression function from the boundary rule.

The right input of this item uses the EOS number, but it only assumes a filler role, with no other PAD number. The target fill is negative, and the loss ignores these locations. The validity cannot be judged by “Is that right? EOS”, because the real EOS event must be counted as a target, and the actual boundary may also occur in the subsequent window. The actual window does not cross the document and is filled only to the right, and the result is that the right fill does not affect effective left-hand queries. Chapter IV will deduce this agreement from a precise index, where only symbols are frozen and no claim is made in favour of arbitrary left-hand filling.

Reversibility of the byte layer is a full-scale nature: for all input text within the area of application, decode(encode(text) is equal to the original text. The test cannot be exhaustive, but it can be combined with examples of tectonic proof and contrasting. The key to proof is that the UTF-8 code itself is reversible, basic token fully covers the bytes, and the next chapter merges only with adjacent bytes without changing the order.

This chapter item increment is a basic byte encoder with no training status and a long-form account. BPE The training text estimates the rules to be integrated in an orderly manner; it is necessary to save the original bytes of each new number while shortening the sequence and to use a summary to prevent the code rules from drifting between training and recovery. Readers should now be able to calculate the original bytes and boundary targets for any given example and avoid putting them behind them. “How much did you train? token” misread as “How many characters have you read?” 。

There is also a border that can be confused: three Latin letters in the text that look like “EOS” will only be coded as their normal ASCII bytes; it has nothing to do with special number 257. Conversely, two hundred and fifty-seven cannot naturally appear by single bytes, because the byte is the maximum 250. As long as the special symbol is injected in the integer layer, the text layer does not have to scan or replace a string spelling. This avoids the user ' s wrongly triggering a stop and makes the word for an empty string code clear: it still has BOS and EOS, with only one end forecast.

The basic byte sequence supports all bytes, while the training text supports only the legitimate UTF-8 subset. Neuronet softmax usually gives positive probability to all categories, so that illegal byte paths are not strictly zero for lack of concentration. Training in legal texts improves quality, but limited training does not automatically transform support collection into form language identifiers. If illegal generation is removed and then shown, the probability of the model itself is not changed, but only the selection of observed samples; Reporting of illegal rates therefore requires a fixed sampling protocol and sufficient samples, and the current single greedy sample only gives one test of legality.

A strict decoding would also be too long. For example, the ASCII slash would have been expressed in a single hexadecimal 2F; UTF- 8 is not allowed if the longer C0AF was used to indicate the same integer. Limiting the shortest code suggests that a valid code point has a defined byte that avoids the same mark being expressed by many original strings. The project relies on the standard library to achieve these byte legitimacy conditions in strict codec and does not prepare an incomplete codec that only checks “The prefix begins with ten.”.

python -m unittest -v tests.test_pipeline.PipelineTests.test_utf8_roundtrip_and_invalid_generation