Final Project: Deliver a Reproducible Transformer

Final Deliverables and Completion Criteria

The final delivery is the same character Transformer that has been continuously constructed in the first 11 chapters: starting with a document number and code, with self-decoding blocks, full parameter training, preservation restoration, independent rating and cache generation. You should be able to repeat from a new depressed directory, explain the meaning of each major load, and define the results using a failure check. If you can only display a single piece of text or run another person’s saved weight, but cannot re-establish the training and evaluation, this course will not be completed.

The final package is divided into two parts: a self-fulfilling and an experimental report. This should include data, configuration, source code, testing and necessary instructions. The report should contain the original indicator, the failure sample and the identity information of the corresponding check point. The course provides a complete reference project to help you match formulas and interfaces, but the reference achievement cannot be understood as running. It is recommended that you complete your own module by chapter, then send the same input to the reference page for comparison, save the first failed intermediate, rather than simply pasting a passable result at the end.

The course's default readers are already writing Python functions, lists, dictionaries and running local programs; linear algebras, probability and guide numbers are already in front. The project relies on PyTorch's volume operations and automatic guidance engines, but attention, fruit masks, decoding blocks and training organizations are prepared by us without pre-training models or n. Transformer. This is a small-scale realization of the complete architecture and engineering chain, not pre-training Chinese models using thousands of words.

Check the Environment and Trace Where Data Enters the System

Depress the project package into the directory where README is located. First, confirm that the interpreter version is at least Python3. 10; the course actually validates the use of Python 3. 12.14 and PyTorch2. 6.0. If the system defaults that python3 is too old to create an environment with an already compliant interpreter, then apply the interpretor within the environment in a uniform manner. Do not command Python or another part to use a new environment and then miscalculate the missing packages to model problems. The following commands do not download pre-training weights and rely on installation to complete, the sample data and all calculations are local.

python3 -c 'import sys; assert sys.version_info >= (3,10), sys.version'
python3 -m venv .venv
.venv/bin/python -m pip install -r requirements.txt
.venv/bin/python -m tests.run_tests --output evidence/my-tests.json

The project example contains 24 new short Chinese documents for the course, with stable numbers ranging from doc001 to doc024; titles are described only and the body of the document is trained. Distinguished files assign 18 trainings, three validations, three tests, and then only the text of the training constructs a sorting of the vocabulary. Do not treat the words " First create all the words and then split the document" as the same operation. For new language materials, the source and range are written, then the document identity is fixed, the division is checked and duplicated; different filenames do not preclude duplication of text.

Read the data and check the number of documents in the list, content and vocabulary to unknown characters. Each code is BOS, real character, EOS; the document breaks window, never starting with the end of the previous document. Each true character and each EOS is counted once in a full evaluation and the available context at the beginning of the window may be shorter. The window length and replacement are protocols that cannot be changed quietly after selecting the best result. The same agreement in Chapters VIII and IX remains valid in the final project.

Validation Stage 1: Each Component Computes What It Claims

Starts longer training by running numerical and behavioural tests. The focus tests not only check output shapes, but also check weights and weights for the future with a manual score, zero weights and value aggregations; the causal tests change future numbers and require prefix output to remain constant; each layer of the gradient tests receives a limited gradient compared to each length of the parameter, and the parameters are selected with a limited score at the float64 centre. “Received gradient” Only part of the connection is demonstrated and additional checks are required for the optimizers to actually update them.

The length, accuracy, and tolerance of the limited differentials should be recorded. The length of steps includes a clear cut-off error, and too small increases the reduced rounding effect; looking at several steps is more reliable than choosing just one pass. If the entire wrong target has been counted in the forward direction, automatic guidance and limited points may also pass together. Therefore, they must be taken back to the right goal in chapter II and the hand count in chapter III to prevent “The two procedures are identical.” from replacing “They're all solving the right problems.”.

Cache tests compare each step of the same prefix, covering word characters, multi-tokens, position deviations, window ceilings and slide windows. Saving recovery tests compares continuous training with a back-to-back training: not only to look at final loss, but also to compare parameters, optimizers, sample sorting, sampler locations, torch random status, and step-by-step Loses. The project opens dropout inspection recovery extraly to prevent an accidental equivalence only when there are no random branches. These two types of tests test different state machines and should not synthesize a green line without coverage.

When reading the test report, confirm that each case is actually executed, has not been skipped, and look at the failed information and source code Hashi. By covering only listed input and nature, the test does not deduce any data, device or accuracy. A practical countermeasure is to intentionally introduce known faults, such as removing future masks, resetting cache locations, leak saving sampler status, confirming that the corresponding test will fail and then be restored. This tests the ability to detect errors that it claims to have detected.

Validation Stage 2: Training on the Same Data and Parameters

The reference command uses CPU, single-line, thirty-two, two-storey, four-headed, window 16, pre-divide, eight batches. Small configurations reduce waiting costs, but all parameters of attention, foreplay, embedding, reclassification and output head are optimized. Optimizers are AdamW, learning zero zero zero, weight decline zero one, gradient paradigm is cut to one; no linear training on fixed attention features has been renamed full Transformer training.

.venv/bin/python -m hero_transformer.train   --output-dir runs/my-model --steps 160 --batch-size 8   --block-size 16 --d-model 32 --n-heads 4 --n-layers 2 --d-ff 128   --dropout 0 --lr 0.003 --seed 7 --threads 1 --eval-every 40

The training catalogue will get checkpoints.pt, metrics.json and run-manifest.json. First, check the logbook: when the list of examples is 3815, the total parameter is 50,000,368 and can be trained. Then look at the full training and validation points from the initial to the final model, rather than just the last random batch of Loses. The difficulty of sampling lots changes so that a single step down or up cannot directly substitute for an ex-post evaluation of the same fixed pool.

In a true reference operation retained for this course, the training set Loss dropped from 5.962669 to 2. 026304, and the validation set from 5.964481 to 5. 435705. The decline was very different, suggesting that the model 's preparation of this small training material is much more than an improvement on the unattractive information. This is the result of an explanation, not the “It's not good.” number that should be deleted. The training curve, file division and uncertainty are seen together to determine whether the next change of data, expression, capacity or optimization protocol should take place.

Validation Stage 3: Independent Scoring, Generation, and Recovery

.venv/bin/python -m hero_transformer.evaluate   --checkpoint runs/my-model/checkpoint.pt --split validation   --batch-size 8 --threads 1 --output runs/my-model/validation.json
.venv/bin/python -m hero_transformer.evaluate   --checkpoint runs/my-model/checkpoint.pt --split test   --batch-size 8 --threads 1 --output runs/my-model/test.json

The independent test for the reference operation is Loss 5. 460169, with a confusion of about 235. 137101; the test set has two hundred and twenty-eight original characters, plus three EOS, with a score of 231 points. Verifying the unknown character ratio is about 13.14 per cent, testing about 17.54 per cent. They are folded to UNK, and not all characters remain in the original character identity; therefore, the confusion cannot be ranked directly with other wordboxes or large models of open vocabulary. The figures refer only to the corresponding checkpoint and rating protocol within the package, and are not the generic constant that each machine should receive for each operation.

.venv/bin/python -m hero_transformer.generate   --checkpoint runs/my-model/checkpoint.pt --prompt 小禾   --max-new-tokens 60 --temperature 0 --threads 1   --output runs/my-model/greedy.json
.venv/bin/python -m hero_transformer.generate   --checkpoint runs/my-model/checkpoint.pt --prompt 小禾   --max-new-tokens 60 --temperature 0 --threads 1 --no-cache   --output runs/my-model/greedy-reference.json

The reference model 's greed for the reminder “Mio.” is exported to “Siu-wo give Siu-wo a check。”, followed by EOS. It has obvious repetitions and lack of consistency. Keeping this example is part of the project report: the model can correctly implement complete training and reasoning algorithms, while still lacking sufficient data and capacity. Changing the repeat example to a handwritten and fluid sentence would confuse software behavior with the editing product and render the receipt ahead meaningless.

.venv/bin/python -m hero_transformer.checkpoint   --checkpoint runs/my-model/checkpoint.pt   --output runs/my-model/checkpoint-summary.json
.venv/bin/python -m hero_transformer.train   --resume runs/my-model/checkpoint.pt --output-dir runs/my-model-continued   --steps 164 --threads 1 --eval-every 4

To test the accuracy of this continuation, the same initial setting of the separate directory is used to train directly to one hundred and sixty-four steps and to compare the two-sided state. This true CLI comparison is kept in the package with a maximum deviation of zero parameters, with optimisers, samplers, random status and the same list. This conclusion covers only the same CPU environment, fixed threads and recorded versions, not the same commitments across GPU, across frames. The quality table of the 160-step model, compared to the recovery of 164 steps here, should be indicated separately and not folded into a vague one. “Best model.” Achievements.

Validation Stage 4: Let Another Reader Reconstruct What You Ran

The first page of the report should be clear about the task, the reader ' s recantable command, model configuration, data source and division, environment, check-point identity and key results. Three types of material are then available: original output of correctness checks, training on fixed data and evaluation results, with examples of failure. At least one case of failure that has not been optimized is retained and an explanation of what it supports and what it cannot support.

Hashi used to bind these records to a clear document, which does not justify the algorithm or the material authorization. Using fixed sorting and coding for the configuration list to calculate SHA256 would avoid obtaining another identity simply because the dictionaries have different dictionaries; modifying seeds, phrases, document contents or structures would create different identities. Model weights, source numbers, and data each have their own habitat, and it is not relevant to call a transect " “Model receipts” ". The following example would merely standardize the list and not substitute for checking actual project documents.

import json,hashlib
def digest(manifest):
    encoded = json.dumps(manifest,sort_keys=True,ensure_ascii=False,separators=(',',':')).encode('utf-8')
    return hashlib.sha256(encoded).hexdigest()
a = {'seed':7,'config':{'width':32,'layers':2},'data_hash':'example-data'}
b = {'data_hash':'example-data','config':{'layers':2,'width':32},'seed':7}
changed = {**a,'seed':8}
assert digest(a)==digest(b)
assert digest(a)!=digest(changed)
print({'manifest':digest(a),'changed_seed':digest(changed)})

Self-evaluations can be divided between deduction, realization, experimentation and delivery, but meet the non-compensable threshold of causal failure, mislabelling, participation in training of test data, inconsistent recovery, reports and actual checkpoints, either of which should be repaired before total scores are discussed. The “Completed” button on the course page is just a log of your current browser and does not automatically prove acceptance through the project, nor does it grant credits or certificates to external course institutions.

By this point, chapter 12 forms a recastable closed circle: task definition determines labels, labels determine target, attention and location determines information path, decoding blocks and backpropagation decisions can be trained functions, training protocols determine parameter trajectory, evaluation and generation of tests for final behaviour. A subsequent expansion model or entry into courses from zero language models, after-training, efficient reasoning etc. can be replaced with a clear module and re-accepted along the same chain without having to rely on new presentations.