AI Hero Club

Multimodal RAG · Lesson 8

Marked Complete by You0 / 8

Saved in this browser only. Marking a lesson complete does not mean the project has passed verification.

Lesson Outline 3 subsections
Course Contents 8 lessons
Full course plan.
Multimodal RAG / Learning Plan ↗

LESSON 08 / 8Suggested study 150 minutes

Deliver a Reproducible Text–image Research Assistant

When starting the assistant from a clean directory, how do we verify that the index, model, answers and perturbation results use the same build?

3 Questions for This Lesson
  • Run the full interface between actual files and quoted answers
  • Resume evaluation and disturbance with the same checkpoint
  • Test all parameters, gradients, sources, leakages and frozen boundaries
Course Project ZIP ↓

From Source Files to a Reproducible Document Assistant

The last chapter consolidates the first seven chapters into an item that can run independently, instead of adding a demonstration that is not related to the text. The download catalogue contains rag_lab source, original, freeze model and training records, query and perturbation reports, tests, and dependencies. Students do not need to copy files from another course, or remote API keys, GPUs or pre-training weights. Readers have the minimum condition of python base, vectors, and guides; courses import PyTorch automatic fractions and volume counts and check their usage with a small sample, limited differentials and real tests.

The environment requires Python to be at least 3:12: NumPy 2:3:05 per se requires this range, not because most of the source code is simply written in support of Python 3:9. Requisites fixtorch 2:6, NumPy 2:3:05 and Pillow:2.3. Author ' s physical environment is Python 3:12, CPU single-wire. Reliance on installation requires a regular software source network, running and validating data are all in download packages and no path to running the outgoing original. Platform fluctuations may still affect the near parallels and not promote the byte-by-the-tem nature of any machine.

The file is clearly charged, but it is surrounded by the same flow of information. Data is responsible for the original PNG and Markdown generation, real pixel reading, fixed features, Chinese word-law units and document location extraction; encoder is responsible for real two-branch networks, two-way losses, training and freezing loading; retrieval is responsible for indexing, indexing, etc. BM25 、 RRF, overlay rehearsing, context selection, original authentication and support answer; run uses these connections as a single query and a fixed question evaluation; stress generates real disturbance from the original map and compares two branches. Tests to call these production functions directly, and does not create a single line that is only for passing.

files→entries→BM25/dual encoder→RRF→coverage rank→whole context→validate→answer/refuse\text{files}\to\text{entries}\to\text{BM25/dual encoder}\to\text{RRF}\to\text{coverage rank}\to\text{whole context}\to\text{validate}\to\text{answer/refuse}
Each arrow corresponds to a running function and an intermediate product with a type meaning; it does not put together the results of different courses.

Example 1: All Intermediate States of a Real Query

Run text queries in the root directory “Lend me the booth. Open”. The program loads the freeze model and indexes the actual files, the word is found on the loan counter to the original line, the branch does not select the image for queries about geometry in the domain, and the integration still keeps the document. Rehearsing confirms the theme and field-by-word matching, the character budget remains complete, answers the original hash and interval, and the final output value and reference. If the result is only a prewrite time, without these functions, the project contract is not fulfilled.

Run in the Project DirectoryThis is end-to-end API, does not read evaluation labels.
from pathlib import Path
from rag_lab.encoder import load_checkpoint
from rag_lab.retrieval import build_index
from rag_lab.run import query_once
root=Path('data/inputs'); checkpoint=Path('evidence/encoder/encoder.pt')
model,vocab,_=load_checkpoint(checkpoint)
index=build_index(root,checkpoint)
r=query_once('借阅台 开放时间',index,model,vocab,root)
assert r['result']['status']=='answered'
assert r['result']['answer']=='周二至周五十四点到十七点。'
assert r['result']['citations'][0]['source']=='documents/borrowing.md'
print(r['context_ids'],r['result'])

And then replace the query with a query. “Red Small Circle”, the same index also contains real pixel vectors, and the synthesiph bag can handle three properties, ultimately supports checking to verify pixel measurements item by item. Add queries to the list “Material” Refusal to answer; replace question with question “Test stations Open” They share the same entry query_once, and the difference is from information and contracts, not from hand-written answers to each question. The output candidate_scores, contact_ids and cities together form a detectable trajectory.

Commands the line to use the Python module entry directly, and the current work directory must be a project root directory. A single query uses python-m rag_lab.run-query, with 39 questions without query in the same module, and specifies a new output directory. Re-evaluation does not require re-evaluation, but with the available encoder.pt. README writes the student 's new report to runs/evaluation and does not cover the event. If students do wish to retrain, choose runs/encoder output to record the new check point hash and use it throughout all subsequent evaluations.

Terminal CommandThe installed Python environment is first activated in the project root directory; a new directory is ordered to maintain the freezing model and report.
python -m rag_lab.run --query '红色 小 圆形'
python -m rag_lab.run --output-dir runs/evaluation
python -m rag_lab.stress --output-dir runs/stress
python -m tests.run_tests

Example 2: Independent Training Versus Frozen Reproduction

Freezes the recovery reading of the author ' s same weight, re-retrieving files, arithmetic queries and disturbances, and checking that the results are consistent with the records; it verifies the chain of reasoning and the integrity of evidence. Retrains update 20 pairs of text starting with the designation of random seeds, with new weights, even if the same configuration is separate from the environment and the hash. Checkpoint torch to store the byte identity of the container may be affected by the serial details such as the name of the output file, and it is not possible to compare only the hash with model values; to distinguish file identities from parameters to be compared on a single sheet basis.

The author trains 220 steps, and tests the package for rapid integration and verification at another 80 steps in a temporary catalogue. The latter examines training losses, 20 training ID exclusions leave IDs, freezes reasoning without weighting, fixed word-based processes can be used instead of replacing the 220 step paper report with 80 steps. The results report and time of the test itself are written in evidence/test-reults and, to keep the original package completely read-only, should run on a decompressed copy.

The total number of parameters can also be checked independently. 8(3⋅3⋅3+1)=2248(3\cdot3\cdot3+1)=224 And the second is... 16(8⋅3⋅3+1)=116816(8\cdot3\cdot3+1)=1168;image linear layers are 32(1024+1)=3280032(1024+1)=32800. The word in the field is four colours, three shapes and two sizes, nine dimensions, plus an unknown entry of 10 dimensions, so the first layer of the text is 32(10+1)=35232(10+1)=352, the bottom level 32(32+1)=105632(32+1)=1056. Five plus one. 3560035600. The vocabulary cannot automatically expand the temporary unknown word in the query to the frozen vocabulary by only nine known units; otherwise, the network parameter shape and check point no longer match. This level-by-level calculation independently checks the program ' s parameter statistics.

Parameter gradient tests fix small batches with seeds, calculate losses and communicate in reverse using a two-precision network, requiring gradients to exist, limited, total absolute values to be positive. Select a weight for the text output layer, each with a small disturbance, divided by two steps of loss, and reconcile automatic fractions. The central difference is usually reduced in square steps away from ReLU rotations, but too small to be affected by float points; the current check of negative seven degrees at one times ten is not supported by full parameter gradients.

gFD=L(θ+εej)−L(θ−εej)2ε,ε=10−5g_{FD}=\frac{L(\theta+\varepsilon e_j)-L(\theta-\varepsilon e_j)}{2\varepsilon},\quad \varepsilon=10^{-5}
Theta is the full parameter, e_j changes only the selected coordinates; it is compared with an automatic fraction on the same batch using float64.

In order to avoid checking only on the temporary 80-step model, the project also has a read-only portal python-m rag_lab.audit. It reconstructs the initialization network based on the main check point and compares each of the same 220-step model with at least one coordinate change and limited values; and checks that the original sill training, pairing of tables and training of IDs are just 20 and four IDs. The audit does not update the model or replace the test model achievement with the main model.

Independent control has different effects. Zero-point two validates the loss log two and four gradients for the sample; BM25 hand-count the validation formula; RRF validates the re-allocation of rankings; actual document interval re-establish the validation reference; index check run data boundaries after the label table is deleted; change the hash to pass through the validation invalid rejection; unknown properties, conflict and untrustworthy notes validation support policy. The test covers these mechanisms and still does not constitute an assurance of random input safety, natural visual or generic dialogue quality.

Recapitulation reports keep the original failures. The nine files of neuromode 39 failed, only two identities were correctly moved in three perturbations, two failures were darkened, and there were a lot of unrecoverable color problems. Students who saw training Loses fall could miss the most valuable part. If a module is repaired, the old report should be retained, new source code, weight, input and all new indicators should be tied to the same version, and no perfect table should be allowed to choose a different model for each graph.

The project boundary is also specific. There is no OCR, ASR or audio; there are no real CLIP weights, natural image generalization, long-language retrieval, vector databases, LLM open generation or multitenant permission system; the context budget is a text character, not token. The model check point reads a credible project file using torch.load and cannot treat an unknown external pickle as a securely opened data stream. The course is designed to recreate multimodel search and evidence processes, and these unrealized capabilities should not be hidden in a single sentence “Extendable” that is supposed to be delivered.

The completion criteria are not to repeat the names RAG, CLIP and RRF, but to explain how each original is encoded, sorted, selected, validated and quoted, to find which layer causes failure and to repeat a good result with the same entry. The count in chapter eight corresponds to the project function for this capability objective: Chapter I defines the source, Chapter II establishes a strong baseline, Chapter III real training, Chapter IV connects the answer, Chapter V binds the account, Chapter VI shows a breakdown, Chapter VII exposes the distribution to failure, and this chapter completes independent delivery and review.

TRY IT YOURSELF

Practice I: Freezing and re-training

Just checking 39 questions and 72 perturbations. Do you have to run encoder training? How do we ensure that author evidence is not covered?

Hint 1
Recovering frozen evaluations requires loading of already authorized weights and does it have to be retrained?
Hint 2
Save 39 questions and 72 perturbations from the new Runs directory and check the pre- and post-checkpoint identity; new training is available in a separate version, without covering author evidence.
Compare with the Solution
No need. Runs rag_lab.run and rag_lab.stress directly using evidence/encoder/encoder.pt and assigns the output to the rans directory. The logs are pre- and post-checkpoint hash, comparing pattern denominators, correct numbers and problem-by-question errors. If retrained, the output catalogues should be used separately and new weights should not be combined to the same conclusion.

TRY IT YOURSELF

Practice 2: Disable tracking

Why should the index be rejected once the document has been reworded? Is the new value proven immediately after reconstruction?

Hint 1
Which of the original bytes before the change is tied to the old index?
Hint 2
What should be done to address the discrepancy between the halsh first, and the distinction between re-restorability and proof of the validity of the new value in reality?
Compare with the Solution
The old index binds the old bytes, and the verification finds inconsistent and does not follow the old quotes. Rebuilding will bring the new index into line with the new original, but only restores traceability and does not prove the facts to be true, valid or authorized.

TRY IT YOURSELF

Practice III: Deployment statement

Students say “through all tests, and therefore support arbitrary photographs and natural issues.” The absence of inferences is noted, and the current acceptance statement is given.

Hint 1
Lists the pixel size, shape syntax and query methods of the actual test field.
Hint 2
Dissociate the fixed field remanding process from undetected natural photographs/open language/OCR/sound, with limited access to the latter not automatically available.
Compare with the Solution
Tests cover fixed 32 pixel geometry, controlled subject field queries and several boundaries; there are no natural photographs, open language, OCRs and voice data. It is possible to establish that, under the same freezing weight of the recorded software version, the original is running to the search, reference and rejection process, fixed questions and disturbance results can be repeated and known failures are fully preserved. Tests do not turn limited calculations into overall proof of capability.

CHECK YOUR UNDERSTANDING

Check Your Understanding

Answer first, then open the explanation to check your reasoning. These questions help reveal gaps in understanding; independent implementation and transfer require separate evaluation.

1. Just freeze reasoning must be recovered?

Show ExplanationThe weights, originals and codes are provided with the items.
2. Test 80 steps to the author's 220 steps?

Show ExplanationThe use of different training tracks should be indicated separately.
3. Full delivery contains audio recognition?

Show ExplanationThe actual master project was PNG and Markdown, not the ASR.
4. Operation capability to complete this lesson?

Show ExplanationRebuilding and checking the flow of information is the ultimate ability of the curriculum.

Manual completion records your progress; it does not certify your project.