01 Chapter / 8
Import Originals, Stable IDs and Traceable Evidence
When an answer includes a citation, how do stable identity and content versions lead back to the exact evidence in the original file?
Learning Objectives
- Creates an evidence unit from actual documents and pictures
- Define ID, hash and Unicode location
- Clear data, monitor labels and answer evidence boundary
02 Chapter / 8
Lexical Retrieval, BM25 and Hand-checkable Baselines
When a document repeats query terms many times, how does BM25 limit term-frequency advantages and account for document length?
Learning Objectives
- Simulate saturation of word frequency, reverse document frequency and length
- Achieving certainty in Chinese binary units
- A clear word of relevance does not amount to evidence support.
03 Chapter / 8
Dual Encoders, Contrastive Objectives and Parameter Training
How does one image–text pair use a shared contrastive objective to update every parameter in both encoders?
Learning Objectives
- So, two-way cross-alkanes from pairing supervision.
- Calculating = logit gradient
- Train and check all images and text parameters
04 Chapter / 8
Text–image Queries, Candidate Fusion and Ranking Diagnostics
When lexical and image–text retrieval produce different lists, how do we fuse candidates and locate the step that lost the evidence?
Learning Objectives
- Connect the frozen double encoder to the actual PNG index
- Countdown rankings are combined and understood in parallel.
- Distinguishing candidate recall, overlaying and factual support
05 Chapter / 8
Complete Evidence Contexts, Source Verification and Abstention
Even when a citation comes from the original, why must we check visible context, conflicting fields and abstention conditions?
Learning Objectives
- Distinguishing character budget from model token budget
- Complete the review of original hash and location
- Execution of support, conflict and unknown types of response contracts
06 Chapter / 8
Evaluate Retrieval, Answers and Abstention Separately
When retrieval hit rates are high but answers are wrong, how do we score recall, ranking, support and abstention separately?
Learning Objectives
- Clear fixed set of questions, denominators and associated labels
- Calculate the hit, MRR and rejection rates
- Failed to locate with frozen trajectory and avoid evaluation of leakage
07 Chapter / 8
Pixel Perturbations, Information Loss and Failure Diagnosis
Why can a frozen encoder fail on an image–text pair after only shifting the image or changing its exposure?
Learning Objectives
- Distinguishing semantic disturbances from information changes that destroy answers
- Comparison of the same frozen encoder and fixed pixel baselines
- Explaining shifting, exposure and grayness fail without selecting local winners
08 Chapter / 8
Deliver a Reproducible Text–image Research Assistant
When starting the assistant from a clean directory, how do we verify that the index, model, answers and perturbation results use the same build?
Learning Objectives
- Run the full interface between actual files and quoted answers
- Resume evaluation and disturbance with the same checkpoint
- Test all parameters, gradients, sources, leakages and frozen boundaries
Multimodal RAG
Turn source images and documents into a searchable, citable evidence pipeline.
Advanced · 8 lessons · Approx. 40 hours
Scope and Experimental Limits
Eight lessons connect source identity and locations, BM25, bidirectional contrastive learning, frozen image–text retrieval, RRF, complete evidence contexts, citation support, abstention, layered evaluation, and real pixel perturbations. The CPU project updates 35,600 parameters and retains nine neural document-query failures and failures under perturbations such as translation. It uses no remote LLM, pretrained CLIP, OCR, or speech recognition.
Train an image–text dual encoder from source PNG images and documents. Build hybrid retrieval, context selection, item-level citations, conflict-aware abstention, and a complete failure evaluation.
View Learning Plan
Course Details
Course Contents
01 / Import Originals, Stable IDs and Traceable Evidence02 / Lexical Retrieval, BM25 and Hand-checkable Baselines
03 / Dual Encoders, Contrastive Objectives and Parameter Training
04 / Text–image Queries, Candidate Fusion and Ranking Diagnostics
05 / Complete Evidence Contexts, Source Verification and Abstention
All 8 lessons ↗
View Learning Plan