LEARNING PLAN/ Progress
Multimodal RAG
Train an image–text dual encoder from source PNG images and documents. Build hybrid retrieval, context selection, item-level citations, conflict-aware abstention, and a complete failure evaluation.
Start with a question, derive the mechanism, then test your implementation with new inputs.
Times are author estimates that include derivation and practice. They have not been measured with learners; work through the sections at your own pace.
WHAT YOU WILL BUILD
What will you build to demonstrate understanding?
- Build a searchable evidence base from the actual PNG and Markdown
- Hand-held BM25, RRF versus two-way target and gradient
- Real training image/text encoder and checking full parameter updates
- Run freezing rights to hybrid search, complete context, reference or rejection
- Separate evaluation of recall, answer and reject and save the real disturbance failure
What should you know before starting?
- Basic Python, file and array operations
- Vectors, Matrix Multiplications and Basic Probability
- Basic guides; running Python 3.11 and PyTorch projects
Use the prerequisites to choose a starting point. You can read freely and return to earlier lessons whenever you need them.
THE LEARNING ROUTE
A clear path, one step at a time.
Saved in this browser only. Marking a lesson complete does not mean the project has passed verification.
Original and manifesting
Establish back-checkable evidence, typographical retrieval and actual double-coding training.
Import Originals, Stable IDs and Traceable Evidence
When an answer includes a citation, how do stable identity and content versions lead back to the exact evidence in the original file?
Lexical Retrieval, BM25 and Hand-checkable Baselines
When a document repeats query terms many times, how does BM25 limit term-frequency advantages and account for document length?
Dual Encoders, Contrastive Objectives and Parameter Training
How does one image–text pair use a shared contrastive objective to update every parameter in both encoders?
Retrieve and answer
Integration of candidates, checking sources, conflicts, rejections and tiered evaluation.
Text–image Queries, Candidate Fusion and Ranking Diagnostics
When lexical and image–text retrieval produce different lists, how do we fuse candidates and locate the step that lost the evidence?
Complete Evidence Contexts, Source Verification and Abstention
Even when a citation comes from the original, why must we check visible context, conflicting fields and abstention conditions?
Evaluate Retrieval, Answers and Abstention Separately
When retrieval hit rates are high but answers are wrong, how do we score recall, ranking, support and abstention separately?
Pressure and recovery.
Test the real image disturbance and complete delivery at the same check point.
Pixel Perturbations, Information Loss and Failure Diagnosis
Why can a frozen encoder fail on an image–text pair after only shifting the image or changing its exposure?
Deliver a Reproducible Text–image Research Assistant
When starting the assistant from a clean directory, how do we verify that the index, model, answers and perturbation results use the same build?
BUILD / TEST / TRANSFER
A reference assistant for the graphics workshop.
Twenty-four original PNGs and seven original documents, 20 training/4 set aside, 35 600 parameter double encoder; the same freezing rights are linked to 39 questions three ways to retrieve 72 real disturbing images.
Runtime Requirements
Python 3.11+, PyTorch 2.6.0, NumPy 2.3.5, and Pillow 12.3.0. A CPU is sufficient.
Read the README after extracting the archive, then follow its environment and run instructions. Reference implementations and learning tasks are provided separately.
Project Checklist
This is a personal checklist, not an automatic grade. Keep your inputs, results, failures and reproduction commands.