LEARNING PLAN/ Progress
Efficient LLM Inference
Use a reproducible CPU inference lab to derive prefill and decoding costs, validate KV caching, budgets, continuous batching, and quantization error, and distinguish correctness, memory savings, and speed with raw measurements.
Start with a question, derive the mechanism, then test your implementation with new inputs.
Times are author estimates that include derivation and practice. They have not been measured with learners; work through the sections at your own pace.
WHAT YOU WILL BUILD
What will you build to demonstrate understanding?
- Indication of prefills, cache decoding and space costs by item and reconciliation of actual loads
- Validate multi-layer cache, global mask, error prefix and absolute position window boundaries
- Distinguishing simulations, approximation and precision by replaying scheduling and quantifying reverses
- Keep correctness, original timing, Hash and incomplete entries on the same freezing model
What should you know before starting?
- Can run the local Python program and understand matrix shapes
- Understanding cause and effect attention, pre-LN, disability, location-by-position feed and complete decoder; link to relevant refresher entry first
- Readable Multiplication, Summation, Maximum Value and Basic Probability; Cost and Error Formula for this course to spread out item by item
- Transformer Chapter 5: Full Decoding BlockIt describes the role of unicoded, leftover and pre-positioned feed in the cause and effect blocks.
- Transformer Chapter 7: Complete DecoderIt tracks input, position and output shape along multiple layers of decoder.
- Transformer Chapter 10: Generating and CacheTo distinguish between complete prefixes, incremental positions and the history of K/V per layer.
Use the prerequisites to choose a starting point. You can read freely and return to earlier lessons whenever you need them.
THE LEARNING ROUTE
A clear path, one step at a time.
Saved in this browser only. Marking a lesson complete does not mean the project has passed verification.
Workload and reuse
Defines the time and cost calibre to validate the cache and to establish a memory book.
Define the Workload Before Deriving Inference Costs
If two implementations both claim more tokens per second, how do we check that they use the same workload and time denominator?
KV Caching: Verify Reuse, Positions and Windows
After retaining previous keys and values, how do we prove that chunked computation equals a full-prefix forward pass?
From KV Elements to Memory Accounting and Admission Budgets
Given layers, batch size and sequence length, how do we calculate cache memory and request capacity before allocation?
Implementation and quality
Distinguishing simulation schedule, approximate quantification and numerical correctness.
Continuous Batching: State, Timelines and Falsifiable Simulations
When a short request arrives before a long one finishes, how does continuous batching change waiting and completion times?
Quantization: From Rounding Errors to Model-quality Limits
After rounding weights to a small set of levels, how do we estimate errors and distinguish fake quantization from real integer execution?
Verify Exact Execution and Approximate Quality Separately
Why do small differences in cached and quantized outputs require different verification criteria?
Measurement and reporting
The fixed trajectory jointly checks the equivalence and time-consuming, preserving the measurement of boundaries.
Measurement: Fixed Traces, Paired Repeats and Performance Limits
Is one faster timing enough? How do we check numerical equivalence and repeated run times on the same inputs?
Final Project: Joint Evidence from One Frozen Build
How can one report combine theoretical costs, simulated scheduling and real CPU measurements without confusing their definitions?
BUILD / TEST / TRANSFER
Auditable CPU Logic Laboratory
Freeze random little decoder to complete cache matching, actual KV bytes, movement control simulations, quantitative errors and fixed tracks to match; deliver source code and real receipts that can be independently downloaded.
Runtime Requirements
Python 3.12 and PyTorch 2.6.0 are recommended. A CPU is sufficient; see the project README for installation instructions.
Read the README after extracting the archive, then follow its environment and run instructions. Reference implementations and learning tasks are provided separately.
Project Checklist
This is a personal checklist, not an automatic grade. Keep your inputs, results, failures and reproduction commands.