Back to Module 1.9: Self-AttentionIn Progress
AI Lesson & Submodule
Why Self-Attention Was Needed
Compare recurrent constraints against parallel attention calculations.
Why This Matters
RNNs process tokens one-by-one, which blocks parallel GPU training and degrades long-context memory.
What You Will Learn
- •Explain sequential bottlenecks
- •Describe recurrence limits
- •Introduce attention idea
Concepts Covered
Sequential bottlenecksVanishing recurrent gradientsParallel attention scaling
Mapped Foundation Project: Mini Attention Notebook
Self-contained interactive workbook computing step-by-step scaled dot-product attention scores from raw matrices input.
Architecture Preview
Dynamic math workbook running forward projections, scaling matrix outputs, and plotting soft heatmaps.
Character Tokens ArrayQKV Projections WeightsQK Dot Product Score
Tech Stack Planned
ReactTypeScriptMath.js
Technical Interview Value
- ?Explain the fundamental training limitations of RNNs/LSTMs that led to the attention mechanism