Phase 10

LLMs from Scratch

Phase 10: LLMs from Scratch. 24 hands-on lessons building AI from first principles in the browser. Free reading; graded exercises and certificate with lifetime access.

  1. Tokenizers: BPE, WordPiece, SentencePiece (graded)
  2. Building a Tokenizer from Scratch (graded)
  3. Data Pipelines for Pre-Training (graded)
  4. Pre-Training a Mini GPT (124M Parameters) (graded)
  5. Scaling: Distributed Training, FSDP, DeepSpeed
  6. Instruction Tuning (SFT) (graded)
  7. RLHF: Reward Model + PPO (graded)
  8. DPO: Direct Preference Optimization (graded)
  9. Constitutional AI and Self-Improvement (graded)
  10. Evaluation: Benchmarks, Evals, LM Harness (graded)
  11. Quantization: Making Models Fit (graded)
  12. Inference Optimization (graded)
  13. Building a Complete LLM Pipeline
  14. Open Models: Architecture Walkthroughs
  15. Speculative Decoding and EAGLE-3 (graded)
  16. Differential Attention (V2) (graded)
  17. Native Sparse Attention (DeepSeek NSA) (graded)
  18. Multi-Token Prediction (MTP) (graded)
  19. DualPipe Parallelism
  20. DeepSeek-V3 Architecture Walkthrough
  21. Jamba — Hybrid SSM-Transformer
  22. Async and Hogwild! Inference (graded)
  23. Speculative Decoding and EAGLE (graded)
  24. Gradient Checkpointing and Activation Recomputation (graded)
0 lifetime access. Curriculum based on AI Engineering from Scratch by Rohit Ghumare (MIT, used under attribution).