Series

Applied LLM Mechanics

47 foundational papers in LLM engineering, implemented from scratch. Each post covers the mechanism, a reproduction on an RTX 3090, and a decision framework for production use. Covers transformers, RAG, fine-tuning, inference optimisation, deployment, agents, and security.

Attention Is All You Need: What the Paper's Heads Are Actually Doing at Each Layer
Every production LLM you interact with today, LLaMA 3, Mistral, Gemma, Claude, runs on multi-head attention as its core computation. The paper that introduced it, "Attention Is All You Need" (Vaswani
Mar 8, 202612 min read4
LLaMA 2: How Three Borrowed Techniques Fit a 70B Model on Two GPUs
The Memory Problem Serving 10 concurrent users with a 70B-scale model at 4K context, using the vanilla transformer architecture from 2017, requires roughly 240GB of GPU memory: about 140GB for weights
Mar 15, 202615 min read1
Mixtral of Experts: Top-2 Routing Gives 47B Capacity at 13B Active Compute
At roughly 2.08e11 cumulative FLOPs in my own run, a dense baseline lands at 25.31 validation perplexity. At nearly the same compute budget, a sparse MoE lands at 20.98. The absolute delta is 4.324, a
Mar 22, 202612 min read

Command Palette