Search for a command to run...
Series
47 foundational papers in LLM engineering, implemented from scratch. Each post covers the mechanism, a reproduction on an RTX 3090, and a decision framework for production use. Covers transformers, RAG, fine-tuning, inference optimisation, deployment, agents, and security.
At roughly 2.08e11 cumulative FLOPs in my own run, a dense baseline lands at 25.31 validation perplexity. At nearly the same compute budget, a sparse MoE lands at 20.98. The absolute delta is 4.324, a
The Memory Problem Serving 10 concurrent users with a 70B-scale model at 4K context, using the vanilla transformer architecture from 2017, requires roughly 240GB of GPU memory: about 140GB for weights
Every production LLM you interact with today, LLaMA 3, Mistral, Gemma, Claude, runs on multi-head attention as its core computation. The paper that introduced it, "Attention Is All You Need" (Vaswani
