KellerJordan/modded-nanogpt

Modded-Nanogpt outlines a speedrun to train a 124M NanoGPT to 3.28 loss on FineWeb in under 90 seconds on 8x H100s, beating the llm.c baseline. It compiles dozens of techniques (Muon optimizer, FP8 matmul, zero-init projections, rotary embeddings, and YaRN/windowing) plus practical run instructions. A track 2 for GPT-2 Medium uses a 2.92 target.

Source

GitHub Trending — Python

Ingested

APR 30, 2026 · 04:08

Editorial score

4.0 / 5