Decoding in Tiles: A Static Megakernel for Llama 3.2 1B on an RTX 3050
An approach to low-latency LLM inference by scheduling a graph of tiles in a megakernel. How close can it get to the memory-bandwidth floor of batch-one decode on a small GPU? This post explains where that floor comes from for Llama 3.2 1B on an RTX 3050, how a token is scheduled into tiles and waves, where the remaining time goes, and how the result compares with PyTorch and llama.cpp.
- CUDA
- GPU programming
- megakernels