X
LLM Optimization Interview Notes: Training and Inference
@gauri__guptaarticlex
TL;DR. A field guide to the memory, compute, and inference tricks that actually keep billion-parameter LLMs trainable and deployable.
Takeaways
- Memory wins come from Flash Attention (tiling + recomputing softmax norm factors), MQA/GQA sharing KV across heads, and activation checkpointing trading compute for RAM.
- Inference speedups stack: KV caching skips recomputing attention, stateful caching reuses conversation prefixes via rolling hashes, and speculative decoding lets a small draft model propose tokens for a big verifier.
- Throughput tricks worth knowing: sequence packing kills padding waste, LongNet/BigBird get linear-complexity long context, and mixed precision (bfloat16) + quantization (PTQ/QAT) squeeze the rest.