Calculateur de VRAM
Estimez la VRAM nécessaire pour exécuter un LLM : les poids du modèle selon la quantification, plus le cache KV pour votre longueur de contexte et votre taille de lot.
Covers activations, CUDA context and fragmentation. A rough allowance, not a measurement.
Fits within 24 GB of VRAM
KV cache elements use FP16 (2 bytes) unless FP32 is selected, matching common inference defaults.
KV cache bytes = 2 (K and V) × layers × KV heads × head dim × context × batch × bytes per element. Weights = parameters × bytes per parameter.
Comment utiliser
- Pick a preset or enter the parameter count and architecture (layers, KV heads, head dimension) yourself.
- Set the quantization, context length and batch size.
- See weights, KV cache, the total with overhead, and the smallest VRAM size class it fits.
Remarques
Preset architecture values are approximations from the models' public configs, and every field stays editable. The KV cache is 2 × layers × KV heads × head dim × context × batch × bytes per element, with FP16 elements (2 bytes) unless FP32 is selected. The overhead allowance (15% by default) is a rough margin for activations and fragmentation; real usage depends on the inference framework and settings. This estimates inference only — training (gradients and optimizer states) needs far more.
Questions fréquentes
- Why do KV heads matter (GQA)?
- Recent models use grouped-query attention, with fewer KV heads than attention heads, which shrinks the KV cache proportionally. The presets use each model's real KV head count.
- The math fits but the model still will not load — why?
- Frameworks need workspace beyond the raw weights and KV cache: CUDA context, tokenizer and embedding extras, and temporary buffers. Raise the overhead percentage for a safer estimate.