VRAM 計算器
估算運行 LLM 所需的 VRAM：按量化方式計算的模型權重，加上與您的上下文長度和批次大小對應的 KV 快取。
Covers activations, CUDA context and fragmentation. A rough allowance, not a measurement.
Fits within 24 GB of VRAM
KV cache elements use FP16 (2 bytes) unless FP32 is selected, matching common inference defaults.
KV cache bytes = 2 (K and V) × layers × KV heads × head dim × context × batch × bytes per element. Weights = parameters × bytes per parameter.
使用方法
- Pick a preset or enter the parameter count and architecture (layers, KV heads, head dimension) yourself.
- Set the quantization, context length and batch size.
- See weights, KV cache, the total with overhead, and the smallest VRAM size class it fits.
注意事項
Preset architecture values are approximations from the models' public configs, and every field stays editable. The KV cache is 2 × layers × KV heads × head dim × context × batch × bytes per element, with FP16 elements (2 bytes) unless FP32 is selected. The overhead allowance (15% by default) is a rough margin for activations and fragmentation; real usage depends on the inference framework and settings. This estimates inference only — training (gradients and optimizer states) needs far more.
常見問題
- Why do KV heads matter (GQA)?
- Recent models use grouped-query attention, with fewer KV heads than attention heads, which shrinks the KV cache proportionally. The presets use each model's real KV head count.
- The math fits but the model still will not load — why?
- Frameworks need workspace beyond the raw weights and KV cache: CUDA context, tokenizer and embedding extras, and temporary buffers. Raise the overhead percentage for a safer estimate.