Mistral-Small-3.2-24B-Instruct-2506 requirements — can you run it?
~23.6B parameters · 26 quantizations measured · updated 2025-08-26 · source: unsloth/Mistral-Small-3.2-24B-Instruct-2506-GGUF
Mistral-Small-3.2-24B-Instruct-2506 is a dense transformer (~23.6B parameters) (57k downloads on Hugging Face), published as GGUF by the community. At the recommended UD-IQ2_XXS quant it needs 9.6 GB of VRAM or system RAM at 8K context. Builds measured on this page range from 8.2 GB (smallest quant) to 58.1 GB (lossless). Every number below comes from the actual GGUF files - not a params x bits formula.
# llama.cpp (auto-downloads the GGUF) llama-cli -hf unsloth/Mistral-Small-3.2-24B-Instruct-2506-GGUF:UD-IQ2_XXS # Ollama (pulls straight from Hugging Face) ollama run hf.co/unsloth/Mistral-Small-3.2-24B-Instruct-2506-GGUF:UD-IQ2_XXS # LM Studio: search "unsloth/Mistral-Small-3.2-24B-Instruct-2506-GGUF" in the model browser
Commands load the exact quant measured on this page.
Every quantization: real file size vs what you actually need
| Quant | File (measured) | VRAM/RAM needed (8K ctx) | Runs on (first fit) |
|---|---|---|---|
| UD-IQ1_S | 5.6 GB | 8.2 GB | RTX 3060 12GB |
| UD-IQ1_M | 6.0 GB | 8.7 GB | RTX 3060 12GB |
| UD-IQ2_XXS | 6.8 GB | 9.6 GB | RTX 3060 12GB |
| UD-IQ2_M | 8.2 GB | 11.4 GB | RTX 4060 Ti 16GB |
| Q2_K | 8.9 GB | 12.2 GB | RTX 4060 Ti 16GB |
| Q2_K_L | 9.0 GB | 12.4 GB | RTX 4060 Ti 16GB |
| UD-Q2_K_XL | 9.3 GB | 12.7 GB | RTX 4060 Ti 16GB |
| UD-IQ3_XXS | 9.4 GB | 12.8 GB | RTX 4060 Ti 16GB |
| Q3_K_S | 10.4 GB | 14.0 GB | RTX 4060 Ti 16GB |
| Q3_K_M | 11.5 GB | 15.3 GB | RTX 3090 24GB |
| UD-Q3_K_XL | 11.9 GB | 15.7 GB | RTX 3090 24GB |
| IQ4_XS | 12.8 GB | 16.8 GB | RTX 3090 24GB |
| IQ4_NL | 13.5 GB | 17.7 GB | RTX 3090 24GB |
| Q4_0 | 13.5 GB | 17.7 GB | RTX 3090 24GB |
| Q4_K_S | 13.5 GB | 17.8 GB | RTX 3090 24GB |
| Q4_K_M | 14.3 GB | 18.7 GB | RTX 3090 24GB |
| UD-Q4_K_XL | 14.5 GB | 19.0 GB | RTX 3090 24GB |
| Q4_1 | 14.9 GB | 19.3 GB | RTX 3090 24GB |
| Q5_K_S | 16.3 GB | 21.1 GB | RTX 3090 24GB |
| Q5_K_M | 16.8 GB | 21.6 GB | RTX 3090 24GB |
| UD-Q5_K_XL | 16.8 GB | 21.6 GB | RTX 3090 24GB |
| Q6_K | 19.3 GB | 24.7 GB | RTX 5090 32GB |
| UD-Q6_K_XL | 20.8 GB | 26.4 GB | RTX 5090 32GB |
| Q8_0 | 25.1 GB | 31.6 GB | RTX PRO 6000 96GB |
| UD-Q8_K_XL | 29.0 GB | 36.3 GB | RTX PRO 6000 96GB |
| BF16 | 47.2 GB | 58.1 GB | RTX PRO 6000 96GB |
KV cache uses a flat +20% context/overhead rule because this repo does not publish its config; exact numbers may differ for long contexts. MoE models keep all experts in memory — total size counts, not just active params. Longer context grows the KV cache — use the selector above.
Can I run Mistral-Small-3.2-24B-Instruct-2506 on my GPU? (UD-IQ2_XXS recommended quant)
| Hardware | Memory | Verdict | Est. speed |
|---|---|---|---|
| RTX 3060 12GB | 12 GB | RUNS | ~40 tok/s |
| RTX 4060 Ti 16GB | 16 GB | RUNS | ~32 tok/s |
| RTX 3090 24GB | 24 GB | RUNS | ~104 tok/s |
| RTX 4090 24GB | 24 GB | RUNS | ~112 tok/s |
| RTX 5090 32GB | 32 GB | RUNS | ~199 tok/s |
| RTX PRO 6000 96GB | 96 GB | RUNS | ~199 tok/s |
| Mac 16GB unified | 16 GB | RUNS | ~11 tok/s |
| Mac 32GB unified | 32 GB | RUNS | ~17 tok/s |
| Mac 64GB unified | 64 GB | RUNS | ~30 tok/s |
| Mac 128GB unified | 128 GB | RUNS | ~44 tok/s |
| Mac 256GB unified | 256 GB | RUNS | ~61 tok/s |
| Mac 512GB unified | 512 GB | RUNS | ~91 tok/s |
| 32GB system RAM (CPU) | 32 GB | RUNS | ~6 tok/s |
| 64GB system RAM (CPU) | 64 GB | RUNS | ~6 tok/s |
| 128GB system RAM (CPU) | 128 GB | RUNS | ~9 tok/s |
| 256GB system RAM (CPU) | 256 GB | RUNS | ~9 tok/s |
| 1TB server RAM (CPU) | 1024 GB | RUNS | ~22 tok/s |
| 2TB server RAM (CPU) | 2048 GB | RUNS | ~22 tok/s |
Speed = decode tok/s estimated from memory bandwidth ÷ measured file size (× MoE active share). Real numbers vary ±30% by runtime and settings.
How much VRAM does Mistral-Small-3.2-24B-Instruct-2506 need?
The honest answer: 9.6 GB at the UD-IQ2_XXS quant with 8K context — that is the measured
file size (6.8 GB) plus KV cache and runtime overhead. The smallest published build needs
8.2 GB; the lossless one needs 58.1 GB.
Cheapest hardware that runs it: Used RTX 3060 12GB (~$250, used market) — see which GPU to buy.
Can I run it with CPU offload?
Yes, if combined RAM+VRAM ≥ file size — but generation speed is limited by memory bandwidth. Expect roughly: DDR5 dual channel ~10-30 tok/s for small MoE actives, single digits for big ones, 0.1-1 tok/s when experts page from disk. CPU-only is fine for batch jobs, painful for chat.
Why do other sites show different numbers?
Most pages compute weights from a formula (params × bits/8). We use the actual file sizes published in GGUF repos, which include the embedding table, unquantized tensors, and container overhead — that is why our numbers can differ from a naive calculation.
FAQ
How much VRAM does Mistral-Small-3.2-24B-Instruct-2506 need?
The measured answer: 9.6 GB at the UD-IQ2_XXS quant with 8K context. The smallest published build needs 8.2 GB; the lossless (F16/BF16) build needs 58.1 GB. These are real GGUF file sizes from Hugging Face, not formula estimates.
Can I run Mistral-Small-3.2-24B-Instruct-2506 on a 24GB GPU (RTX 3090/4090)?
Yes — at UD-IQ2_XXS (9.6 GB). Quants that fit 24GB: UD-IQ1_S, UD-IQ1_M, UD-IQ2_XXS, UD-IQ2_M, Q2_K, Q2_K_L, UD-Q2_K_XL, UD-IQ3_XXS, Q3_K_S, Q3_K_M, UD-Q3_K_XL, IQ4_XS, IQ4_NL, Q4_0, Q4_K_S, Q4_K_M, UD-Q4_K_XL, Q4_1, Q5_K_S, Q5_K_M, UD-Q5_K_XL.
How fast will Mistral-Small-3.2-24B-Instruct-2506 run?
Decode speed is memory-bandwidth-bound. At UD-IQ2_XXS: roughly 112 tok/s on an RTX 4090, 104 tok/s on an RTX 3090, 44 tok/s on an M-series Mac with 128GB, and 5.6 tok/s on CPU with dual-channel DDR4 — estimates from measured file size, MoE active-parameter share and memory bandwidth.
Can I run Mistral-Small-3.2-24B-Instruct-2506 on CPU with system RAM?
Yes, if your usable RAM is at least the file size. At UD-IQ2_XXS you need ~9.6 GB of RAM. CPU generation is memory-bandwidth-bound: expect single-digit tok/s for large MoE models, 10-30 tok/s for small active params, and below 1 tok/s when experts page from disk.
Why do VRAM numbers for Mistral-Small-3.2-24B-Instruct-2506 differ between sites?
Most sites compute weights from a formula (params x bits/8). ModelFit uses the actual GGUF file sizes published on Hugging Face, which include the embedding table, unquantized tensors, and container overhead — so our numbers reflect what you really download and load.