As a rough rule of thumb at the common Q4_K quantization, an 8B model needs about 6 to 8GB of VRAM, a 32B model needs about 20 to 24GB, and a 70B model needs roughly 40GB or more, so pick your graphics card below to see exactly what it can run.
Your GeForce RTX 4090 (24GB) can run models up to about 33B at Q4_K fully on the GPU; 72B needs offload.
Showing 88 matching models for All models: 75 run fully, 11 need offload, 2 won't fit.
| Model | Type | Best quant | Loaded memory | Fit |
|---|---|---|---|---|
| DeepSeek Coder 33B DeepSeek Coder | coding | Q4_K | 22.5GB | Runs fully on GPU |
| Qwen 2.5 32B Qwen | chat | Q4_K | 22GB | Runs fully on GPU |
| Qwen 2.5-Coder 32B Qwen Coder | coding | Q4_K | 22GB | Runs fully on GPU |
| QwQ 32B Qwen | chat | Q4_K | 22GB | Runs fully on GPU |
| DeepSeek R1 32B DeepSeek | chat | Q4_K | 22GB | Runs fully on GPU |
| DeepSeek R1 Distill Qwen 32B DeepSeek | chat | Q4_K | 22GB | Runs fully on GPU |
| Fireworks AI Mixtral 8x7B Mixtral | chat | Q2_K | 20.4GB | Runs fully on GPU |
| Phi-3.5 MoE Phi | chat | Q2_K | 19GB | Runs fully on GPU |
| Cohere Aya 23 35B Aya | chat | Q3_K | 19.1GB | Runs fully on GPU |
| Yi 1.5 34B Yi | chat | Q3_K | 18.8GB | Runs fully on GPU |
| Code Llama 34B Code Llama | coding | Q3_K | 18.8GB | Runs fully on GPU |
| GLM-Z1 GLM | chat | Q4_K | 22GB | Runs fully on GPU |
| Gemma 2 27B Gemma | chat | Q4_K | 19.5GB | Runs fully on GPU |
| Gemma 3 27B Gemma | chat | Q4_K | 19.5GB | Runs fully on GPU |
| Mistral Small 3 22B Mistral | chat | Q6_K | 22.5GB | Runs fully on GPU |
| WizardLM 2 22B WizardLM | chat | Q6_K | 22.5GB | Runs fully on GPU |
| Qwen 2 15B Qwen | chat | Q8 | 19.4GB | Runs fully on GPU |
| StarCoder 15B StarCoder | coding | Q8 | 19.4GB | Runs fully on GPU |
| Qwen 2.5 14B Qwen | chat | Q8 | 18.4GB | Runs fully on GPU |
| Qwen 2.5-Coder 14B Qwen Coder | coding | Q8 | 18.4GB | Runs fully on GPU |
One row per card: the largest model class that fits fully at Q4_K, and the largest that still runs with CPU offload, assuming 32GB of system RAM and a 4K context length.
| Graphics card | VRAM | Largest fully on GPU (Q4_K) | Largest with CPU offload |
|---|---|---|---|
| GeForce RTX 2060 | 6GB | 4B (Gemma 3 4B) | 47B (Fireworks AI Mixtral 8x7B) |
| GeForce RTX 2070 | 8GB | 8B (Llama 3.1 8B) | 47B (Fireworks AI Mixtral 8x7B) |
| GeForce RTX 2080 | 8GB | 8B (Llama 3.1 8B) | 47B (Fireworks AI Mixtral 8x7B) |
| GeForce RTX 2080 Ti | 11GB | 14B (Qwen 2.5 14B) | 70B (Llama 3.1 70B) |
| GeForce RTX 3060 (12GB) | 12GB | 15B (Qwen 2 15B) | 72B (Qwen 2 72B) |
| GeForce RTX 3060 Ti | 8GB | 8B (Llama 3.1 8B) | 47B (Fireworks AI Mixtral 8x7B) |
| GeForce RTX 3070 | 8GB | 8B (Llama 3.1 8B) | 47B (Fireworks AI Mixtral 8x7B) |
| GeForce RTX 3070 Ti | 8GB | 8B (Llama 3.1 8B) | 47B (Fireworks AI Mixtral 8x7B) |
| GeForce RTX 3080 (10GB) | 10GB | 12B (Gemma 3 12B) | 47B (Fireworks AI Mixtral 8x7B) |
| GeForce RTX 3080 (12GB) | 12GB | 15B (Qwen 2 15B) | 72B (Qwen 2 72B) |
| GeForce RTX 3080 Ti | 12GB | 15B (Qwen 2 15B) | 72B (Qwen 2 72B) |
| GeForce RTX 3090 | 24GB | 35B (Cohere Aya 23 35B) | 90B (Llama 3.2 Vision 90B) |
| GeForce RTX 4060 | 8GB | 8B (Llama 3.1 8B) | 47B (Fireworks AI Mixtral 8x7B) |
| GeForce RTX 4060 Ti (8GB) | 8GB | 8B (Llama 3.1 8B) | 47B (Fireworks AI Mixtral 8x7B) |
| GeForce RTX 4060 Ti (16GB) | 16GB | 22B (Mistral Small 3 22B) | 72B (Qwen 2 72B) |
| GeForce RTX 4070 | 12GB | 15B (Qwen 2 15B) | 72B (Qwen 2 72B) |
| GeForce RTX 4070 Ti SUPER | 16GB | 22B (Mistral Small 3 22B) | 72B (Qwen 2 72B) |
| GeForce RTX 4080 | 16GB | 22B (Mistral Small 3 22B) | 72B (Qwen 2 72B) |
| GeForce RTX 4090 | 24GB | 35B (Cohere Aya 23 35B) | 90B (Llama 3.2 Vision 90B) |
| GeForce RTX 5060 | 8GB | 8B (Llama 3.1 8B) | 47B (Fireworks AI Mixtral 8x7B) |
| GeForce RTX 5060 Ti (8GB) | 8GB | 8B (Llama 3.1 8B) | 47B (Fireworks AI Mixtral 8x7B) |
| GeForce RTX 5060 Ti (16GB) | 16GB | 22B (Mistral Small 3 22B) | 72B (Qwen 2 72B) |
| GeForce RTX 5070 | 12GB | 15B (Qwen 2 15B) | 72B (Qwen 2 72B) |
| GeForce RTX 5070 Ti | 16GB | 22B (Mistral Small 3 22B) | 72B (Qwen 2 72B) |
| GeForce RTX 5080 | 16GB | 22B (Mistral Small 3 22B) | 72B (Qwen 2 72B) |
| GeForce RTX 5090 | 32GB | 47B (Fireworks AI Mixtral 8x7B) | 90B (Llama 3.2 Vision 90B) |
| RTX A6000 | 48GB | 72B (Qwen 2 72B) | 90B (Llama 3.2 Vision 90B) |
| RTX 6000 Ada Generation | 48GB | 72B (Qwen 2 72B) | 90B (Llama 3.2 Vision 90B) |
| Radeon RX 7600 | 8GB | 8B (Llama 3.1 8B) | 47B (Fireworks AI Mixtral 8x7B) |
| Radeon RX 7600 XT | 16GB | 22B (Mistral Small 3 22B) | 72B (Qwen 2 72B) |
| Radeon RX 7700 XT | 12GB | 15B (Qwen 2 15B) | 72B (Qwen 2 72B) |
| Radeon RX 7800 XT | 16GB | 22B (Mistral Small 3 22B) | 72B (Qwen 2 72B) |
| Radeon RX 7900 XT | 20GB | 27B (Gemma 2 27B) | 72B (Qwen 2 72B) |
| Radeon RX 7900 XTX | 24GB | 35B (Cohere Aya 23 35B) | 90B (Llama 3.2 Vision 90B) |
| Radeon RX 9070 | 16GB | 22B (Mistral Small 3 22B) | 72B (Qwen 2 72B) |
| Radeon RX 9070 XT | 16GB | 22B (Mistral Small 3 22B) | 72B (Qwen 2 72B) |
| Arc A770 (8GB) | 8GB | 8B (Llama 3.1 8B) | 47B (Fireworks AI Mixtral 8x7B) |
| Arc A770 (16GB) | 16GB | 22B (Mistral Small 3 22B) | 72B (Qwen 2 72B) |
| Arc B580 | 12GB | 15B (Qwen 2 15B) | 72B (Qwen 2 72B) |
The memory rule: a model's weights at the chosen quant, plus the context-length allowance, plus a small VRAM headroom for driver and runtime overhead, must fit inside the available memory. "Fully on GPU" requires that total to fit in VRAM alone with headroom to spare; "CPU offload" means it only fits once usable system RAM is added in, which is much slower because data has to move between RAM and VRAM for every token.
Context allowance is a deliberately simple, conservative estimate, not a per-model architecture calculation. 4K context is treated as already covered by the base memory estimate; 16K and 32K add a flat extra allowance that scales with model size, because larger models generally have larger key-value caches per token. Models with fewer key-value heads (most modern models use grouped-query attention) typically need less than this estimate; older dense-attention models may need more.
Multiple identical GPUs pool their VRAM for a large model split across cards (as llama.cpp and vLLM both support), minus a small flat overhead per extra card for duplicated runtime buffers. This assumes identical cards; mixing different GPUs is not modeled here.
Real speed and stability also depend on your backend and drivers, whether a model needs a vision adapter for multimodal input, and what else is using the GPU at the time. Treat every figure here as a planning estimate, not a guarantee.
GPU VRAM figures were checked against each manufacturer's own product or specification page (nvidia.com, amd.com, intel.com) on 2026-09-25. For the RX 7900 XT and RX 7900 XTX, amd.com's own product page did not respond at check time, so the manufacturer-stated figure was cross-checked against multiple add-in-board partner specification pages (ASUS, MSI, Sapphire, XFX) that mirror it. A card is left out entirely if its VRAM could not be verified this way.
Backend support is noted in plain terms, not benchmarked. NVIDIA cards use CUDA, which almost every local LLM tool supports well. AMD cards use ROCm (official support varies by specific card, operating system and driver version) or the slower but more broadly compatible Vulkan backend that most llama.cpp-based apps also offer. Intel Arc cards use Intel oneAPI-based tooling (such as IPEX-LLM) or Vulkan; support is newer and less universal than CUDA. Check that your chosen app supports your specific card before buying hardware for local LLM use.
Model memory estimates reuse the same practical loaded-memory figures as the Mac model fit calculator: GGUF / llama.cpp-class quantized local inference estimates, not a vendor benchmark or a guarantee of a specific tokens-per-second number.
GPU specifications checked against manufacturer pages on 2026-09-25.
At the common Q4_K quantization, a 70B model needs roughly 40GB of loaded memory, plus more for longer context. That does not fit on any single consumer GPU today, so you need a workstation card with 48GB or more, two pooled consumer GPUs, or CPU offload into system RAM, which is much slower.
Yes, for smaller models. An 8GB card comfortably runs models around 7B to 8B at Q4_K with headroom to spare, and can often stretch a bit further at a lower quant. Anything much larger will need CPU offload or a bigger card.
Yes, through CPU offload: the layers that do not fit in VRAM run on the CPU instead, using system RAM. It works, but it is much slower than running fully on the GPU, since data has to move between RAM and VRAM for every token. Treat it as a way to run a bigger model occasionally, not as a daily driver.
One GPU with enough VRAM to hold the whole model is simpler and usually faster. Two identical GPUs can pool their VRAM for a bigger model through tensor or layer splitting in tools like llama.cpp and vLLM, but most consumer cards have no fast direct link between them, so the interconnect can become the bottleneck on some workloads.
They can run them, but support is less mature than NVIDIA's CUDA. AMD cards use ROCm, where official support varies by card and driver version, or the more broadly compatible but slower Vulkan backend. Intel Arc cards use oneAPI based tooling or Vulkan. Check that your chosen app supports your specific card before buying hardware for this purpose.
Tools like the GPU calculator start in the community, where I post the prompts and workflows I actually use with clients.
Join the communityTwenty free minutes. I will tell you where AI fits in your business, or that it does not yet.
Book a free 20-minute call