GPU hosting guide
GPU VPS for LLMs: How Much VRAM Do You Need?
Model size is only half the story. VRAM, quantization, and context length decide whether a rented GPU actually runs your model.
Affiliate disclosure: some links below are affiliate links. We may earn a commission at no extra cost to you. Verify current pricing and availability with each provider.
When you actually need a GPU
You do not need a GPU to build an AI product. If your app calls an external model API, orchestrates tools, or processes text, a normal CPU VPS is enough. A GPU becomes necessary when you run model inference yourself: local chat models, embeddings at scale, image generation, or speech models.
The trade-off is simple. A GPU gives you much faster tokens per second, but it costs far more per hour and sits idle most of the time for small projects. Many teams start with an API and move to self-hosting only when volume or privacy justifies it.
VRAM by model size
A rough rule of thumb: FP16 needs about 2 bytes per parameter, INT8 about 1 byte, and 4-bit about 0.5 byte. Add 10–30% for overhead and KV cache, more for long context or batching.
| Model size | FP16 | INT8 | 4-bit | Typical use |
|---|---|---|---|---|
| 7B | ~14 GB | ~7 GB | ~4–6 GB | Chat, summarization, small agents |
| 13B | ~26 GB | ~13 GB | ~7–9 GB | Better reasoning, coding help |
| 34B | ~68 GB | ~34 GB | ~18–22 GB | Higher quality, still single-node |
| 70B | ~140 GB | ~70 GB | ~35–40 GB | Strong general models, multi-GPU |
These are estimates, not guarantees. The tokenizer, architecture, and context window all affect real memory use. A 7B model at 4-bit can fit in 6 GB for short prompts, but a long conversation with a big KV cache may need more.
Quantization in plain terms
Quantization stores model weights with less precision. FP16 is the default for quality. INT8 roughly halves memory with a small quality loss. 4-bit formats such as GGUF, AWQ, and GPTQ cut memory to about a quarter, and are the usual choice for consumer GPUs and CPU inference.
For most small projects, a 4-bit 7B or 13B model is the sweet spot: good enough quality, fits on affordable hardware, and runs with tools like Ollama or llama.cpp. Drop to lower precision only if you have measured the quality trade-off.
Context length and KV cache
Weights are not the only thing in VRAM. The KV cache grows with context length and batch size. Doubling the context can add gigabytes on larger models. If you serve multiple users at once, budget for concurrency, not just a single request.
Practical approach: pick a model that fits comfortably with room to spare, cap the context window, and limit concurrent requests. Running out of VRAM mid-request usually means a crash or a fallback to CPU, both of which hurt user experience.
GPU cloud options
Prices change constantly, so treat the list below as a starting point and check current hourly rates. For occasional use, stop instances when idle. For steady workloads, reserved or committed plans are usually cheaper.
RunPod
Community and secure cloud GPUs, billed by the hour or second. Good for experiments and short jobs.
Vast.ai
Marketplace of rented GPUs with competitive prices. Quality and reliability vary by host, so check reviews.
Lambda
GPU cloud aimed at AI workloads, with reserved and on-demand options. Good when you need predictable performance.
Vultr GPU
GPU instances alongside regular VPS in many regions. Convenient if you already use Vultr for other services.
Running an LLM on a rented GPU
- Pick an instance whose VRAM comfortably exceeds the 4-bit estimate for your model.
- Start from an image with NVIDIA drivers and CUDA, or install them if needed.
- Install Ollama or llama.cpp and download a quantized model.
- Expose the local API through a reverse proxy with authentication. Never expose it raw to the internet.
- Stop or destroy the instance when you are done, or set a budget alert.
Ollama is the fastest path for a single model. llama.cpp gives more control over quantization and CPU offload. For production, containerize the service and add health checks and logging.
When a CPU VPS is enough
If you only need occasional answers, a CPU server with 8–16 GB RAM can run a 4-bit 7B model with llama.cpp. Expect a few tokens per second, which is fine for batch jobs and internal tools but slow for interactive chat.
CPU inference is also useful as a fallback when GPU capacity is unavailable. It keeps a small product running without committing to GPU costs.
Cost control checklist
- Stop GPU instances when idle; do not leave them running overnight by habit.
- Prefer smaller quantized models unless quality clearly requires more.
- Cap context length and concurrency.
- Use serverless or on-demand GPUs for spiky workloads.
- Compare self-hosting against API pricing at your actual volume.
FAQ
Can I run a 70B model on a single GPU?
Not usually. A 70B model needs roughly 140 GB in FP16 or 35–40 GB at 4-bit, before KV cache. That means a high-VRAM GPU or multiple GPUs. For most small teams, an API is cheaper than renting that much VRAM.
Is a gaming GPU enough for LLM inference?
VRAM is the limiting factor, not the model name. A gaming GPU with 16–24 GB VRAM can run 7B and some 13B models at 4-bit. Datacenter GPUs are better for long-running, multi-user workloads.
What is the cheapest way to run a 7B model?
Run a 4-bit quantized model with llama.cpp or Ollama on a CPU server with 8–16 GB RAM, or rent a small GPU by the hour. For occasional use, a hosted API is usually cheaper than keeping a GPU running.
Do I need to install CUDA myself?
Most GPU cloud images include NVIDIA drivers and CUDA. Tools such as Ollama handle the runtime. Check the provider image list before you start, and prefer images labeled for PyTorch or CUDA.
Last updated: November 2026. VRAM figures are estimates. Test your model and context length on the target hardware before committing to a long rental.