The short answer
Hosted models are a service. You send a request to a provider API and they run the model on their hardware. Open-weight models are weight files you download and run on hardware you control. The line is about who operates the model, not who built it: a provider can serve an open-weight model behind an API.
The main trade is operational load against control. Hosted models cut the work to an API key and a request. Open-weight models give you the weights, the serving stack and the responsibility for both. Data path, version pinning and cost shape usually decide more projects than raw model quality.
The choice is not permanent. Many teams prototype on hosted models, then move a workload to open weights when traffic, privacy review or unit cost justifies it. Keep prompts, tool schemas and test cases portable so the switch is a config change, not a rewrite.
| Aspect | Hosted models | Open-weight models |
|---|---|---|
| Where the model runs | Provider data centers, reached through an API. | Your servers, your cloud account or rented GPU capacity. |
| Who operates it | The provider handles scaling, patching and capacity. | Your team handles serving, scaling, patches and uptime. |
| Data path and retention | Prompts leave your network; contract terms and settings govern retention and training use. | Prompts stay where you run the model; you set retention and access controls. |
| Model versions | The provider deprecates and upgrades versions on its schedule. | You pin a weight file and upgrade when you choose. |
| Customization | System prompts, tools and fine-tuning within provider limits. | LoRA, quantization, serving stack changes and full weight access. |
| Cost shape | Per token, plus fine-tuning storage and integration work. | GPU hours, storage, networking and operations labor. |
| Skills needed | API integration, prompt design and evaluation. | ML infrastructure, GPU operations and model serving. |
| Time to first request | Minutes after account and key setup. | Days to weeks for serving, testing and hardening. |
How hosted models works
You sign up, create an API key, and call an endpoint. The provider owns the GPUs, batching, scaling and patching. You pick a model name and send messages, documents or tool definitions. The response comes back as text, JSON or a tool call. For a basic app this is a few lines of code and a secret in your vault.
Control sits at the edge. You set the system prompt, temperature, max tokens, tool schema and response format. You can use provider features such as batch jobs for bulk work, prompt caching for repeated input, fine-tuning endpoints for style or classification, and private network endpoints for enterprise accounts. You cannot change the weights. You can only steer them with inputs and settings.
Watch the service boundary. Model versions get deprecated. Rate limits shape peak traffic. Data processing terms decide whether prompts are retained, used for training or sent across regions. Check subprocessors, audit logs and zero data retention options. Pin model versions in config and run your own test set before you upgrade.
How open-weight models works
You download weights from a model hub or vendor, check the license, and run them with an inference server. The files are large, often in safetensors format. The license matters: Apache 2.0 and MIT are permissive; Llama and Gemma terms carry usage limits. Read the license before you ship the model in a product.
You need hardware and a serving stack. A GPU needs enough VRAM for the weights plus the KV cache that holds context. vLLM, TensorRT-LLM, llama.cpp and Ollama are common choices. Quantization with GGUF, AWQ or GPTQ shrinks memory at some quality cost. You handle autoscaling, batching, health checks, model loading and upgrades. That is real engineering work.
The payoff is control. You can keep prompts and outputs inside your network, pin a weight file for as long as you keep it, add LoRA adapters, and inspect logits or activations when you need to. You can tune the serving stack for latency or throughput. You also own every outage and every security patch for the runtime. You can rent managed hosting for open weights, but then you are back to a service contract with a model you picked.
When to choose hosted models
Choose hosted models when speed to a working feature matters more than infrastructure control. If your team is two backend developers and no ML engineer, an API integration beats standing up GPU nodes. You get a production endpoint with scaling and monitoring on day one.
Choose hosted models when demand is spiky or small. A support assistant used by a dozen agents, a document classifier that runs in bursts, or a prototype that may be killed next month does not justify reserved GPUs. Pay per token ties cost to use. You avoid idle hardware.
Choose hosted models when you need a strong general model and can accept the provider terms. That fits drafting, summarization, translation and code help where the task is broad. Check data terms first. If the provider cannot meet your retention or region needs, stop there and look at open weights.
When to choose open-weight models
Choose open-weight models when data cannot leave your environment. A model running in your VPC or on-prem cluster keeps prompts, documents and outputs under your controls. You still need network policy, key management and audit logs. The model does not remove those duties.
Choose open-weight models when volume is steady and high enough to keep GPUs busy. At a constant load, a reserved GPU can cost less than metered tokens, but only if you count operations labor. Model the break-even with your real token counts, context lengths and utilization. A GPU that sits idle is a bad deal.
Choose open-weight models when you need deep customization or long-term stability. LoRA fine-tuning, a smaller distilled model, or a pinned weight file for years is easier when you hold the artifact. That matters for product features where a provider deprecation would break customer workflows.
Costs and effort to consider
Hosted cost is usage-based: input tokens, output tokens, cached input, fine-tuning training and storage. The bill scales with traffic, and context length drives it up. Add integration work, a test harness, retries and observability. The hidden cost is switching: prompts tuned to one model may need rework.
Open-weight cost has three parts: GPUs, serving engineering and ongoing operations. GPU capacity can be rented by the hour or reserved. You pay for storage, networking, monitoring and staff time. Add security patching, model upgrades and capacity planning. The hidden cost is idle capacity and the rare late-night page when the inference server stalls.
Run a small trial on your own tasks. Pick 20 to 30 real prompts, define pass criteria, and compare quality, latency and cost per request. For hosted models, test rate limits and model versions. For open-weight models, test the serving stack at your peak context length. Then check the license and data terms. If you want help shaping the decision, Cloudverg's AI readiness assessment is $5,500/month, a fixed scope of two weeks that ranks use cases and builds a token, GPU and run-cost model. Other options include hiring an ML engineer or running an in-house proof of concept.
Frequently asked questions
Are open-weight models free to use?
The weights may be free to download, but the license can still set limits. Running them costs GPU capacity, storage, networking and staff time. A hosted API has no hardware bill but charges per token.
Can I use hosted models with sensitive data?
Sometimes, depending on the provider's terms and the controls you enable. Check retention, training use, region, subprocessors and private endpoint options. If those terms do not fit your policy, run an open-weight model inside your own network.
Do open-weight models need a GPU?
For production use, yes, in nearly all cases. Small models can run on a CPU for tests, but real traffic needs GPU memory for weights and context. You also need a serving stack and someone to operate it.