Every company building with large language models faces the same early choice: call a hosted model through an API, run an open-weight model yourself, or tune a model for your task. Each path is right for some workloads and wrong for others.
Four questions that decide it
- What data will the model see? Public content, internal documents and regulated personal data carry very different obligations.
- How much will you use it? Occasional use favors paying per token. Steady high volume can make your own GPUs cheaper.
- How hard is the task? Complex reasoning still favors the largest hosted models. Narrow, repeated tasks often run well on small models.
- How fast must it answer? Interactive features need low latency close to users. Batch jobs can trade speed for cost.
The options compared
| Option | Strengths | Watch out for |
|---|---|---|
| Hosted API | Best general quality, no infrastructure, fast to start | Per-token cost at scale, provider data terms, rate limits |
| Private endpoint in your cloud | Managed models with traffic kept in your cloud tenancy | Model choice limited to what your cloud offers |
| Open weights on your GPUs | Full data control, predictable cost at volume, no vendor lock-in | You run the serving stack, scaling and upgrades |
| Fine-tuned small model | Low cost and latency for narrow tasks, consistent output format | Needs good training data and re-tuning as needs change |
A practical pattern
Many teams end up with a router: a small, cheap model handles routine requests and a larger model takes the hard ones. Retrieval supplies company knowledge to both, so neither needs to memorize it. An evaluation set decides which requests go where.
Serving open-weight models
If you self-host, the serving layer matters as much as the model. Engines such as vLLM, NVIDIA NIM, Triton Inference Server with TensorRT-LLM, and Hugging Face Text Generation Inference batch requests and manage GPU memory so the same hardware serves far more users. Plan GPU memory for the model weights plus the context length you need.
Start with evaluation, not infrastructure
Before committing to any option, build an evaluation set from real examples and score two or three candidates on it. The results usually make the decision obvious. Our AI team can run this comparison for you.