In short
A local LLM is an open-weight language model that runs on hardware you control: your own server room or a rented GPU in a data center you chose. Prompts never reach an outside provider, there is no per-token bill, and the model does not change unless you change it.
The trade is hardware and people. Self-hosting rarely saves money for a single assistant. It pays off in three situations:
- Data may not leave your perimeter. Banks, government agencies, healthcare and industrial companies whose security teams forbid any external cloud, including local ones.
- Volume is very high. Millions of documents a day, round-the-clock batch jobs, hundreds of employees using an assistant at once.
- You need full control. A frozen model version, your own fine-tuning, operation without internet access.
We see the first case most often with banks and public-sector clients in Kazakhstan and Azerbaijan. This article covers which model to pick, what server it needs and how to estimate the cost. Model facts and prices were checked against primary sources on 29 October 2026.
When a local model is required and when a cloud API is enough
Start with the rules that actually apply to you. In Kazakhstan, Article 12(2) of the Law on Personal Data and Their Protection requires personal data to be stored in databases located in the country. That rules out foreign clouds for customer data, but not necessarily a cloud inside Kazakhstan. Azerbaijan's personal data law has no direct localization clause. It restricts transfers abroad, and the Central Bank has advised banks against keeping customer data in foreign clouds.
In practice the stricter rule usually comes from the security team, not the statute. A useful test:
- The security team allows an in-country cloud. Use a hosted API there and pay per token. Revisit self-hosting only if the bill grows past what a GPU costs.
- It allows only infrastructure you control. Rent a dedicated GPU server in a local data center under a contract that gives you the machine and the disks, or buy one.
- It allows nothing outside your building. Buy the server and plan for air-gapped operation from day one: offline model downloads, offline docs, no telemetry.
Which open-weight models fit a company
| Model | Parameters (total / active) | Context | License | Notes |
|---|---|---|---|---|
| gpt-oss-120b | 117B / 5.1B | 131,072 | Apache 2.0 | OpenAI says it runs on a single 80 GB GPU; native function calling |
| gpt-oss-20b | 21B / 3.6B | 131,072 | Apache 2.0 | fits in 16 GB of memory, good for a pilot |
| Qwen3.6-35B-A3B | 35B / 3B | 262,144 | Apache 2.0 | fast mixture-of-experts model, tool calling in vLLM |
| Qwen3.8-27B | 27B dense | 262,144 | Apache 2.0 | newest dense Qwen, August 2026 |
| T-pro-it-2.1 (T-Bank) | 32B, built on Qwen3-32B | 40,960 | Apache 2.0 | Russian-tuned; the authors report better tool calling than Qwen3-32B |
| GigaChat3.1-10B-A1.8B (Sber) | 10B / 1.8B | 262,144 | MIT | small and fast, for classification and short answers |
| GigaChat3.5-432B-A28B (Sber) | 432B / 28B | 262,144 | MIT | strong, needs an 8-GPU server |
| DeepSeek V4.1 Flash | 552B | up to 1M | MIT | a cluster model, not a single-server one |
Three things matter more than the leaderboard position:
- License. Apache 2.0 and MIT allow commercial use with no revenue conditions. Not every open model does. Qwen3.8-Flash-Next ships under its own community license with conditions for large services, and Yandex's open YandexGPT 5 Lite asks you to contact Yandex past 10 million output tokens a month. Read the LICENSE file before you build on a model.
- Mixture-of-experts versus dense. MoE models such as Qwen3.6-35B-A3B and gpt-oss only use a fraction of their parameters per token, so they are fast. All parameters still have to sit in GPU memory.
- Your languages. If your users write in Russian, Kazakh or Azerbaijani, test those languages specifically. Models trained with a focus on Russian, such as T-Pro and GigaChat, belong in the comparison for Russian-language work.
How much GPU memory you need
For the weights, multiply the parameter count by the bytes per parameter: 2 bytes in BF16, 1 byte in FP8, about half a byte in 4-bit formats. Then add room for the KV cache, which holds the context of every request in flight. We budget 20 to 50 percent on top of the weights, depending on context length and how many people use the system at once.
| Model | Weights | Hardware that fits |
|---|---|---|
| gpt-oss-20b | about 16 GB | one 24–48 GB card |
| GigaChat3.1-10B-A1.8B, BF16 | about 20 GB | one 48 GB card |
| Qwen3.6-35B-A3B, FP8 | about 35 GB | one 48–96 GB card |
| T-pro-it-2.1, BF16 | about 64 GB | a tight fit on one 80 GB H100, comfortable on 96 GB |
| gpt-oss-120b, MXFP4 | fits in 80 GB | one H100 or H200 |
| GigaChat3.5-432B-A28B | hundreds of GB | an 8-GPU server |
For reference: NVIDIA H100 has 80 or 94 GB, H200 has 141 GB, L40S and RTX 6000 Ada have 48 GB, RTX PRO 6000 Blackwell has 96 GB.
That gives three typical setups:
- Pilot. One 48–96 GB card with a 20–35B model. Enough for a document assistant serving a few dozen employees and for proving quality.
- Production server. One or two 80–141 GB cards running gpt-oss-120b or Qwen3.6-35B-A3B with long context. Handles a customer-facing agent and an internal assistant under real load.
- Large models. An 8-GPU server for GigaChat 3.5 or DeepSeek-class models. Only worth it when mid-size models fail your tests and the data still cannot leave.
What a local LLM costs
Hardware prices vary a lot by country and supply channel, so we give the method and two checked reference points rather than one number.
Rental. A GPU running around the clock is about 730 hours a month. Lambda, a US GPU cloud, lists H100 at $3.29–4.29 per GPU-hour, which is roughly $2,400–3,100 a month for one card. Local data centers in regulated markets usually charge more than US hyperscale clouds, so ask for a quote.
API versus GPU. On the same day we checked both price lists in Russia, where in-country hosted models and in-country GPUs are both on sale. One rented H100 cost about 623,000 rubles a month. A support agent handling 2,000 conversations a month on a strong hosted model, Yandex's Alice AI LLM, would use about 28,000 rubles of tokens. That is a factor of more than twenty. The ratio will differ in your market, but the order of magnitude rarely does: one assistant almost never justifies a dedicated GPU on cost alone.
Purchase. Get vendor quotes and add what the sticker price leaves out:
- People. Someone updates drivers, models and the inference server, and watches load and failures. That is part of a DevOps engineer's time or a support contract.
- Power and cooling if the server sits in your building. An 8-GPU server draws several kilowatts.
- Model upgrades. Open models get a new generation every few months. Each switch means re-running your tests and sometimes a new quantization.
Software: what runs the model
The production default is vLLM. It exposes an OpenAI-compatible API, supports tool calling (the --enable-auto-tool-choice flag plus a --tool-call-parser for the model family), splits a model across several GPUs and can serve its docs offline for air-gapped sites. T-Bank recommends SGLang for T-Pro, which offers similar features.
For a single-machine pilot, Ollama and llama.cpp are easier to install and also speak the OpenAI API. They are not built for dozens of concurrent users.
The OpenAI-compatible interface matters because it keeps your application independent of where the model runs. You can prototype against a hosted model in your country and later point the same code at your own server.
For retrieval, run an embedding model and a reranker next to the LLM: BAAI bge-m3 (MIT, over 100 languages, up to 8,192 tokens) or Qwen3-Embedding with Qwen3-Reranker (Apache 2.0). They are small and share the same GPU.
What a local model can realistically handle
A document assistant (RAG) is the best fit. Retrieval decides quality: if search finds the right paragraph, a 20–35B model summarizes it about as well as a frontier model. Our article on how RAG works beyond vector embeddings covers the search side, and the AI document assistant page shows how we build these systems.
An agent with tools is harder. It has to pick the right function, fill its parameters, read the CRM response and decide what to do next. Mid-size open models make these mistakes more often than the largest hosted ones. Keep the tool set small, make parameter schemas strict and validate each step in code. The AI agents page describes how we design them.
Classification and field extraction (routing tickets, sorting email, reading invoice fields) works well even on small models such as GigaChat3.1-10B or gpt-oss-20b.
Pick the model with a test, not a leaderboard: 30 to 50 real questions with correct answers, two or three candidates, and a comparison of accuracy, tool errors and latency. The method is in why AI projects need evals. If the local model loses to a hosted one on your data, that is useful too: you learned the price of the closed perimeter before buying hardware. For an internal assistant, the access and source rules in internal ChatGPT for a company apply unchanged on a local model.
How to run a pilot
- Pick one workflow and write 30 to 50 test questions with expected answers.
- Rent one 48–96 GB card in a local data center for a month instead of buying.
- Run vLLM with two or three candidate models and score them on your test set.
- Put the winner in front of part of the real users and measure load: concurrent requests, context length, memory used by the KV cache.
- Size the purchase from those numbers, not from a vendor's reference architecture.
That way you neither buy an H200 for a job a card at a third of the price could do, nor buy hardware for a model that failed your tests.
FAQ
On one 80 GB GPU, gpt-oss-120b and Qwen3.6-35B-A3B are strong candidates. For Russian-language work, add T-pro-it-2.1 to the comparison. For a pilot on modest hardware, try gpt-oss-20b or GigaChat3.1-10B-A1.8B. Make the final choice with a test set built from your own questions.
Multiply parameters by bytes per parameter (2 for BF16, 1 for FP8, about 0.5 for 4-bit) and add 20 to 50 percent for the KV cache. A 35B model in FP8 needs about 35 GB plus cache, and gpt-oss-120b fits on one 80 GB GPU.
For one assistant, almost never. When we compared same-day prices in Russia, one rented H100 cost more than twenty times the monthly token bill of a 2,000-conversation support agent on a strong hosted model. Self-hosting wins when data cannot leave your perimeter or when the server is busy around the clock with several systems.
Yes, if the model supports tool calling (gpt-oss, Qwen3.6, T-Pro and GigaChat 3.5 do) and the agent is designed for it: few tools, strict schemas, steps checked in code. Mid-size open models choose the wrong tool more often than the largest hosted models, so test on your own scenarios.
The personal data law requires personal data to be stored in databases located in Kazakhstan. It does not name AI models, but any system that stores customer data, including an assistant's logs and search index, falls under the rule. Many banks and agencies go further and allow only infrastructure they control.
Sources
- OpenAI, gpt-oss-120b model card on Hugging Face, 2025: 117B parameters, 5.1B active, runs on a single 80 GB GPU, Apache 2.0.
- Qwen, Qwen3.6-35B-A3B model card on Hugging Face, 2026: 35B parameters, 3B active, 262,144-token context, tool calling in vLLM.
- vLLM, Tool Calling documentation, 2026: tool calling in the OpenAI-compatible server.
- NVIDIA, H100 Tensor Core GPU, 2026: H100 memory size.
- Republic of Kazakhstan, Law on Personal Data and Their Protection No. 94-V, official Russian text as amended in 2026: storage of personal data in databases located in Kazakhstan.