How well do GPT, Claude and Gemini handle Uzbek, Kyrgyz, Azerbaijani, Armenian and Georgian

Short version

If you are launching a support bot, a sales assistant or an internal search tool for customers in Tashkent, Bishkek, Baku, Yerevan or Tbilisi, the first question is whether the model can actually work in the local language. Public evidence exists for all five languages, but it is uneven: some leaderboards were updated this month, others still test 2024 models.

This is a reading of published benchmarks as of 29 October 2026. We are not publishing our own scores for these languages. Every number below comes from the cited work, with its date and the model versions it tested.

• Uzbek. UzLiB, updated 14 October 2026, puts Gemini on top (Gemini 4 Argon at 0.84), with GPT-6 Luna at 0.81 and Claude Sonnet 5.5 at 0.78. These are current models.
• Kyrgyz. In KyrgyzLLM-Bench (July 2026), Claude 4.5 Sonnet led the closed models and GPT-5.1 came second. Gemini 2.5 Flash refused benign Kyrgyz prompts, and the authors ask readers to discount its score.
• Azerbaijani. We found no recent three-way comparison. The latest is TUMLU, on 2024 models: Claude 3.5 Sonnet 84.4%, GPT-4o 82.4%, Gemini 1.5 Pro 78.6%.
• Armenian. ArmBench-LLM 1.0 (April 2026) ranks Gemini 3 Flash first and GPT-5.2 Pro second. Claude 4.5 and 4.6 were excluded over unstable outputs.
• Georgian. In GeoLogicQA (2025), Gemini 2.5 Flash solved 83% of reasoning tasks and GPT-4o 64%. Claude was not tested.

Use these results to pick two or three candidates. Then choose the model on your own conversations, because none of these benchmarks contains your products, your customers' spelling or your escalation rules.

What these benchmarks test

Most of the work below is multiple choice. The model reads a question and options and returns a letter. That makes it cheap to compare dozens of models, and it measures language and subject knowledge. It does not measure how a model behaves in a customer conversation.

Missing from every benchmark here:

• customers who type Uzbek in Latin script without apostrophes, then switch to Russian;
• typos, abbreviations and voice notes from messengers;
• your SKUs, branch names and tariffs;
• the rule "if you do not know, hand over to a human";
• register: a model can know the grammar and still sound like a translated manual.

So for each language it matters what was tested, not only who ranked first.

Uzbek: UzLiB

UzLiB tests Uzbek linguistics: correct spelling, word meaning, meaning in context and fill-in-the-blank. The questions come from quizzes run by Uzbek language channels on Telegram, checked by hand and converted to Latin script. Models run at temperature=1.0, close to everyday use.

Source: UzLiB leaderboard, 14 October 2026. Random choice scores 0.25. The human score is the average share of correct answers per question among people who took the original quizzes.

The overall gap between the three families is small. The fill-in column is where they split: GPT-6 Luna is well behind Gemini there. The bottom of the leaderboard matters too. Flagship models from earlier generations do not guarantee good Uzbek: Claude Opus 4.5 scores 0.60 and GPT 5.2 scores 0.66. The model version moves the result more than the vendor.

UzLiB does not cover Cyrillic Uzbek, code-switching with Russian, or chat style. For a bot in Tashkent, those are the main things.

Kyrgyz: KyrgyzLLM-Bench

KyrgyzLLM-Bench is a July 2026 preprint by researchers from Kyrgyzstan, Russia and Canada. It has two natively written datasets, KyrgyzMMLU (questions from the national school testing programme, plus medicine) and KyrgyzRC (reading comprehension), alongside human-edited translations of WinoGrande, HellaSwag, BoolQ and TruthfulQA. Closed models ran on a 100-question-per-subject subset.

* The authors report that Gemini 2.5 Flash hit safety refusals on benign Kyrgyz prompts, so its scores understate the model.

Three findings carry over to product work. Reading comprehension is close to ceiling: given a passage, strong models average 91–95%. That is good news for retrieval-based assistants, where the model reads your document instead of recalling facts. Recall without context is weaker: the leader scores 67–69% on KyrgyzMMLU subjects. And closed models are unstable across prompt formats. GPT-5.1 fell from 88% to 13% on BoolQ once few-shot examples were added. It did not forget Kyrgyz; it stopped answering in the expected format. In production, only a regression suite catches that kind of failure.

These models are a generation behind today's releases, so the order may differ on current versions.

Azerbaijani: no recent comparison

The most complete comparison is TUMLU, a nine-language Turkic benchmark accepted at ACL 2025. Its questions come from school and entrance exams, written by native speakers rather than translated. Models ran 5-shot at temperature=0.

Source: TUMLU, Table 2.

Azerbaijani was easier for these models than Kyrgyz or Uzbek. But these are 2024 models, and as of 29 October 2026 we found no published comparison of current GPT, Claude and Gemini versions in Azerbaijani. If a vendor names "the best model for Azerbaijani", ask what it was tested on.

Armenian: ArmBench-LLM

ArmBench-LLM 1.0 comes from Metric AI Lab in Yerevan, published 2 April 2026. It covers classification, multiple choice, grammar and spacing correction (useful after OCR), summarisation, translation, reading comprehension and Armenian university exams.

• gemini-3-flash-preview ranks first at 0.635.
• gpt-5.2-pro is second at 0.617.
• Claude 3.7 Sonnet scores 0.607.
• Claude 4.5 and 4.6 were left out. The authors saw consistency problems serious enough to invalidate the scores.

Two of their notes matter for budgets. Gemini 3 Flash beat the pricier Gemini 3 Pro in Armenian, the reverse of general leaderboards. And the full run cost them $3.28 on Gemini 3 Flash against $160.20 on GPT-5.2 Pro. Part of that gap is tokenisation: models split Armenian script into very different numbers of tokens, so the same reply costs different amounts.

Georgian: GeoLogicQA

GeoLogicQA was built at Tbilisi State University: 100 reasoning problems in Georgian adapted from the Kangaroo maths competition and checked by native speakers. It appeared at the low-resource NLP workshop at RANLP in September 2025.

Claude was not part of the study, and GPT appears as a 2024 model. We found no published comparison of current models on Georgian customer conversations. A hundred logic problems say little about how a model answers a shopper asking about delivery to Batumi.

Summary across the five languages

No single model leads in all five languages on this evidence. The leader changes by language and by version.

Why a leaderboard is not a model decision

Take an electronics retailer in Tashkent. A customer writes: "Assalomu alaykum, konditsioner 12 lik bormi? yetkazib berish Chilonzorga qancha", then adds in Russian, "and do you have instalments?". The bot has to read apostrophe-free Latin Uzbek and Russian, parse the colloquial "12 lik", find the product, quote the shop's delivery fee and avoid inventing instalment terms. UzLiB tells you the model knows Uzbek grammar. It does not tell you whether it quotes the right delivery fee.

Three more reasons to test on your own data:

• Versions move faster than papers. Every vendor has shipped new models since KyrgyzLLM-Bench was run.
• Behaviour beats knowledge. Gemini's refusals in Kyrgyz and GPT-5.1's BoolQ collapse are not language gaps. In a chat they look like a silent bot or "I can't help with that".
• Cost depends on the script. Armenian and Georgian text often takes more tokens than English of the same length, and the ratio differs by model.

How to test a model on your own conversations before launch

This is how we choose models: build an eval set from real customer questions before committing to a vendor. The general method is in What are AI evals and why projects need them. For a language decision:

1. Collect 50–100 real messages from your support inbox, messengers and agent chats. Strip names and phone numbers. Keep the real mix: Latin and Cyrillic, code-switching, typos.
2. Write the expected outcome for each: answer, ask a clarifying question or hand over; which facts must appear; which language to reply in.
3. Add the hard cases. Questions with no answer in your knowledge base, discount requests, complaints, half-Russian messages. For voice, add 10–20 recordings and score transcription separately.
4. Run two or three candidates from the benchmarks above. For Uzbek or Armenian, include Gemini. For Kyrgyz, include Claude. Same prompt, same knowledge base, same settings.
5. Grade with a native speaker using plain labels: correct, invented fact, wrong language, sounds machine-translated, should have escalated. Once the labels are stable, an LLM judge can take over part of the work, calibrated against the human grades.
6. Price a typical conversation, not a million tokens. Tokenisation differs by language and model.
7. Keep the set after launch. Every model update and prompt change runs through it, and failures from live chats get added.

We run the same loop for Kazakh. In the Magnum employee knowledge base, 32 of 160 checks are written in Kazakh and rerun on every change. How language mixing shapes a whole project is covered in AI development in Kazakhstan: language and channel cases.

If you need an AI agent that works in Uzbek, Kyrgyz or Armenian, we run this model comparison in the first days of the pilot, before any integration work. The full engagement, from audit to support, is on the AI implementation page.

FAQ

Sources

1. Tahrirchi. UzLiB Leaderboard. Updated 14 October 2026. Model scores on the Uzbek linguistic benchmark.
2. Turatali T., Turdubaeva A., Izmailov R., Alekseev A., Nikolenko S. KyrgyzLLM-Bench: Benchmarking Kyrgyz Language Understanding. arXiv, 2026. Claude 4.5 Sonnet, GPT-5.1 and Gemini 2.5 Flash results in Kyrgyz, Gemini refusals.
3. Isbarov J. et al. TUMLU: A Unified and Native Language Understanding Benchmark for Turkic Languages. ACL 2025. Scores in Azerbaijani, Kyrgyz and Uzbek.
4. Metric AI Lab. ArmBench-LLM 1.0: Benchmarking LLMs on Armenian Language Tasks. Hugging Face, 2026. Armenian ranking, Claude 4.5/4.6 exclusion, run costs.
5. Koberidze I., Elizbarashvili A., Tsintsadze M. GeoLogicQA: A Benchmark for Evaluating Logical Reasoning in Georgian. LowResNLP, RANLP 2025. Model accuracy on Georgian reasoning tasks.