Skip to content
Indian Teams Now Build Tokenizers That Read Tamil and Telugu in About Two Tokens a Word, Against Four or More for Some Global Ones
AI & Technology

Indian Teams Now Build Tokenizers That Read Tamil and Telugu in About Two Tokens a Word, Against Four or More for Some Global Ones

Illustration by tuput

English

Gnani.ai, Sarvam and Krutrim AI have each published a tokenizer that cuts the number of pieces a chatbot needs for Indian-language text. Gnani's, released on 19 May 2026, reports about 2.0 tokens per word for Tamil and Telugu where the tokenizer it started from needed 3.7 and 3.9.

· · 9 min read

Indian research teams have rebuilt the first step a chatbot performs, the tokenizer that cuts text into pieces, so that Tamil and Telugu need about two pieces per word. Researchers at Gnani.ai and NVIDIA reported on 19 May 2026 an average of 1.9969 tokens per word for Tamil and 2.0632 for Telugu. Nemotron 3 Nano’s own tokenizer, the one they started from, needed 3.7480 and 3.9212 on the same test text.

Two other Indian efforts point the same way. Sarvam AI published a tokenizer with Sarvam-1 on 24 October 2024 and reported 1.4 to 2.1 tokens per word across ten Indian languages. Researchers at Krutrim AI in Bangalore posted a paper in November 2025 whose tokenizer, MUTANT-Indic, cuts the average count across English, 22 Indian languages and code by 39.5% against the tokenizer of Meta’s LLaMA 4, by the authors’ own measurement.

Every fertility figure below comes from the team that built the tokenizer, so each is a maker’s claim.

What a token is and what fertility counts

A language model never sees letters or whole words. A tokenizer first cuts the text into tokens, and the model reads and writes only those. OpenAI’s help centre says a token can be a character, part of a word, a whole word or a piece of punctuation. For English it gives rough rules: one token is about four characters or three-quarters of a word, so 100 tokens are about 75 words. Other languages, it adds, can have different relationships between characters, words and tokens.

Most current tokenizers use a method called byte pair encoding. The Hugging Face documentation describes it: start from single characters, then repeatedly merge the pair that appears next to each other most often in a sample of text, until the vocabulary reaches a target size. Common words end up as single tokens, and rare words are split into pieces. A script that is plentiful in the sample gets long, efficient tokens. A script that is scarce gets cut into small ones.

Fertility is the measure of that effect: the average number of tokens needed per word. Both Sarvam and Gnani define it the same way, and Sarvam puts English at about 1.4. A lower number means the same sentence takes fewer tokens. For background on how a model uses those tokens, see how a chatbot predicts the next word and how a large language model is trained.

Why the bill and the memory both count tokens

Tokens are the unit of billing and of memory. Token-based pricing, OpenAI’s help centre says, depends on the model and the kind of token, with input and output priced separately. Anthropic’s pricing page quotes its rates per million tokens. Tokens also set the context window, the amount a model can hold in view at once.

A further warning on that page applies to any comparison of vendors: a lower price per million tokens does not guarantee a lower total cost, because models can tokenize the same text differently. Anthropic’s pricing page makes a similar point about its own models, saying its newer tokenizer produces approximately 30% more tokens for the same text.

In Gnani’s worked example, a 1,000-word Telugu document comes to about 2,063 tokens with its tokenizer, 3,921 with the unmodified Nemotron tokenizer and 4,753 with the tokenizer of Qwen 3.5. That is roughly half the inference cost for identical content, Gnani says. At its average of 1.88 tokens per word against the baseline’s 3.20, Gnani adds, about 37% more words fit in a fixed token budget.

The 2023 study that measured the gap

Orevaoghene Ahia and six co-authors supplied the academic anchor in a paper posted to arXiv on 23 May 2023 and published at the EMNLP 2023 conference. A parallel set of sentences in many languages went through the tokenizer behind ChatGPT’s gpt-3.5-turbo model. Languages with their own scripts, Telugu among them, needed up to five times as many tokens as Latin-script languages for the same information, the authors found.

On the XLSum summarization benchmark they report spending up to four times more for both prompting and generation in Telugu and Amharic. Their abstract concludes that speakers of a large number of the supported languages were overcharged while obtaining poorer results, and that these speakers tended to come from regions where the APIs were less affordable to begin with. Training-data imbalance is the cause the authors hypothesize, since the tokenizer was primarily trained on Latin-script languages, mainly English.

That study covers one vendor’s tokenizer as it stood in 2023. Krutrim’s table, further down, includes OpenAI’s open model GPT-OSS, which needs 1.72 tokens a word for Hindi and 6.26 for Odia.

Sarvam: a tokenizer built before the model

Sarvam-1, released as an open model on 24 October 2024, is a 2-billion-parameter model for ten Indian languages and English. Existing multilingual models often need 4 to 8 tokens per word for Indic scripts, Sarvam says, while its own tokenizer reaches 1.4 to 2.1 across all supported languages. The vocabulary holds 68,096 tokens, 4,096 of them reserved for later additions.

The company argues that this also stretches its training data. The 2-trillion-token Indic corpus, it estimates, gives a training signal equivalent to 6 to 8 trillion tokens passed through other popular tokenizers. Training took 1,024 GPUs of Yotta’s Shakti cluster over five days.

On 6 March 2026 Sarvam released two larger open models, Sarvam 30B and Sarvam 105B, with a tokenizer that covers all 22 scheduled Indian languages across 12 scripts. According to the post, it outperforms other open-source tokenizers and is significantly more efficient for low-resource languages such as Odia, Santali and Manipuri (Meitei). The post shows the comparison as a chart with no values in the text, so no figures can be quoted from it. Sarvam also reports that the tokenizer widens its measured serving advantage for the 30B model by as much as 10 times on Indian-language generation. For where these models sit in the national programme, see India is building AI in its own languages.

Gnani: extending a tokenizer without retraining the model

Gnani.ai builds voice and conversational AI for Indian languages. Gnani’s post, with NVIDIA authors and dated 19 May 2026, starts from the tokenizer of NVIDIA’s Nemotron 3 Nano, which was inherited from a Mistral model and trained mainly on English and European text. A single Indian-language word can expand to four or more tokens under it, Gnani says. Gnani’s goal was to extend the tokenizer for Hindi, Bengali, Tamil and Telugu without retraining the model or changing its architecture.

Two methods were compared. One continues byte pair encoding on the existing merge table using Indian-language text, aiming for more than 32,000 new merge rules. The other trains a separate Indian-language tokenizer and appends its tokens to the vocabulary as whole strings. Tamil supplies Gnani’s example. The word வாடிக்கையாளர், “customer”, is common in its business deployments. The suffixed form வாடிக்கையாளர்களுக்கு, “to the customers”, may never have appeared in training, and under the second method such a word can fall back to character-by-character pieces. Under the first method, Gnani says, it is built from learned parts.

Reported results favour the first method, which Gnani calls Indic Continued BPE. An average of 1.8802 across the four languages compares with 2.0063 for Sarvam 30B, 2.1132 for Gemma 3 27B, 2.79 for GPT-OSS-20B and 3.1992 for unmodified Nemotron. Per language, Hindi falls from 2.0578 to 1.4899, Bengali from 3.0697 to 1.9709, Tamil from 3.7480 to 1.9969 and Telugu from 3.9212 to 2.0632. On Hindi (1.4093) and Bengali (1.6942), Gnani’s own table puts Sarvam 30B ahead, and the post says its training weights favoured the Dravidian languages.

Gnani’s text says it compared against eight public models in one place and seven in another, and speaks of nine tokenizers in all, while the table lists eight rows, two of them Gnani’s own. The table also gives identical Hindi and Bengali values for Sarvam 30B and Gemma 3 27B. The post does not explain either point.

Krutrim’s MUTANT, and why the tables do not line up

MUTANT, a paper by Souvik Rana, Arul Menezes, Ashish Kulkarni, Chandra Khatri and Shubham Agarwal of Krutrim AI, appeared on arXiv on 5 November 2025 under the title IndicSuperTokenizer. The version revised on 22 March 2026 carries the name MUTANT and presents a recipe: a vocabulary allocated by script, language-aware pre-tokenization, and two stages of training, first word pieces and then frequent multi-word units.

The abstract reports a 39.5% better average fertility than LLaMA 4 and 18% better than Sutra, a tokenizer the authors call the current best, and a 44% gain in inference throughput over LLaMA 4 while keeping comparable results on English and Indian-language benchmarks. That comparison covers English, 22 Indian languages and code. In the paper’s Table 3, MUTANT-Indic has the lowest fertility in 20 of 24 columns. For Tamil it reports 2.12 against 5.87 for LLaMA 4, for Telugu 1.88 against 4.53, and for Hindi 1.23 against 1.83. The text singles out LLaMA 4 on Oriya, at 10.51.

Three sources cannot be merged into one league table. GPT-OSS gets 3.13 for Telugu in the MUTANT paper and 3.90 in Gnani’s post. Sarvam’s 2024 tokenizer is reported at 1.4 to 2.1 across its ten languages by Sarvam and at 2.63 for Telugu in the MUTANT table. Test text, languages and the word-counting method differ in each, which is why only the figures within a single table are comparable. None has been reproduced by an outside group in the sources opened for this article.

What fewer tokens changes and what it leaves alone

A smaller token count lowers cost and fits more text in the context window. Better answers do not follow by themselves. MUTANT’s authors report that their tokenizer kept results comparable on the benchmarks they tested, which is the claim to look for in any such paper. Gnani argues that no amount of fine-tuning on Indian-language data compensates for a tokenizer that fragments the text before the model sees it.

Tokenization also shapes tasks that have nothing to do with price. A February 2024 paper by Aaditya Singh and DJ Strouse found that how GPT-3.5 and GPT-4 split numbers into tokens changes their accuracy on arithmetic, with right-to-left grouping giving largely improved performance.

MUTANT’s authors say they will release their evaluation dataset and framework, so that other groups can run the same comparison.

Share
Copied!

Sources & further reading

  1. Gnani AI and NVIDIA, Building a State-of-the-Art Indic Tokenizer for Nemotron 3 Nano (19 May 2026)
  2. Sarvam AI, Sarvam 1: The first Indian language LLM (24 October 2024)
  3. Sarvam AI, Sarvam 30B and Sarvam 105B (6 March 2026)
  4. Rana, Menezes, Kulkarni, Khatri and Agarwal, MUTANT: A Recipe for Multilingual Tokenizer Design (arXiv 2511.03237, v2 of 22 March 2026)
  5. arXiv 2511.03237v1, IndicSuperTokenizer: An Optimized Tokenizer for Indic Multilingual LLMs (5 November 2025)
  6. Ahia et al., Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models (arXiv, 23 May 2023)
  7. ACL Anthology, Ahia et al., EMNLP 2023 proceedings entry
  8. Singh and Strouse, Tokenization counts: the impact of tokenization on arithmetic in frontier LLMs (arXiv, 22 February 2024)
  9. OpenAI Help Center, Understanding and counting tokens
  10. Claude Platform Docs, Pricing
  11. Hugging Face Transformers documentation, Tokenization algorithms

Researched and written with the help of AI tools and edited for accuracy. Provided for general information and discussion only, not professional advice. See our editorial standards and disclaimer. Spotted an error? Tell us.

#tokenizer#fertility#sarvam#gnani ai#indic languages#large language models#krutrim#tamil

Enjoyed this? Get the next one.

One good read at a time, straight to your inbox. No spam, unsubscribe anytime.

More in AI & Technology
Eight Google Authors Posted the Transformer Paper on 12 June 2017, and Its Big Model Scored 28.4 BLEU in English to German After 3.5 Days on Eight GPUs
A walk through the design behind today's chatbots, using the paper's own tables and Google's own example of a word that has to look at a river. It also covers the parts of the 2017 design that later models kept, changed or dropped.
Retrieval-Augmented Generation, Described in a 22 May 2020 Paper by Patrick Lewis and 11 Co-Authors, Lets a Chatbot Fetch Passages From an Index of 21 Million Before It Answers
A plain-words walk through the four steps behind chatbots that consult documents: the 2020 experiment that swapped one Wikipedia dump for another, the way cutting text into chunks loses context, and the measured fixes.
Sarvam's Saaras V4 Transcribes All 22 Scheduled Languages in Five Output Formats, and Its Bulbul V4 Flash Voice Model Starts Speaking in Under 250 Milliseconds, Sarvam Reports
One model listens and another talks, which is enough to run a phone call in Hindi, Tamil or Bengali. The release details, Sarvam's case that word error rate misjudges Indian speech, and the public tests that stand behind the claims.
← all articles