AI Language Token Tax Calculator — Sinhala, Tamil & 20+ Languages

Large language models bill per token, not per word or character. A token is a fragment of text a byte-pair-encoding (BPE) tokenizer produces from a fixed vocabulary. OpenAI publishes two vocabularies: cl100k_base (GPT-4, GPT-3.5) and o200k_base (GPT-4o and the o-series). This tool runs the real vocabularies locally through gpt-tokenizer, a JavaScript port of OpenAI's tiktoken, so the counts match what OpenAI charges you.

1. Exact token count. Your text is encoded with the selected vocabulary: tokens = encode(text).length. No API, no key, deterministic.
2. Character basis. Characters are counted as Unicode code points, so a Sinhala grapheme counts as one. Chars-per-token and tokens-per-word show how densely the vocabulary packs your script.
3. The language tax. The benchmark table tokenizes the same hand-translated sentence in each language and reports multiplier = tokens(lang) ÷ tokens(English), with English fixed at 1.00×. Petrov et al. (2023) show this ratio reaches ~15× across the world's languages, and non-Latin scripts pay the most.
4. Byte-fallback ceiling.Code points missing from the vocabulary encode as their UTF-8 bytes, and a Sinhala or Tamil character is three bytes. So the theoretical maximum token count for any text is its UTF-8 byte length. The tool cross-checks every count against that ceiling and reports how much the vocabulary's merges saved — proof the count is sane and a direct measure of why o200k_base beats cl100k_base for these scripts.
5. Cost. Per-request cost is tokens × price ÷ 1,000,000. The English estimate uses OpenAI's documented ≈4-characters-per-token rule, so the “surcharge” is the extra you pay purely for the script. Enter a monthly request volume to project the monthly language tax.

Prices default to a documented snapshot of OpenAI's published rates (verified 2026-07-18) and are editable, because published prices move more often than tokenizer vocabularies do.

• OpenAI tiktoken — official BPE tokenizer, cl100k_base & o200k_base vocabularies
• gpt-tokenizer — the ISC-licensed browser port that produces the exact counts
• Petrov, La Malfa, Torr & Bibi (2023) — Language Model Tokenizers Introduce Unfairness Between Languages (NeurIPS 2023)
• OpenAI Help Center — What are tokens and how to count them? (≈4 chars/token rule)
• OpenAI API pricing — input $/1M tokens per model

Token counts were regenerated with gpt-tokenizer and the default prices reconciled to OpenAI's published pricing on 2026-07-18. Counts are exact for OpenAI models only; Claude and Gemini use different, non-public vocabularies and are out of scope for v1.