-
Is OpenAI imposing a Token Tax on Non-English Languages?
The token count on international languages are significantly higher than English. The post discusses the issue of OpenAI's tokenizer assigning more tokens to non-English languages compared to English, effectively imposing a "token tax" on internat...The token count on international languages are significantly higher than English. The post discusses the issue of OpenAI's tokenizer assigning more tokens to non-English languages compared to English, effectively imposing a "token tax" on internat...
Page 1 of 1
Powered by Optimal Access