100k characters = ~25k input token??? is this a joke?
- User
100k characters = ~25k input token??? is this a joke?
- Assistant
The relationship between characters and tokens isn't a fixed ratio because tokens can be as short as one character or as long as a whole word or more, depending on the tokenizer and the language.
However, as a rough estimate for English text:
- 1 token ≈ 4 characters on average.
So, 100,000 characters ÷ 4 ≈ 25,000 tokens.
This is not a joke but a typical approximation used in many NLP contexts, including OpenAI's token counting. The exact number can vary depending on the text's complexity, punctuation, and language.
If you want a more precise token count, you can use tokenizers like the one from the
tiktokenlibrary (used by OpenAI) to tokenize your specific text.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.