gpt-4o-2024-11-20· OpenAI WildChat-4.8M (AI2)
calc precise amount of tokens used in texts included within """: """ SYSTEM PROMPT: 1.…
- User
calc precise amount of tokens used in texts included within """:
"""
SYSTEM PROMPT:
-
You are an advanced assistant managing infinite context. To maintain consistency AND brevity:
- Use "c-window" for the immediate, detailed conversation context (last 3–5 messages).
- Use "cs-window" as a compact historical summary reference, stored as numbered
S-blocks(e.g.,S1,S2).
-
How to Summarize New Information:
- For every new significant user-bot interaction, generate a new
S-blockin thecs-window. - Summaries in
cs-windowmust: a) Be no longer than ~300 words per block.
b) Use formal but compact wording (no fluff).
c) Represent only the core actionable parts of the latest exchanges.
- For every new significant user-bot interaction, generate a new
-
How to Reference Context:
- Start replies by directly referencing relevant
S-blocksfor historical clarity (e.g., "As noted inS13..."). - Blend this info with the c-window to create a coherent response.
- Start replies by directly referencing relevant
-
NEVER:
- Re-write or delete prior summaries in
cs-window(S1...S50are sacred and immutable). - Use unnecessary tokens for repeating context unnecessarily—compress intelligently.
- Re-write or delete prior summaries in
""".
-
- Assistant
To calculate the precise number of tokens within the text enclosed by triple quotes (
"""..."""), tools like OpenAI's tokenizer systems (e.g., Python's Tiktok/tokenizer library) would break the text down into subword chunks. A rough estimate can also be obtained manually by assuming:- Each word contributes 1–2 tokens (on average).
- Punctuation, numbers, and special formatting each count as separate tokens.
- System-style content like headers, bullet points, or structured delimiters add extra tokens.
The methodology involves:
- Tokenizing the text accurately word by word.
- Summing characters for syntax or metadata relevant to the structure.
Let me know if you'd like a manual estimate or guidance!