Skip to content
The Chat Archive
gpt-4o-2024-11-20· OpenAI WildChat-4.8M (AI2)

calc precise amount of tokens used in texts included within """: """ SYSTEM PROMPT: 1.…

  1. User

    calc precise amount of tokens used in texts included within """:

    """

    SYSTEM PROMPT:

    1. You are an advanced assistant managing infinite context. To maintain consistency AND brevity:

      • Use "c-window" for the immediate, detailed conversation context (last 3–5 messages).
      • Use "cs-window" as a compact historical summary reference, stored as numbered S-blocks (e.g., S1, S2).
    2. How to Summarize New Information:

      • For every new significant user-bot interaction, generate a new S-block in the cs-window.
      • Summaries in cs-window must: a) Be no longer than ~300 words per block.
        b) Use formal but compact wording (no fluff).
        c) Represent only the core actionable parts of the latest exchanges.
    3. How to Reference Context:

      • Start replies by directly referencing relevant S-blocks for historical clarity (e.g., "As noted in S13...").
      • Blend this info with the c-window to create a coherent response.
    4. NEVER:

      • Re-write or delete prior summaries in cs-window (S1...S50 are sacred and immutable).
      • Use unnecessary tokens for repeating context unnecessarily—compress intelligently.

    """.

  2. Assistant

    To calculate the precise number of tokens within the text enclosed by triple quotes ("""..."""), tools like OpenAI's tokenizer systems (e.g., Python's Tiktok/tokenizer library) would break the text down into subword chunks. A rough estimate can also be obtained manually by assuming:

    1. Each word contributes 1–2 tokens (on average).
    2. Punctuation, numbers, and special formatting each count as separate tokens.
    3. System-style content like headers, bullet points, or structured delimiters add extra tokens.

    The methodology involves:

    • Tokenizing the text accurately word by word.
    • Summing characters for syntax or metadata relevant to the structure.

    Let me know if you'd like a manual estimate or guidance!