Skip to content
TheChatArchiveThe Chat Archive
Claudeclaude-3-5-sonnet-20241022· AnthropicHugging Face LMArena Human Preference 140K

After WordPiece tokenization, what does the word "Information" become?

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Googlegemma-3-27b-it’s answer.
  1. User

    After WordPiece tokenization, what does the word "Information" become?

  2. Claude Assistant

    Let me break this down step by step:

    The word "Information" would typically be tokenized by WordPiece into:

    "In" + "form" + "ation"

    So it becomes three subword tokens. WordPiece tends to split words into common subwords while keeping frequently occurring words as single tokens. The exact tokenization can sometimes vary depending on the specific WordPiece vocabulary being used, but this is the most common way "Information" would be split.

    Would you like me to explain more about how WordPiece tokenization works or why it splits words in this particular way?

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.