Skip to content
TheChatArchiveThe Chat Archive
Claudeclaude-2.1· AnthropicHugging Face Arena Human Preference 55K

How would \u123\u1234 initially be tokenized in C? Think carefully.

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with gpt-4-1106-preview’s answer.
  1. User

    How would \u123\u1234 initially be tokenized in C? Think carefully.

  2. Claude Assistant

    In C, \u123\u1234 would initially be tokenized as a single token representing an invalid universal character name.

    The \u syntax indicates the start of a universal character name, which should be followed by 4 hex digits. \u123 on its own would be a valid universal character name, but \u1234 is not valid because it has 5 hex digits.

    However, the C preprocessor and tokenizer don't know this is invalid until after the tokenization phase. The entire \u123\u1234 sequence would initially be treated as a single preprocessing token. It would not be split into separate tokens at this point.

    Only during a later phase, when the code is interpreted or compiled, would this be detected as an invalid universal character name. But the initial tokenization sees it as a single token.

    So in summary, the key thing is that initial tokenization in C does not look deeply at validity. Sequences like this are tokenized as a whole, and only later stages detect the invalid universal character name.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.