Skip to content
TheChatArchiveThe Chat Archive
Databricksdolly-v2-12b· DatabricksHugging Face Arena Human Preference 55K

What is the easiest tokenizer for taking data and vectorizing it

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with alpaca-13b’s answer.
  1. User

    What is the easiest tokenizer for taking data and vectorizing it

  2. Databricks Assistant

    The Valencia tokenizer is very easy to use and is quite fast as well.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.