Skip to content
TheChatArchiveThe Chat Archive
Qwenqwq-32b· Alibaba (Qwen)Hugging Face LMArena Human Preference 140K

i still get differnt results. index should be files, columns pretok

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Qwenqwen-max-2025-01-25’s answer.
  1. User

    i still get differnt results. index should be files, columns pretok

  2. Qwen Assistant

    Ah, I see the confusion now! The original from_tok structure is a nested dictionary where:

    • The outer key is the file (which should become the DataFrame's index)
    • The inner key is the pretok (which should become the DataFrame's columns)
    • The value is f (the value stored in each cell)

    To replicate this wide format in Polars (where files are rows and pretoks are columns), we need to pivot the data. Here's the corrected approach:

    # Step 1: Flatten the nested dictionary into a long format DataFrame
    flattened_data = [
        {"file": file, "pretok": pretok, "bytes_gpt2": f}
        for file, file_stats in compression_stats.items()
        for pretok, (f, t) in file_stats.items()
    ]
    df_long = pl.DataFrame(flattened_data)
    
    # Step 2: Pivot to wide format (files as rows, pretoks as columns)
    from_df = (
        df_long
        .pivot(index="file", columns="pretok", values="bytes_gpt2")
        .sort(pl.col("file"))  # Optional: sort rows by file name
    )
    
    # If you want to sort columns by "bytes_gpt2" (pretok values), it's more complex
    # but possible with additional steps if needed
    

    Key Differences from Your Previous Code:

    1. Pivoting (pivot): Converts the long format into the desired wide format where:
      • Rows = file
      • Columns = pretok
      • Values = bytes_gpt2
    2. No flattening ambiguity: The nested structure is preserved through pivoting.

    Example Input/Output:

    Suppose compression_stats is:

    compression_stats = {
        "file1": {"pretok_a": (10, 20), "pretok_b": (15, 25)},
        "file2": {"pretok_a": (5, 15), "pretok_c": (20, 30)},
    }
    

    Resulting from_df:

    shape: (2, 4)
    ┌───────┬──────────┬──────────┬──────────┐
    │ file  ┆ pretok_a ┆ pretok_b ┆ pretok_c │
    │ ---   ┆ ---      ┆ ---      ┆ ---      │
    │ str   ┆ i64      ┆ i64      ┆ i64      │
    ╞═══════╪══════════╪══════════╪══════════╡
    │ file1 ┆ 10       ┆ 15       ┆ null     │
    │ file2 ┆ 5        ┆ null     ┆ 20       │
    └───────┴──────────┴──────────┴──────────┘
    

    Notes:

    • Missing values: Columns for pretok values not present in every file will show null.
    • Sorting: If you want to sort the columns by bytes_gpt2 values (e.g., largest to smallest), you’d need to compute column names dynamically, which is more involved. Let me know if that’s required!
    • Performance: Polars is highly optimized for this kind of operation, even with large datasets.

    Let me know if you need further adjustments! 😊

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.