i still get differnt results. index should be files, columns pretok
- User
i still get differnt results. index should be files, columns pretok
Assistant
Ah, I see the confusion now! The original
from_tokstructure is a nested dictionary where:- The outer key is the
file(which should become the DataFrame's index) - The inner key is the
pretok(which should become the DataFrame's columns) - The value is
f(the value stored in each cell)
To replicate this wide format in Polars (where files are rows and pretoks are columns), we need to pivot the data. Here's the corrected approach:
# Step 1: Flatten the nested dictionary into a long format DataFrame flattened_data = [ {"file": file, "pretok": pretok, "bytes_gpt2": f} for file, file_stats in compression_stats.items() for pretok, (f, t) in file_stats.items() ] df_long = pl.DataFrame(flattened_data) # Step 2: Pivot to wide format (files as rows, pretoks as columns) from_df = ( df_long .pivot(index="file", columns="pretok", values="bytes_gpt2") .sort(pl.col("file")) # Optional: sort rows by file name ) # If you want to sort columns by "bytes_gpt2" (pretok values), it's more complex # but possible with additional steps if neededKey Differences from Your Previous Code:
- Pivoting (
pivot): Converts the long format into the desired wide format where:- Rows =
file - Columns =
pretok - Values =
bytes_gpt2
- Rows =
- No flattening ambiguity: The nested structure is preserved through pivoting.
Example Input/Output:
Suppose
compression_statsis:compression_stats = { "file1": {"pretok_a": (10, 20), "pretok_b": (15, 25)}, "file2": {"pretok_a": (5, 15), "pretok_c": (20, 30)}, }Resulting
from_df:shape: (2, 4) ┌───────┬──────────┬──────────┬──────────┐ │ file ┆ pretok_a ┆ pretok_b ┆ pretok_c │ │ --- ┆ --- ┆ --- ┆ --- │ │ str ┆ i64 ┆ i64 ┆ i64 │ ╞═══════╪══════════╪══════════╪══════════╡ │ file1 ┆ 10 ┆ 15 ┆ null │ │ file2 ┆ 5 ┆ null ┆ 20 │ └───────┴──────────┴──────────┴──────────┘
Notes:
- Missing values: Columns for
pretokvalues not present in every file will shownull. - Sorting: If you want to sort the columns by
bytes_gpt2values (e.g., largest to smallest), you’d need to compute column names dynamically, which is more involved. Let me know if that’s required! - Performance: Polars is highly optimized for this kind of operation, even with large datasets.
Let me know if you need further adjustments! 😊
- The outer key is the
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.