Fill this in: mark 10 random samples as dev set inputdf["split"] = "test"…
- User
Fill this in:
# mark 10 random samples as dev set input_df["split"] = "test" input_df.loc[input_df.sample(n=10, random_state=SEED).index, "split"] = "dev" # add the split column to segmented_dus_2025-07-14.csv and save the result segmented_dus = pd.read_csv("processing_data/segmented_dus_2025-07-14.csv", index_col=0) # the dataframes don't share the same number of rows, but they share the same discussion ids and turns # TODO - Assistant
To complete your TODO, we need to merge input_df (which has the split column) into segmented_dus, based on shared columns — specifically discussion_id and turn (assuming these are the common identifiers). Here's how you can fill in the TODO: pythonCopyEdit# mark 10 random samples as dev set input_df["split"] = "test" input_df.loc[input_df.sample(n=10, random_state=SEED).index, "split"] = "dev"
add the split column to segmented_dus_2025-07-14.csv and save the result
segmented_dus = pd.read_csv("processing_data/segmented_dus_2025-07-14.csv", index_col=0)
the dataframes don't share the same number of rows, but they share the same discussion ids and turns
merge the 'split' column from input_df into segmented_dus
segmented_dus = segmented_dus.merge( input_df[["discussion_id", "turn", "split"]], on=["discussion_id", "turn"], how="left" )
fill any missing split values with 'test'
segmented_dus["split"] = segmented_dus["split"].fillna("test")
save the result
segmented_dus.to_csv("processing_data/segmented_dus_with_split.csv")
This safely joins the split values and ensures that any unmatched rows default to "test". Let me know if you want a different default or a check for missing merges.
- User
Now fill in this condition:
if args.split == "dev": # filter the input_df to only include dev samples input_df = input_df[input_df["split"] == "dev"] # update the output_csv to include the split in the name args.output_csv = args.output_csv.replace(".csv", "_dev.csv") parsed_df = pd.DataFrame( columns=parsed_cols) # if the output_csv already exists, load it to avoid overwriting if os.path.exists(args.output_csv): # the parsed_df only includes the samples that were parsed before # not the entire input_df parsed_df = pd.read_csv(args.output_csv) for index, row in input_df.iterrows(): # if the discussion+turn is already parsed, skip it if CONDITION: print(f"[SKIP] discussion {row['discussion']} already parsed.") continue - Assistant
To complete the CONDITION, we need to check whether the current discussion and turn combination from input_df already exists in parsed_df. Assuming both dataframes contain the columns discussion and turn, the condition would look like this: pythonCopyEditif ( (parsed_df["discussion"] == row["discussion"]) & (parsed_df["turn"] == row["turn"]) ).any(): print(f"[SKIP] discussion {row['discussion']} already parsed.") continue
But since you need to fill in CONDITION directly in the if statement, here’s the inline version: pythonCopyEditif ((parsed_df["discussion"] == row["discussion"]) & (parsed_df["turn"] == row["turn"])).any():
Full context inserted: pythonCopyEditfor index, row in input_df.iterrows(): # if the discussion+turn is already parsed, skip it if ((parsed_df["discussion"] == row["discussion"]) & (parsed_df["turn"] == row["turn"])).any(): print(f"[SKIP] discussion {row['discussion']} already parsed.") continue
Let me know if your column names are different (e.g., discussion_id vs discussion).
- User
mark 10 random samples as dev set
input_df["split"] = "test" input_df.loc[input_df.sample(n=10, random_state=SEED).index, "split"] = "dev" # mark 10 other turns as train set # select only the rows that are not in the dev set - Assistant
To mark 10 other samples as the train set (excluding the ones already in the dev set), you can filter out the dev samples first and then randomly sample 10 from the remaining rows. Here's the completed code: pythonCopyEdit# mark 10 random samples as dev set input_df["split"] = "test" dev_indices = input_df.sample(n=10, random_state=SEED).index input_df.loc[dev_indices, "split"] = "dev"
mark 10 other turns as train set
select only the rows that are not in the dev set
remaining_for_train = input_df[~input_df.index.isin(dev_indices)] train_indices = remaining_for_train.sample(n=10, random_state=SEED + 1).index # different seed to avoid overlap input_df.loc[train_indices, "split"] = "train"
This ensures:
dev and train sets are mutually exclusive
random_state ensures reproducibility while using different seeds for each split avoids overlapping samples
Let me know if you want to balance by discussion, label, etc.