split .csv file into chunks by regexp
- User
split .csv file into chunks by regexp
Assistant
To split a
.csvfile into chunks based on a regular expression, you can use Python with thepandasandrelibraries. The idea is to:- Read the CSV file.
- Identify rows that match a regex pattern.
- Group rows based on the pattern.
- Save each group to a separate file.
✅ Example Use Case
Suppose you want to split a CSV file into chunks where each chunk contains rows where a specific column (e.g.,
"category") matches a regex pattern like^A\d+,^B\d+, etc.
✅ Python Script
import pandas as pd import re import os # === CONFIGURATION === input_csv = 'data.csv' output_dir = 'chunks' pattern_column = 'category' regex_pattern = r'^(A\d+|B\d+|C\d+)' # Adjust this to your needs # ===================== # Create output directory os.makedirs(output_dir, exist_ok=True) # Read CSV df = pd.read_csv(input_csv) # Function to extract the key from the pattern def extract_key(value): match = re.match(regex_pattern, str(value)) return match.group(0) if match else 'unknown' # Add a key column df['chunk_key'] = df[pattern_column].apply(extract_key) # Group and save for key, group in df.groupby('chunk_key'): filename = f"{output_dir}/chunk_{key}.csv" group.drop(columns=['chunk_key']).to_csv(filename, index=False) print(f"Saved: {filename}")
✅ Notes
- You can change
regex_patternto match whatever grouping logic you need. - If the pattern is more complex (e.g., multiple columns or nested logic), you can adjust the
extract_keyfunction accordingly. - If you want to split by line content (not a specific column), you can read the file line-by-line and use regex directly.
✅ Alternative: Split by Line Content (No CSV Parsing)
If your CSV is simple and you want to split by regex on any line:
import re input_file = 'data.csv' output_dir = 'chunks' pattern = re.compile(r'^A\d+') # Example: lines starting with A followed by digits os.makedirs(output_dir, exist_ok=True) with open(input_file, 'r') as f: header = next(f) # Save header chunk_files = {} for line in f: match = pattern.search(line) key = match.group(0) if match else 'unknown' if key not in chunk_files: chunk_files[key] = open(f"{output_dir}/chunk_{key}.csv", 'w') chunk_files[key].write(header) chunk_files[key].write(line) # Close all files for f in chunk_files.values(): f.close()
Let me know your exact regex or column name if you want a tailored version.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.