i have bunch of csv files i have following code to merge them: import pandas as pd…
- User
i have bunch of csv files i have following code to merge them: import pandas as pd import os
Directory where all your CSV files are located
csv_directory = r"C:\Users\arisa\Desktop\day_spot"
List all CSV files in the given directory
csv_files = [file for file in os.listdir(csv_directory) if file.endswith('.csv')]
Placeholder for storing the data frames
data_frames = []
Loop over the list of csv files
for csv_file in csv_files: # Read the CSV file and append it to the list of data frames file_path = os.path.join(csv_directory, csv_file) df = pd.read_csv(file_path) data_frames.append(df)
Concatenate all data frames to create a single data frame
combined_df = pd.concat(data_frames, ignore_index=True)
Save the combined data frame to a new CSV file
combined_csv_path = r'C:\Users\arisa\Desktop\combined_day.csv' combined_df.to_csv(combined_csv_path, index=False)
print(f'Combined CSV created at: {combined_csv_path}')
i want to update the code so that in final file: i want to remove all the columns except the ones that: A.The columns which their names that contains: Open, High, Low, Close,macd,sma,wma,ema,rsi,cci,stoch,_short,_ichimoku,_dr B.The columns which their names that ends whith: volume_base, volume_crypto, tradecount C.The columns after y_High_1d(including y_High_1d itself)
- Assistant
To update your code to only include specific columns in the final combined CSV file, you will need to filter the columns of each DataFrame before appending it to the
data_frameslist. This can be achieved by using DataFrame column selection based on your criteria (i.e., columns containing specific substrings or columns ending with specific strings, and selecting columns after a specific column including itself). Here's how you can modify your code:import pandas as pd import os # Directory where all your CSV files are located csv_directory = r"C:\Users\arisa\Desktop\day_spot" # Specify substrings for filtering specific columns include_substrings = ["Open", "High", "Low", "Close", "macd", "sma", "wma", "ema", "rsi", "cci", "stoch", "_short", "_ichimoku", "_dr"] include_suffixes = ["volume_base", "volume_crypto", "tradecount"] # Define the column after which to include all columns (including itself) column_cutoff = "y_High_1d" # List all CSV files in the given directory csv_files = [file for file in os.listdir(csv_directory) if file.endswith('.csv')] # Placeholder for storing the data frames data_frames = [] # Loop over the list of csv files for csv_file in csv_files: # Read the CSV file file_path = os.path.join(csv_directory, csv_file) df = pd.read_csv(file_path) # Filter columns based on the criteria # A and B: Columns that contain/include specific substrings or suffixes filtered_columns = [col for col in df.columns if any(sub in col for sub in include_substrings) or col.endswith(tuple(include_suffixes))] # C: Columns after specified column (including itself) if column_cutoff in df.columns: cutoff_index = df.columns.get_loc(column_cutoff) columns_after_cutoff = df.columns[cutoff_index:] filtered_columns.extend(columns_after_cutoff) # Drop duplicate column names that might occur due to combining criteria A+B with C filtered_columns = list(dict.fromkeys(filtered_columns)) # Select only the filtered columns for the DataFrame df_filtered = df[filtered_columns] data_frames.append(df_filtered) # Concatenate all filtered data frames combined_df = pd.concat(data_frames, ignore_index=True) # Save the combined DataFrame to a new CSV file combined_csv_path = r'C:\Users\arisa\Desktop\combined_day.csv' combined_df.to_csv(combined_csv_path, index=False) print(f'Combined CSV created at: {combined_csv_path}')What This Modified Code Does:
- It iterates through each CSV file to be merged.
- For each CSV file, it filters columns based on the specified criteria (A, B, and C).
- It ensures no duplicate columns are added when combining criteria A and B with C.
- It then appends the filtered DataFrame to the
data_frameslist. - Finally, it combines all data frames into a single one and writes it to a new CSV file.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.