I have a classification dataset with 18 folders corresponding to 18 classes in it. write…
- User
I have a classification dataset with 18 folders corresponding to 18 classes in it. write a python script to divide this dataset into train, test and valid in ratio 80:10:10
- Assistant
To split a dataset consisting of different folders (each representing a class) into training, testing, and validation sets with an 80:10:10 ratio, you can use the following Python script. This script assumes that your data is structured as follows: you have a main dataset directory containing subdirectories, each named after one of your 18 classes, with corresponding images or files inside them.
We'll use the
shutilmodule to copy files and theosmodule to work with directories. We'll also use thesklearnmodule'strain_test_splitfunction for splitting. If you don't already havesklearninstalled, you can install it usingpip install scikit-learn.import os import shutil from sklearn.model_selection import train_test_split def create_train_test_val_folders(root_dir): """ Create train, test, and val directories with subdirectories for each class. """ folders = ['train', 'test', 'val'] for folder in folders: path = os.path.join(root_dir, folder) if not os.path.exists(path): os.makedirs(path) for i in range(1, 19): # Assuming classes are numbered from 1 to 18 class_path = os.path.join(path, f'class_{i}') if not os.path.exists(class_path): os.makedirs(class_path) def split_dataset(dataset_dir, train_size=0.8, test_size=0.1, val_size=0.1): """ Split the dataset into training, testing, and validation sets. """ assert train_size + test_size + val_size == 1, "The proportions must sum up to 1!" # First, create train, test, and val folders with subfolders for each class create_train_test_val_folders(dataset_dir) classes = os.listdir(dataset_dir) classes = [c for c in classes if os.path.isdir(os.path.join(dataset_dir, c))] # Filter out non-directory files if any for c in classes: current_class_dir = os.path.join(dataset_dir, c) files = os.listdir(current_class_dir) files = [f for f in files if os.path.isfile(os.path.join(current_class_dir, f))] # Split data train_files, temp_files = train_test_split(files, test_size=test_size + val_size, random_state=42) val_files, test_files = train_test_split(temp_files, test_size=val_size / (test_size + val_size), random_state=42) # Function to copy files def copy_files(files, dest_folder): for f in files: src_path = os.path.join(current_class_dir, f) dest_path = os.path.join(dest_folder, c, f) shutil.copy(src_path, dest_path) # Copy train, val, and test files copy_files(train_files, os.path.join(dataset_dir, 'train')) copy_files(val_files, os.path.join(dataset_dir, 'val')) copy_files(test_files, os.path.join(dataset_dir, 'test')) if __name__ == "__main__": # Update 'path_to_your_dataset' to the path where your dataset is stored dataset_dir = 'path_to_your_dataset' split_dataset(dataset_dir)This script accomplishes the splitting by doing the following:
- Creating necessary directories for the train, test, and validation subsets if they don't already exist.
- Iterating over each class directory in the main dataset directory to split the files into training (80%), validation (10%), and testing (10%) sets.
- Copying these files into the appropriate subdirectories under
train,test, andvaldirectories.
Make sure to set
dataset_dirto the correct path of your dataset directory. Ensure that the script has write permissions to the dataset directory to create new folders and copy files.