Skip to content
TheChatArchiveThe Chat Archive
gpt-4-0125-preview· OpenAIAi2 WildChat-4.8M (AI2)

I have a classification dataset with 18 folders corresponding to 18 classes in it. write…

  1. User

    I have a classification dataset with 18 folders corresponding to 18 classes in it. write a python script to divide this dataset into train, test and valid in ratio 80:10:10

  2. Assistant

    To split a dataset consisting of different folders (each representing a class) into training, testing, and validation sets with an 80:10:10 ratio, you can use the following Python script. This script assumes that your data is structured as follows: you have a main dataset directory containing subdirectories, each named after one of your 18 classes, with corresponding images or files inside them.

    We'll use the shutil module to copy files and the os module to work with directories. We'll also use the sklearn module's train_test_split function for splitting. If you don't already have sklearn installed, you can install it using pip install scikit-learn.

    import os
    import shutil
    from sklearn.model_selection import train_test_split
    
    def create_train_test_val_folders(root_dir):
        """
        Create train, test, and val directories with subdirectories for each class.
        """
        folders = ['train', 'test', 'val']
        for folder in folders:
            path = os.path.join(root_dir, folder)
            if not os.path.exists(path):
                os.makedirs(path)
            for i in range(1, 19):  # Assuming classes are numbered from 1 to 18
                class_path = os.path.join(path, f'class_{i}')
                if not os.path.exists(class_path):
                    os.makedirs(class_path)
    
    def split_dataset(dataset_dir, train_size=0.8, test_size=0.1, val_size=0.1):
        """
        Split the dataset into training, testing, and validation sets.
        """
        assert train_size + test_size + val_size == 1, "The proportions must sum up to 1!"
        
        # First, create train, test, and val folders with subfolders for each class
        create_train_test_val_folders(dataset_dir)
        
        classes = os.listdir(dataset_dir)
        classes = [c for c in classes if os.path.isdir(os.path.join(dataset_dir, c))]  # Filter out non-directory files if any
        
        for c in classes:
            current_class_dir = os.path.join(dataset_dir, c)
            files = os.listdir(current_class_dir)
            files = [f for f in files if os.path.isfile(os.path.join(current_class_dir, f))]
            
            # Split data
            train_files, temp_files = train_test_split(files, test_size=test_size + val_size, random_state=42)
            val_files, test_files = train_test_split(temp_files, test_size=val_size / (test_size + val_size), random_state=42)
            
            # Function to copy files
            def copy_files(files, dest_folder):
                for f in files:
                    src_path = os.path.join(current_class_dir, f)
                    dest_path = os.path.join(dest_folder, c, f)
                    shutil.copy(src_path, dest_path)
            
            # Copy train, val, and test files
            copy_files(train_files, os.path.join(dataset_dir, 'train'))
            copy_files(val_files, os.path.join(dataset_dir, 'val'))
            copy_files(test_files, os.path.join(dataset_dir, 'test'))
    
    if __name__ == "__main__":
        # Update 'path_to_your_dataset' to the path where your dataset is stored
        dataset_dir = 'path_to_your_dataset'
        split_dataset(dataset_dir)
    

    This script accomplishes the splitting by doing the following:

    1. Creating necessary directories for the train, test, and validation subsets if they don't already exist.
    2. Iterating over each class directory in the main dataset directory to split the files into training (80%), validation (10%), and testing (10%) sets.
    3. Copying these files into the appropriate subdirectories under train, test, and val directories.

    Make sure to set dataset_dir to the correct path of your dataset directory. Ensure that the script has write permissions to the dataset directory to create new folders and copy files.