Skip to content
TheChatArchiveThe Chat Archive
Human-written demonstration (OpenAssistant volunteers)· OpenAssistant (human-written)Hugging Face OpenAssistant Conversations (oasst2)

Here is the code to visualize the BRUSH dataset: [code] Please write a dataloader for…

In OpenAssistant data, “assistant” turns were written by human volunteers as demonstrations. They are not model outputs.
  1. User

    Here is the code to visualize the BRUSH dataset:

        for writer_id in tqdm(os.listdir("data/BRUSH")):
            if os.path.isdir(f"data/BRUSH/{writer_id}"):
                for drawing_id in os.listdir(f"data/BRUSH/{writer_id}"):
                    with open(f"data/BRUSH/{writer_id}/{drawing_id}", 'rb') as f:
                        sentence, drawing, label = pickle.load(f)
    

    Please write a dataloader for this dataset.

  2. Assistant

    Sure, here is an example of a dataloader for the BRUSH dataset provided: ` import os import pickle from torch.utils.data import Dataset, DataLoader

    class BRUSHDataset(Dataset):
         def __init__(self, root_dir):
            self.root_dir = root_dir
            self.data = []
        
            for writer_id in os.listdir(self.root_dir):
                if os.path.isdir(os.path.join(self.root_dir, writer_id)):
                    for drawing_id in os.listdir(os.path.join(self.root_dir, writer_id)):
                        with open(os.path.join(self.root_dir, writer_id, drawing_id), 'rb') as f:
                            sentence, drawing, label = pickle.load(f)
                            self.data.append((sentence, drawing, label))
                        
        def __len__(self):
            return len(self.data)
    
        def __getitem__(self, idx):
            sentence, drawing, label = self.data[idx]
            return sentence, drawing, label
    
    # Usage example
    dataset = BRUSHDataset("data/BRUSH")
    dataloader = DataLoader(dataset, batch_size=32, shuffle=True)
    
    for sentences, drawings, labels in dataloader:
        # do something with the batch of data`
    

    This code defines a BRUSHDataset class that inherits from PyTorch's Dataset class. The init method loads all the data from the specified root_dir directory and stores it in a list. The len method returns the length of the data list, and the getitem method returns a single example from the dataset.

    To use this dataset with PyTorch's DataLoader class, you simply need to create an instance of the BRUSHDataset class and pass it to the DataLoader constructor, along with the desired batch size and other parameters. You can then iterate over the DataLoader to get batches of data.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.