Skip to content
TheChatArchiveThe Chat Archive
vicuna-13b· LMSYSHugging Face Arena Human Preference 55K

Can you write code to load plain text files into a hugging face data sets data set?

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Claudeclaude-1’s answer.
  1. User

    Can you write code to load plain text files into a hugging face data sets data set?

  2. Assistant

    Yes, you can use the Hugging Face's Dataset class to load plain text files into a dataset. Here's a simple example:

    from transformers import Dataset, DataLoader
    import pandas as pd
    
    # Load the plain text files into a pandas dataframe
    df = pd.read_csv("path/to/file1.txt", encoding="utf-8")
    df = pd.read_csv("path/to/file2.txt", encoding="utf-8")
    df = pd.read_csv("path/to/file3.txt", encoding="utf-8")
    
    # Create a dataset object
    dataset = Dataset.from_dataframe(df, tokenizer=None, padding=True, truncation=True)
    
    # Create a data loader
    data_loader = DataLoader(dataset, batch_size=32, shuffle=True)
    
    # Use the data loader to iterate over the dataset
    for batch in data_loader:
        print(batch)
    

    In this example, we first load the plain text files into a pandas dataframe using the pd.read_csv function. We then create a Dataset object by calling the Dataset.from_dataframe method and passing in the dataframe, along with some optional parameters (e.g. tokenizer, padding, truncation). We then create a data loader using the DataLoader class, which will load the data in batches and shuffle the order of the batches for random sampling. Finally, we iterate over the data loader using a for loop to access the batches of data.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.

Identical first prompt