Skip to content
The Chat Archive
gpt-4-0613· OpenAI Arena Human Preference 55K

texts = [text.split() for text in df["textprocessed"]] wordsList = [] for text in texts:…

  1. User

    texts = [text.split() for text in df["text_processed"]]

    wordsList = [] for text in texts: wordsList.extend(text)

    wordsList = list(set(wordsList)) print(len(wordsList))

    with open("./new_ap/vocab.txt", "w") as f: for word in wordsList: f.write(word + "\n")

    this have really bad time complexity, not optimized and reeeallly slow, please fix

    with open("./new_ap/ap.dat", "w") as f: for text in texts: f.write(str(len(text)) + " ") for word in text: f.write(str(wordsList.index(word)) + ":" + str(text.count(word)) + " ") f.write("\n")

  2. Assistant

    The code you've provided is writing a vocab file and a document-word frequency file from a given dataframe. The slow part is finding the index of each word in the wordsList and counting the frequency of each word in the text.

    A more efficient approach would be to use a dictionary to store the index of each word in the wordsList. This would enable O(1) look-up time when writing the document-word frequency file. Additionally, using a Counter object to get word frequencies in a text would be more efficient than repeatedly calling the count method.

    Here is a more optimized version of your code:

    from collections import Counter
    
    # Creating a list of unique words
    wordsList = list(set([word for text in df["text_processed"] for word in text.split()]))
    
    print(len(wordsList))
    
    # Creating a dictionary with word:index pairs for O(1) lookup
    wordsIndex = {word: index for index, word in enumerate(wordsList)}
    
    # Writing the vocab file
    with open("./new_ap/vocab.txt", "w") as f:
        for word in wordsList:
            f.write(word + "\n")
    
    # Writing the document-word frequency file
    with open("./new_ap/ap.dat", "w") as f:
        for text in df["text_processed"]:
            text = text.split()
            word_frequencies = Counter(text)
            f.write(str(len(text)) + " ")
            
            for word, freq in word_frequencies.items():
                f.write(str(wordsIndex[word]) + ":" + str(freq) + " ")
            
            f.write("\n")
    

    This version of the code should be significantly faster than the original one.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.