texts = [text.split() for text in df["textprocessed"]] wordsList = [] for text in texts:…
- User
texts = [text.split() for text in df["text_processed"]]
wordsList = [] for text in texts: wordsList.extend(text)
wordsList = list(set(wordsList)) print(len(wordsList))
with open("./new_ap/vocab.txt", "w") as f: for word in wordsList: f.write(word + "\n")
this have really bad time complexity, not optimized and reeeallly slow, please fix
with open("./new_ap/ap.dat", "w") as f: for text in texts: f.write(str(len(text)) + " ") for word in text: f.write(str(wordsList.index(word)) + ":" + str(text.count(word)) + " ") f.write("\n")
- Assistant
The code you've provided is writing a vocab file and a document-word frequency file from a given dataframe. The slow part is finding the index of each word in the wordsList and counting the frequency of each word in the text.
A more efficient approach would be to use a dictionary to store the index of each word in the wordsList. This would enable O(1) look-up time when writing the document-word frequency file. Additionally, using a Counter object to get word frequencies in a text would be more efficient than repeatedly calling the count method.
Here is a more optimized version of your code:
from collections import Counter # Creating a list of unique words wordsList = list(set([word for text in df["text_processed"] for word in text.split()])) print(len(wordsList)) # Creating a dictionary with word:index pairs for O(1) lookup wordsIndex = {word: index for index, word in enumerate(wordsList)} # Writing the vocab file with open("./new_ap/vocab.txt", "w") as f: for word in wordsList: f.write(word + "\n") # Writing the document-word frequency file with open("./new_ap/ap.dat", "w") as f: for text in df["text_processed"]: text = text.split() word_frequencies = Counter(text) f.write(str(len(text)) + " ") for word, freq in word_frequencies.items(): f.write(str(wordsIndex[word]) + ":" + str(freq) + " ") f.write("\n")This version of the code should be significantly faster than the original one.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.