Write some python code, that will be running in a Google Colab instance, to generate a…
- User
Write some python code, that will be running in a Google Colab instance, to generate a vocabulary based on a given dataset. You will load a file, named "dataset.txt" and generate a vocabulary of subwords and save it to "vocab.txt". Do not use any library, instead writing clean and pure python code. Here is how the vocabulary should be formed:
- At first we should have all the possible ASCII characters, including the extension table. That means that the vocab will contain ASCII values from 0 to 255 as a fallback, if a text cannot be encoded afterwards
- You find the best algorithm to create a list of words and subwords that will encode the text, creating a vocabulary. We should have control parameters, such as the vocabulary size and max length of words in the vocab.
- Words and subwords will be sorted by frequencies (keeping only the first vocab_size subwords) and then sorting them alphabetically to store them into the final "vocab.txt" file. Each subword will be separated by a newline in the final file
What I recommend doing is:
- Create an algorithm for finding the most common substrings in the main string, with a size greater than the max word size
- Repeatedly trying to cut these most frequent substrings into subwords, with as much sense as possible for these cuts.
- Iteratively repeat the cutting mechanism for even the subwords created from previous cuts, if they exceed the max word size themselves. A word like: "something" could be split into subwords like: "some", "thing", for example.
- Assistant
Certainly! Below is a Python script tailored for a Google Colab environment that generates a vocabulary of subwords based on a given dataset. The script adheres to your specifications, including initializing with all ASCII characters, controlling vocabulary size and maximum subword length, and ensuring proper sorting before saving to
vocab.txt.Overview of the Approach
-
Initialization:
- ASCII Characters: Start by initializing the vocabulary with all 256 ASCII characters (from
chr(0)tochr(255)). This ensures that any text can be encoded using these fallback characters. - Parameters: Define
vocab_sizeandmax_word_lengthto control the size of the vocabulary and the maximum length of any subword, respectively.
- ASCII Characters: Start by initializing the vocabulary with all 256 ASCII characters (from
-
Data Loading:
- Dataset Reading: Read the contents of
dataset.txtinto a single string for processing.
- Dataset Reading: Read the contents of
-
Tokenization and Frequency Analysis:
- Token Representation: Represent the text as a list of characters (tokens) for efficient processing.
- Frequency Counting: Iteratively identify the most frequent substrings (up to the
max_word_length) and add them to the vocabulary. After each addition, replace occurrences of the substring in the token list with the new subword to prevent overlapping and redundancy.
-
Vocabulary Sorting and Saving:
- Sorting: After building the vocabulary, sort the subwords first by their frequency in descending order and then alphabetically.
- Saving: Write the sorted vocabulary to
vocab.txt, ensuring each subword is on a new line.
-
Iterative Subword Addition:
- The core of the algorithm involves repeatedly finding the most frequent substring and adding it to the vocabulary until the desired
vocab_sizeis reached or no more valid substrings can be found.
- The core of the algorithm involves repeatedly finding the most frequent substring and adding it to the vocabulary until the desired
Implementation Details
-
Efficiency Considerations: To manage computational complexity, especially with large datasets, the script limits the search for frequent substrings to a manageable scope by considering substrings up to
max_word_length. -
Pure Python Approach: The script avoids using external libraries, relying solely on Python's built-in functionalities to ensure compatibility and ease of execution in environments like Google Colab.
The Python Script
# Define parameters vocab_size = 1000 # Desired vocabulary size max_word_length = 10 # Maximum length of any subword # Initialize vocabulary with all ASCII characters vocab = {chr(i): 0 for i in range(256)} # Dictionary to hold subwords and their frequencies # Read the dataset with open('dataset.txt', 'r', encoding='utf-8', errors='ignore') as f: text = f.read() # Initialize the token list tokens = list(text) # Function to count substring frequencies def count_substrings(tokens, max_len): substring_freq = {} n = len(tokens) for i in range(n): for j in range(1, max_len + 1): if i + j <= n: substr = ''.join(tokens[i:i+j]) if substr in vocab: continue # Already in vocab if substr in substring_freq: substring_freq[substr] += 1 else: substring_freq[substr] = 1 return substring_freq # Iteratively add subwords to the vocab while len(vocab) < vocab_size: substring_freq = count_substrings(tokens, max_word_length) if not substring_freq: break # No more substrings to add # Find the most frequent substring most_freq_substr = max(substring_freq, key=lambda k: substring_freq[k]) freq = substring_freq[most_freq_substr] if freq < 1: break # No meaningful substrings found # Add to vocab vocab[most_freq_substr] = freq # Replace occurrences in tokens with the new subword i = 0 while i <= len(tokens) - len(most_freq_substr): # Check if the substring matches if ''.join(tokens[i:i+len(most_freq_substr)]) == most_freq_substr: # Replace with the new subword token tokens = tokens[:i] + [most_freq_substr] + tokens[i+len(most_freq_substr):] i += 1 # Move past the new token else: i += 1 # Move to the next position print(f"Vocabulary size after adding subwords: {len(vocab)}") # Now, sort the vocabulary # First, sort by frequency in descending order, then alphabetically # Create a list of tuples (subword, frequency) vocab_items = list(vocab.items()) # Sort by frequency descending, then alphabetically vocab_items.sort(key=lambda x: (-x[1], x[0])) # Take the top vocab_size subwords vocab_sorted = [item[0] for item in vocab_items[:vocab_size]] # Write to vocab.txt with open('vocab.txt', 'w', encoding='utf-8') as f: for subword in vocab_sorted: f.write(subword + '\n') print("Vocabulary has been saved to 'vocab.txt'.")Instructions to Run in Google Colab
-
Upload
dataset.txt:- In your Google Colab environment, ensure that
dataset.txtis uploaded to the current working directory. You can upload files by clicking on the folder icon on the left sidebar and then clicking the upload button.
- In your Google Colab environment, ensure that
-
Set Parameters (Optional):
- You can adjust
vocab_sizeandmax_word_lengthat the beginning of the script to suit your requirements.
- You can adjust
-
Execute the Script:
- Run the script in a Colab cell. It will process the dataset, build the vocabulary, and save it as
vocab.txtin the current directory.
- Run the script in a Colab cell. It will process the dataset, build the vocabulary, and save it as
-
Download
vocab.txt:- After execution, you can download
vocab.txtby clicking on the folder icon, locating the file, and selecting the download option.
- After execution, you can download
Notes and Considerations
-
Performance: This script is suitable for moderately sized datasets. For extremely large datasets, optimizations may be necessary to handle memory and processing time efficiently.
-
Error Handling: The script uses
errors='ignore'while reading the file to skip characters that can't be decoded in UTF-8, ensuring robustness against encoding issues. -
Extensibility: While the current implementation focuses on substrings up to a specified length, it can be extended or modified to incorporate more sophisticated tokenization strategies as needed.
I hope this script meets your requirements and provides a solid foundation for generating a customized vocabulary based on your dataset!
-