o1-preview-2024-09-12· OpenAI WildChat-4.8M (AI2)
Generate code for a simple ai language model in python
- User
Generate code for a simple ai language model in python
- Assistant
Certainly! Below is a Python code for a simple n-gram language model. This model will learn the probabilities of word sequences from a given text corpus and then use those probabilities to generate new text.
In this example, we'll use a bigram model (n=2), which looks at pairs of words to predict the next word.
import random import nltk from nltk.util import ngrams from collections import defaultdict, Counter # Ensure you have the necessary NLTK data files nltk.download('punkt') class NGramLanguageModel: def __init__(self, n): self.n = n # The 'n' in n-gram self.ngrams = defaultdict(Counter) self.vocab = set() def train(self, text): """ Train the n-gram language model with the given text. :param text: The training text as a string. """ tokens = nltk.word_tokenize(text.lower()) self.vocab.update(tokens) # Create n-grams from the tokens n_grams = ngrams(tokens, self.n, pad_left=True, pad_right=True, left_pad_symbol='<s>', right_pad_symbol='</s>') for gram in n_grams: prefix = gram[:-1] word = gram[-1] self.ngrams[prefix][word] += 1 def generate(self, num_words=50): """ Generate text using the trained language model. :param num_words: The number of words to generate. :return: A generated text string. """ current = ('<s>',) * (self.n - 1) generated_words = [] for _ in range(num_words): next_words = self.ngrams[current] if not next_words: break # Cannot proceed if there are no next words next_word = random.choices(list(next_words.keys()), weights=next_words.values())[0] if next_word == '</s>': break generated_words.append(next_word) current = (*current[1:], next_word) return ' '.join(generated_words) # Example usage: if __name__ == "__main__": # Sample training text training_text = """ Once upon a time, there was a kingdom far away. The king and the queen ruled the land with kindness. The people of the kingdom were happy and prosperous. One day, a dragon appeared in the kingdom. """ # Create and train the model model = NGramLanguageModel(n=2) # Bigram model model.train(training_text) # Generate text generated_text = model.generate(num_words=30) print("Generated Text:") print(generated_text)Explanation:
-
Imports:
randomfor random word selection.nltkfor tokenizing text and handling n-grams.defaultdictandCounterfor counting occurrences of word sequences.
-
NGramLanguageModel Class:
- The constructor
__init__initializes the n-gram model. - The
trainmethod processes the input text:- Tokenizes the text into words.
- Updates the vocabulary.
- Creates n-grams with padding symbols
<s>(start) and</s>(end). - Counts occurrences of each n-gram prefix and next word.
- The
generatemethod produces new text:- Starts with the start symbol(s).
- At each step, chooses the next word based on the learned probabilities.
- Stops if it reaches the end symbol
</s>or after generating the specified number of words.
- The constructor
-
Example Usage:
- A sample text is used to train the model.
- The model generates new text based on the learned probabilities.
Note: Before running this code, ensure that you have the NLTK library installed and that you've downloaded the necessary data files with
nltk.download('punkt').Output Example:
Generated Text: one day , a dragon appeared in the kingdom . the king and the queen ruled the land with kindness .This simple n-gram model learns word transition probabilities from the training text and can generate new sentences that mimic the style of the input.
-