Train monolingual FastText word embeddings for English and Hindi using appropriate…
- User
Train monolingual FastText word embeddings for English and Hindi using appropriate corpora. If computational resources are limited, you may use pre-trained FastText embeddings. If training your own embeddings, you 10,000 articles from Wikipedia for each language. b. Limit your vocabulary to the top 100,000 most frequent words in each language. c. Extract a list of word translation pairs from the MUSE dataset to use as a bilingual lexicon for supervised alignment.
- Assistant
Training monolingual FastText word embeddings for English and Hindi, limiting the vocabulary, and extracting word translation pairs from the MUSE dataset can be accomplished using Python and relevant libraries such as Gensim. Below is a step-by-step guide to help you through the process.
Prerequisites
-
Environment Setup:
- Python 3.x: Ensure you have Python installed. You can download it from here.
- Install Required Libraries:
pip install gensim nltk - Additional Tools:
- Wikipedia Dumps: If you plan to train your embeddings, download Wikipedia dumps for English and Hindi.
- MUSE Dataset: Download the MUSE dataset from Facebook Research.
-
Computational Resources:
- Training Embeddings: Requires significant computational power. If resources are limited, it's recommended to use pre-trained FastText embeddings.
A. Training Monolingual FastText Word Embeddings
You have two options:
- Using Pre-trained FastText Embeddings
- Training Your Own FastText Embeddings
Option 1: Using Pre-trained FastText Embeddings
FastText provides pre-trained embeddings for many languages, including English and Hindi. This is the recommended approach if computational resources are limited.
-
Download Pre-trained FastText Embeddings:
- English:
- Hindi:
-
Load Pre-trained Embeddings using Gensim:
from gensim.models import KeyedVectors # Load English embeddings english_vectors = KeyedVectors.load_word2vec_format('cc.en.300.vec', binary=False) # Load Hindi embeddings hindi_vectors = KeyedVectors.load_word2vec_format('cc.hi.300.vec', binary=False)
Option 2: Training Your Own FastText Embeddings
If you prefer to train your own embeddings using specific corpora (e.g., 10,000 Wikipedia articles), follow these steps.
-
Prepare the Corpus:
- Download Wikipedia Articles:
- English and Hindi Wikipedia dumps can be downloaded from Wikimedia Downloads.
- Extract 10,000 articles for each language. You can use the
wikiextractortool:git clone https://github.com/attardi/wikiextractor.git cd wikiextractor python3 WikiExtractor.py -o extracted_en path_to_english_dump --min_articles=10000 python3 WikiExtractor.py -o extracted_hi path_to_hindi_dump --min_articles=10000
- Download Wikipedia Articles:
-
Preprocess the Data:
- Tokenization and Cleaning: Ensure the text is tokenized and clean. For example, using NLTK:
import os import nltk from nltk.tokenize import word_tokenize nltk.download('punkt') def preprocess_corpus(input_dir, output_file): with open(output_file, 'w', encoding='utf-8') as outfile: for root, dirs, files in os.walk(input_dir): for file in files: with open(os.path.join(root, file), 'r', encoding='utf-8') as infile: for line in infile: tokens = word_tokenize(line.lower()) outfile.write(' '.join(tokens) + '\n') # Preprocess English Corpus preprocess_corpus('extracted_en', 'processed_en.txt') # Preprocess Hindi Corpus preprocess_corpus('extracted_hi', 'processed_hi.txt')
- Tokenization and Cleaning: Ensure the text is tokenized and clean. For example, using NLTK:
-
Train FastText Embeddings using Gensim:
from gensim.models import FastText import gensim def train_fasttext(input_file, model_path, vector_size=300, window=5, min_count=5, workers=4): sentences = gensim.models.word2vec.LineSentence(input_file) model = FastText(sentences=sentences, vector_size=vector_size, window=window, min_count=min_count, workers=workers) model.save(model_path) # Train English FastText Model train_fasttext('processed_en.txt', 'fasttext_en.model') # Train Hindi FastText Model train_fasttext('processed_hi.txt', 'fasttext_hi.model') -
Load Trained Models:
# Load English model fasttext_en = FastText.load('fasttext_en.model') # Load Hindi model fasttext_hi = FastText.load('fasttext_hi.model')
B. Limiting Vocabulary to the Top 100,000 Most Frequent Words
Whether using pre-trained embeddings or your own, you might want to restrict the vocabulary to the top 100,000 most frequent words to manage memory and focus on more relevant words.
If Using Pre-trained Embeddings:
- Filter Vocabulary:
def filter_top_n(words_vectors, top_n=100000): # Assuming the words are ordered by frequency most_common_words = words_vectors.index_to_key[:top_n] words_vectors = words_vectors.key_to_index.keys() filtered_vectors = {word: words_vectors[word] for word in most_common_words} return filtered_vectors # For Gensim 4.x def filter_gensim_model(model, top_n=100000): restricted_vocab = model.index_to_key[:top_n] model.vectors = model.vectors[:top_n] model.index_to_key = restricted_vocab model.key_to_index = {word: idx for idx, word in enumerate(model.index_to_key)} return model # Apply filtering english_filtered = filter_gensim_model(english_vectors, top_n=100000) hindi_filtered = filter_gensim_model(hindi_vectors, top_n=100000) # Save the filtered models english_filtered.save('filtered_fasttext_en.model') hindi_filtered.save('filtered_fasttext_hi.model')
If Training Your Own Embeddings:
-
Frequency-Based Filtering During Training: Gensim's
FastTextautomatically filters out infrequent words based on themin_countparameter. To approximate the top 100,000 words, you may need to adjustmin_countor perform post-training filtering as shown above. -
Post-Training Filtering: After training, follow the same steps as above to restrict the vocabulary.
C. Extracting Word Translation Pairs from the MUSE Dataset for Bilingual Lexicon
The MUSE (Multilingual Unsupervised and Supervised Embeddings) dataset provides bilingual dictionaries which can be used for supervised alignment of word embeddings.
1. Download the MUSE Dataset
-
Clone the MUSE Repository:
git clone https://github.com/facebookresearch/MUSE.git cd MUSE -
Download the Bilingual Dictionaries: The bilingual dictionaries are usually located in the
datadirectory. Alternatively, you can download them directly from MUSE's website.
2. Extract Word Translation Pairs
Assuming you have downloaded the dictionary files, typically in
.txtformat with one pair per line separated by a tab.def extract_translation_pairs(dictionary_file, language1, language2, output_file=None): translation_pairs = [] with open(dictionary_file, 'r', encoding='utf-8') as f: for line in f: word1, word2 = line.strip().split('\t')[:2] translation_pairs.append((word1, word2)) if output_file: with open(output_file, 'w', encoding='utf-8') as out: for pair in translation_pairs: out.write(f"{pair[0]}\t{pair[1]}\n") return translation_pairs # Example Usage: # Replace 'en-hi.txt' with the actual path to the MUSE English-Hindi dictionary translation_pairs = extract_translation_pairs('data/dictionaries/en-hi.txt', 'en', 'hi', 'en-hi_pairs.txt') print(f"Number of translation pairs extracted: {len(translation_pairs)}")3. Utilizing the Bilingual Lexicon for Supervised Alignment
Once you have the translation pairs, they can be used to align English and Hindi embeddings in a shared vector space using techniques like Procrustes alignment.
Here's a simplified example using Gensim and NumPy:
import numpy as np from sklearn.preprocessing import normalize def get_embedding_matrix(pairs, model_en, model_hi): X = [] Y = [] for en_word, hi_word in pairs: if en_word in model_en and hi_word in model_hi: X.append(model_en[en_word]) Y.append(model_hi[hi_word]) return np.array(X), np.array(Y) # Load filtered FastText models model_en = KeyedVectors.load('filtered_fasttext_en.model') model_hi = KeyedVectors.load('filtered_fasttext_hi.model') # Get embedding matrices X, Y = get_embedding_matrix(translation_pairs, model_en, model_hi) # Normalize the embeddings X = normalize(X) Y = normalize(Y) # Compute the orthogonal Procrustes problem def procrustes_alignment(X, Y): # Compute the matrix for the linear transformation M = np.dot(Y.T, X) U, _, Vt = np.linalg.svd(M) # Compute orthogonal matrix W = np.dot(U, Vt) return W W = procrustes_alignment(X, Y) # Align Hindi embeddings to English model_hi_aligned = model_hi.copy() model_hi_aligned.vectors = np.dot(model_hi.vectors, W) # Save the aligned Hindi model model_hi_aligned.save('aligned_fasttext_hi.model')Note: This is a simplified example. In practice, you'll want to use more sophisticated methods and handle edge cases, potentially using libraries like Aligned Word Embeddings.
Summary
-
Training Embeddings:
- Pre-trained Embeddings: Recommended if computational resources are limited.
- Training Own Embeddings: Involves preprocessing the corpus and using Gensim's FastText implementation.
-
Limiting Vocabulary:
- During Training: Use
min_countto filter infrequent words. - Post-training: Manually restrict the vocabulary to the top 100,000 words.
- During Training: Use
-
Extracting Translation Pairs:
- MUSE Dataset: Provides bilingual dictionaries which can be extracted and used for supervised alignment.
By following the steps outlined above, you should be able to prepare monolingual FastText embeddings for English and Hindi, limit their vocabularies appropriately, and extract word translation pairs for aligning the embeddings into a shared bilingual space.
Additional Resources
- Gensim Documentation: https://radimrehurek.com/gensim/
- MUSE Repository: https://github.com/facebookresearch/MUSE
- FastText Documentation: https://fasttext.cc/docs/en/python-module.html
- WikiExtractor Tool: https://github.com/attardi/wikiextractor
Feel free to reach out if you need further assistance!
-