Extract TF-IDF keywords using MapReduce style logic. The map phase counts word…
- User
Extract TF-IDF keywords using MapReduce style logic. The map phase counts word frequencies per document, reduce aggregates global document frequencies. Calculate TF-IDF for a 10GB corpus without loading everything into RAM.
- Assistant
Explaining TF-IDF extraction with MapReduce
Here's a high-level MapReduce approach for extracting TF-IDF from a large corpus (e.g., 10GB) without loading everything into RAM:
- Map Phase (Term Frequency): For each document, emit (word@docID, 1) for each word occurrence; combine locally to sum term counts per document.
- Reduce Phase (Term Frequency): Sum counts to get term frequency (TF) per word per document.
- Second MapReduce Job (Document Frequency): Map emits (word, docID) for each unique word-document pair; reduce aggregates counts of documents containing the word to compute document frequency (DF).
- Final Computation: Join TF and DF counts with total document count (N) outside MapReduce or in another job, then compute TF-IDF = TF * log(N / DF) per word per document.
This multi-pass pipeline scales by streaming data through mappers and reducers, avoiding full corpus loading in RAM. Each stage processes chunks, suitable for large-scale data like 10GB.
Pseudo code for the first map phase (Python-style):
for line in sys.stdin: doc_id, text = parse(line) word_counts = Counter(tokenize(text)) for word, count in word_counts.items(): print(f"{word}@{doc_id}\t{count}")The reducer sums counts for each key to output TF. Next steps similarly aggregate DF and compute TF-IDF. This pattern fits classic MapReduce and streaming frameworks used for big data. Below is a common MapReduce-style pattern for computing TF–IDF on a large corpus (e.g., 10GB) without loading everything into RAM. You typically run three conceptual passes: one for term frequency (TF), one for document frequency (DF), and one to combine them into TF–IDF. The logic is the same whether you use Hadoop, Spark, or a streaming framework—you stream documents line by line and rely on map/reduce to aggregate.
Overall Data Flow
-
Job 1 (Compute TF per document)
- Map: Read a document, output
(word, docID, countInDoc). - Reduce: Sum counts per
(word, docID)→(word, docID, TF). - Optionally store document length or max TF for normalization.
- Map: Read a document, output
-
Job 2 (Compute DF per word)
- Map: Read
(word, docID, TF), emit(word, docID)so each doc counts once. - Reduce: Aggregate unique docIDs for each word to get
DF(word)(number of docs containing the word). - Also compute total number of documents
N.
- Map: Read
-
Job 3 (Compute TF–IDF)
- Map: Join TF data from Job 1 with DF from Job 2 (and total
N). - Reduce: For each
(word, docID), compute
[ \text{TF–IDF} = \text{TF}(word, docID) \times \log\frac{N}{\text{DF}(word)} ] - Output top keywords per document or keep all TF–IDF scores.
- Map: Join TF data from Job 1 with DF from Job 2 (and total
Example (Hadoop Streaming–Style Pseudocode)
Job 1: TF per document
Mapper (TFMapper.py):
import sys import re WORD_RE = re.compile(r"[A-Za-z0-9]+") for line in sys.stdin: # Expect a format like "docID<TAB>text" parts = line.strip().split("\t", 1) if len(parts) != 2: continue doc_id, text = parts counts = {} for w in WORD_RE.findall(text.lower()): counts[w] = counts.get(w, 0) + 1 # Emit (word, docID, countInDoc) for word, c in counts.items(): print(f"{word}\t{doc_id}\t{c}")Reducer (TFReducer.py):
import sys current_word = None current_doc = None current_sum = 0 for line in sys.stdin: word, doc_id, cnt = line.strip().split("\t") cnt = int(cnt) if (word != current_word) or (doc_id != current_doc): if current_word is not None: # emit TF print(f"{current_word}\t{current_doc}\t{current_sum}") current_word, current_doc, current_sum = word, doc_id, cnt else: current_sum += cnt # last key if current_word is not None: print(f"{current_word}\t{current_doc}\t{current_sum}")This job outputs lines of the form:
word docID TFJob 2: DF per word
Mapper (DFMapper.py):
import sys for line in sys.stdin: word, doc_id, tf = line.strip().split("\t") # emit once per (word, docID) print(f"{word}\t{doc_id}")Reducer (DFReducer.py):
import sys current_word = None docs = set() N = 0 # to count total docs (optional, you may track separately) for line in sys.stdin: word, doc_id = line.strip().split("\t") # You might want to track total documents by passing doc_id to a side channel # if not unique. Typically you'd count total docs in a separate step or known a priori. if word != current_word: if current_word is not None: # emit DF print(f"{current_word}\t{len(docs)}") current_word = word docs = set() docs.add(doc_id) if current_word is not None: print(f"{current_word}\t{len(docs)}")This job outputs lines of the form:
word DF(word)You also need
N(total number of documents). You can get that in various ways:- If each input line is one document, you can count lines during ingest.
- Or do a separate job to count distinct docIDs.
Job 3: Compute TF–IDF
You need to join the outputs of Job 1 (TF per document) with Job 2 (DF per word) plus the total doc count
N. You can do this with a distributed join or by loading DF andNin a distributed cache, then:Mapper (TFIDFMapper.py):
import sys import math # Suppose we loaded DF(word) and N into memory, from distributed cache # For large vocabularies, consider a key-based shuffle join instead. # Pseudocode: # df_dict = load_df() # from a side file, e.g., HDFS cached # N = load_total_doc_count() for line in sys.stdin: word, doc_id, tf = line.strip().split("\t") tf = float(tf) df = df_dict.get(word, 1) # avoid zero DF idf = math.log10(N / df) tfidf = tf * idf # output (docID, word, tfidf) for post-processing print(f"{doc_id}\t{word}\t{tfidf}")Reducer (TFIDFReducer.py) [optional]:
- You might just sort or pick top k words per document. For example:
import sys current_doc = None word_scores = [] def emit_top_k(doc_id, scores, k=10): # scores is list of (word, tfidf) scores.sort(key=lambda x: x[1], reverse=True) for w, sc in scores[:k]: print(f"{doc_id}\t{w}\t{sc}") for line in sys.stdin: doc_id, word, tfidf = line.strip().split("\t") tfidf = float(tfidf) if doc_id != current_doc: if current_doc is not None: emit_top_k(current_doc, word_scores) current_doc = doc_id word_scores = [] word_scores.append((word, tfidf)) if current_doc is not None: emit_top_k(current_doc, word_scores)
Memory Considerations for a 10GB Corpus
- Streaming: Each mapper handles chunks of input; no need to load the entire corpus in memory—just process line by line.
- Combiner: For TF, use a combiner (same code as reducer) to reduce network overhead by summing partial counts locally.
- Distributed Cache: Store DF data (and total doc count) in a distributed cache if it fits. Otherwise perform a shuffle-based join by key (word).
- Multiple Passes: You might do a small pre-pass to count total docs (
N) if not known from the input structure.
Summary
- Job 1: Output
(word, docID, TF). - Job 2: Calculate
DF(word)plus total doc countN. - Job 3: Join TF with DF and compute
TF–IDF = TF * log(N/DF), then optionally pick top keywords.
This approach keeps memory usage low, relying on streaming and MapReduce shuffles rather than loading the entire 10GB corpus at once.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.