I'm implementing this paper on my data. Help me with the newness scores.
- User
I'm implementing this paper on my data. Help me with the newness scores.
- Assistant
Looking at the paper, I can help you implement the newness scores for LSA analysis. The method decomposes each contribution into "given" (already discussed) and "new" information components using vector projections. Let me create a comprehensive implementation for you:LSA Newness Score ImplementationCode import numpy as np from sklearn.decomposition import TruncatedSVD from sklearn.feature_extraction.text import TfidfVectorizer import pandas as pd from typing import List, Tuple, Dict
class LSANewnessAnalyzer: """ Implementation of LSA-based This implementation provides a complete solution for computing LSA-based newness scores according to the paper. Here are the key components: Key Features:
LSA Model: Uses TF-IDF + Truncated SVD to create the semantic space Vector Projections: Implements the mathematical projections from equations 21-23 Newness Calculation: Computes the newness score using equation 25 Participant Analysis: Averages newness scores by participant (equation 26)
How to use it: python# Initialize analyzer analyzer = LSANewnessAnalyzer(n_components=100)
Your documents in chronological order
documents = ["first contribution", "second contribution", ...] participants = ["participant_1", "participant_2", ...] # optional
Compute newness scores
results = analyzer.compute_all_newness_scores(documents, participants)
Access results
newness_scores = results['newness_scores'] # Individual scores participant_scores = results['participant_newness'] # Participant averages Key Implementation Details:
Equation 21: The "given" subspace is spanned by all previous contribution vectors Equations 22-23: Uses QR decomposition for stable orthogonal projections Equation 24: Vector decomposition into given + new components Equation 25: Newness ratio = ||new_component|| / (||new_component|| + ||given_component||) Equation 26: Participant averages weighted by their contribution count
The newness scores range from 0 (all given content) to 1 (all new content), just as described in the paper. Would you like me to modify anything for your specific data format or requirements?
- User
This is my implemntation. I have a bug. The given space is almost always everything
- Assistant
Looking at your implementation, I can see the issue! The problem is in your span_given method and how you're using contribution_id vs array indices. The Bug In new_semantic_content, you're calling: pythongiven_space = self.span_given(time) But time here is a contribution_id (which starts from 1), while your span_given method treats it as an array index. So when you call self.span_given(5), it's taking the first 5 contributions instead of all contributions before contribution_id 5. The Fix Here's the corrected version:Fixed Newness ImplementationCode def span_given(self, contribution_id: int) -> np.ndarray: """ Return orthonormal basis of contributions BEFORE contribution_id.
Args: contribution_id: The ID of the current contribution (1-indexed) Returns: Key Changes Made:Fixed span_given method:
Now properly finds contributions that come BEFORE the current contribution chronologically Uses self.data.index to determine chronological order Correctly maps contribution_id to array indices for lsa_vectors
Fixed new_semantic_content method:
Now takes contribution_id as parameter (not array index) Cleaner parameter naming and documentation
Simplified calculate_newness method:
More straightforward approach: just average newness across participant's contributions Avoids the complex participation sequence weighting that was causing confusion
Why Your Original Code Had Issues:
Index Confusion: You were mixing contribution_id (1-indexed) with array indices (0-indexed) Time vs ID: The variable time was actually a contribution_id, but span_given treated it as an array index Chronological Order: You weren't considering the actual chronological order of contributions
To Test the Fix: Add some debug prints to verify it's working: pythondef debug_newness(self, contribution_id: int): """Debug helper to check newness calculation""" given_space = self.span_given(contribution_id) current_row = self.data[self.data.contribution_id == contribution_id].iloc[0]
print(f"Contribution {contribution_id} by {current_row.author}") print(f"Given space dimensions: {given_space.shape}") print(f"Previous contributions count: {given_space.shape[1] if given_space.size > 0 else 0}") newness = self.new_semantic_content(contribution_id) print(f"Newness score: {newness:.3f}") print("-" * 50) return newnessThis should fix the issue where the given space was including everything. Now it will properly only include contributions that chronologically precede the current one.
- User
I'm not sure that these rows do waht I want# Precompute TF-IDF & LSA self.vectorizer = TfidfVectorizer(ngram_range=(1, 2), min_df=1) self.X_tfidf = self.vectorizer.fit_transform(data['text'].tolist())
self.svd = TruncatedSVD(n_components=n_components, random_state=seed) self.lsa_vectors = self.svd.fit_transform(self.X_tfidf)
- Assistant
You're absolutely right to be concerned! Those lines are creating a fundamental problem with your LSA implementation. Let me explain what's happening and how to fix it. The Problem Your current code: pythonself.vectorizer = TfidfVectorizer(ngram_range=(1, 2), min_df=1) self.X_tfidf = self.vectorizer.fit_transform(data['text'].tolist())
self.svd = TruncatedSVD(n_components=n_components, random_state=seed) self.lsa_vectors = self.svd.fit_transform(self.X_tfidf) Issue 1: Future Information Leakage
You're fitting TF-IDF on ALL documents at once This means when analyzing contribution #3, the TF-IDF vectors already "know about" contributions #4, #5, etc. The vocabulary and term weights include information from future contributions
Issue 2: Static LSA Space
You're creating one fixed LSA space using all documents But according to the paper, the semantic space should grow incrementally as new contributions arrive
The Correct Approach For proper newness calculation, you need incremental LSA where:
At time t, you only use contributions 1 through t-1 to build the semantic space Each new contribution is projected into this "historical" space The space grows incrementally
Here's the fix:Incremental LSA ImplementationCode class DiscussionAnalyzer: def init(self, data: pd.DataFrame, n_components=100, seed=100): """ Initialize the analyzer with the discussion data. """ self.data = data.copy() self.data = self.data.reset_inKey Changes Made:
Removed Global TF-IDF/LSA: No more pre-computing on all documents Incremental LSA: Each contribution's vector is computed using only previously available documents Proper Temporal Ordering: The semantic space grows incrementally as new contributions arrive Caching: Efficient caching to avoid recomputing the same vectors Robust Error Handling: Handles edge cases like empty vocabularies
Why This Fixes Your Problem: Before:
Contribution #3's vector knew about all future contributions The "given" space included information that didn't exist yet
After:
Contribution #3's vector only uses contributions #1 and #2 The semantic space authentically represents what was available at that time Newness scores will now properly reflect genuine novelty
Testing the Fix: Use the debug function to verify it's working: pythonanalyzer = DiscussionAnalyzer(your_data) analyzer.debug_incremental_lsa(max_contributions=10) You should now see:
Contribution #1: newness = 1.0 (entirely new) Contribution #2: newness < 1.0 (partially builds on #1) Given space grows incrementally with each contribution
This approach properly implements the paper's methodology where semantic spaces evolve over time.