Can you explain this implementation? def crploglikelihood(alpha, counts): """ CRP…
- User
Can you explain this implementation? def crp_log_likelihood(alpha, counts): """ CRP marginal log-likelihood. counts: list of topic counts [n1, n2, ..., nK] """ if alpha <= 0: return -np.inf T = sum(counts) K = len(counts) return ( K * np.log(alpha) + np.sum(gammaln(np.array(counts))) + gammaln(alpha) - gammaln(T + alpha) ) def estimate_alpha(counts, alpha_max=100.0): """ MLE estimate of CRP alpha given topic counts. """ result = minimize_scalar( lambda a: -crp_log_likelihood(a, counts), bounds=(1e-6, alpha_max), method="bounded" ) return result.x
- Assistant
This code is a more formal and "complete" implementation of the Chinese Restaurant Process math. It calculates the Exchangeable Partition Probability Function (EPPF).While the previous method I gave you solved for α by setting the derivative to zero (finding the peak directly), this code defines the entire "mountain" (the likelihood curve) and uses a generic search algorithm to find the top.Here is the breakdown of the implementation:1. crp_log_likelihood(alpha, counts)This function calculates the log-probability of seeing a specific seating arrangement (partition) given an α.The math formula for the probability of a partition B with cluster sizes n1,n2,...nK is: P(B)=Γ(N+α)αK⋅Γ(α)i=1∏K(ni−1)!The code implements the logarithm of this equation to avoid numerical underflow (probabilities get tiny, logs stay manageable).K * np.log(alpha): Corresponds to αK (the power becomes multiplication in log-space). This rewards having more tables if α is high.gammaln(alpha) - gammaln(T + alpha): Corresponds to the normalizing term Γ(N+α)Γ(α). This accounts for the "rich get richer" denominators accumulating as customers enter.Note: T is the total number of customers (N).np.sum(gammaln(np.array(counts))): Corresponds to ∏(ni−1)!.Math Note: In Python, gammaln(x) computes ln((x−1)!). So gammaln(counts) correctly computes the factorials of the group sizes. This part measures how "clumped" the data is.2. estimate_alpha(counts, alpha_max=100.0)This function finds the best α.lambda a: -crp_log_likelihood(a, counts): Optimization algorithms usually "minimize" things (search for the bottom of a valley). Since we want to maximize likelihood (find the peak), we feed the optimizer the negative likelihood. Minimizing the negative is the same as maximizing the positive.minimize_scalar: This is a standard SciPy function that tries different values of a (between 0 and 100) until it finds the one that results in the lowest output (highest likelihood).Comparison to the Previous MethodMathematically, this produces the exact same result as the solve_alpha code I wrote for you earlier.Your previous code: Used the derivative equation αK=∑α+i1. This is faster because it goes straight to the solution.This code: Calculates the full probability score. This is slower (requires an iterative search), but it is useful if you want to compare models. For example, if you wanted to ask "Does Model A with Alpha=2 explain the data better than Model B with Alpha=5?", you would need this function to get the actual scores.Summary: Use this version if you need the actual log-likelihood values for model comparison. Use the previous version if you just need the value of α quickly.
- User
I want two separate implementations: 1. at the conversation level. 2. at the prefix level
- Assistant
Pythonimport pandas as pd import numpy as np from scipy.special import digamma from scipy.optimize import brentq
Define the solver function (MLE for Alpha)
def solve_alpha_mle(N, K): """ Solves for alpha using the derivative of the CRP log-likelihood. Equation: K/alpha + digamma(alpha) - digamma(N + alpha) = 0 This depends ONLY on N (total items) and K (total categories), not individual counts. """ if N <= 0 or K <= 0: return np.nan if K == 1: return 1e-4 # Close to 0 if K >= N: return 1000.0 # Arbitrary large number for max diversity
# Objective function: derivative of log-likelihood def f(alpha): return (K / alpha) + digamma(alpha) - digamma(N + alpha) # Bracket search for root try: # Check signs at bounds low, high = 1e-5, 1000.0 f_low = f(low) f_high = f(high) if f_low * f_high > 0: # If both have same sign, return the bound that is closer to 0 if abs(f_low) < abs(f_high): return low else: return high return brentq(f, low, high) except Exception: return np.nanLoad data
df = pd.read_csv('topic_novelty.csv')
--- 1. Conversation Level Implementation ---
conversation_results = [] for filename, group in df.groupby('filename'): N = len(group) K = group['topic'].nunique() alpha = solve_alpha_mle(N, K) conversation_results.append({ 'filename': filename, 'total_turns': N, 'unique_topics': K, 'alpha': alpha })
df_conversation = pd.DataFrame(conversation_results) df_conversation.to_csv('conversation_level_alpha.csv', index=False)
--- 2. Prefix Level Implementation ---
prefix_results = [] for filename, group in df.groupby('filename'): # Ensure temporal order group = group.sort_values('turn_number') # Assuming turn_number sorts correctly, otherwise index
seen_topics = set() for i in range(len(group)): row = group.iloc[i] topic = row['topic'] seen_topics.add(topic) N_prefix = i + 1 K_prefix = len(seen_topics) alpha_prefix = solve_alpha_mle(N_prefix, K_prefix) prefix_results.append({ 'filename': filename, 'turn_number': row['turn_number'], 'step': N_prefix, 'current_topic': topic, 'unique_topics_so_far': K_prefix, 'alpha': alpha_prefix })df_prefix = pd.DataFrame(prefix_results) df_prefix.to_csv('prefix_level_alpha.csv', index=False)
print("Analysis Complete.") print("Conversation Level Head:") print(df_conversation.head()) print("\nPrefix Level Head:") print(df_prefix.head())
Code outputAnalysis Complete. Conversation Level Head: filename total_turns unique_topics alpha 0 affordable_care_act_in_the_uni_Against_24turns_2ppl.txt 42 30 45.529284 1 affordable_care_act_in_the_uni_Against_24turns_5ppl.txt 48 28 27.247006 2 affordable_care_act_in_the_uni_Against_8turns_2ppl.txt 19 14 22.462794 3 affordable_care_act_in_the_uni_Against_8turns_5ppl.txt 26 22 64.712648 4 affordable_care_act_in_the_uni_For_24turns_2ppl.txt 46 26 23.945643
Prefix Level Head: filename turn_number step current_topic unique_topics_so_far alpha 0 affordable_care_act_in_the_uni_Against_24turns_2ppl.txt 1 1 ACA Affordability and Quality 1 0.000100 1 affordable_care_act_in_the_uni_Against_24turns_2ppl.txt 1 2 Insurance Coverage Expansion 2 1000.000000 2 affordable_care_act_in_the_uni_Against_24turns_2ppl.txt 2 3 ACA Affordability and Quality 2 1.414214 3 affordable_care_act_in_the_uni_Against_24turns_2ppl.txt 2 4 Small Business Impact 3 3.766435 4 affordable_care_act_in_the_uni_Against_24turns_2ppl.txt 2 5 Premium Increases 4 7.106196
Here are the two implementations you requested, applied to your dataset.1. Conversation Level ImplementationThis approach treats the entire discussion as a single "finished meal" and calculates one static α value that represents the overall creativity of that group.Use Case: Comparing different groups to see which one was more diverse overall.Pythonimport pandas as pd import numpy as np from scipy.special import digamma from scipy.optimize import brentq
def solve_alpha(N, K): """ Finds alpha such that the derivative of Log-Likelihood is zero. Equation: K/alpha + psi(alpha) - psi(N + alpha) = 0 """ if K == 1: return 0.001 # Total conformity if K >= N: return 100.0 # Total novelty
# Derivative function def f(alpha): return (K / alpha) + digamma(alpha) - digamma(N + alpha) try: # Find root between extremely low and high alpha return brentq(f, 1e-5, 1000.0) except: return np.nanLoad Data
df = pd.read_csv('topic_novelty.csv')
--- Implementation ---
conversation_data = []
for filename, group in df.groupby('filename'): N = len(group) # Total turns in conversation K = group['topic'].nunique() # Total unique topics found
alpha = solve_alpha(N, K) conversation_data.append({ 'filename': filename, 'total_turns': N, 'unique_topics': K, 'overall_alpha': alpha })df_conversation = pd.DataFrame(conversation_data)
df_conversation.to_csv('conversation_alphas.csv')
print(df_conversation.head()) 2. Prefix Level ImplementationThis approach recalculates α at every single step of the conversation. It answers: "Given what we have seen so far, what is the current creativity rate?"Use Case: Tracking the flow of a conversation. Does it start creative and then die down? Does it get stuck in a loop and then suddenly break out?Python# --- Implementation --- prefix_data = []
for filename, group in df.groupby('filename'): # Sort chronologically group = group.sort_values('turn_number')
seen_topics = set() for i in range(len(group)): # Get data for the "prefix" (rows 0 to i) current_topic = group.iloc[i]['topic'] seen_topics.add(current_topic) N_so_far = i + 1 K_so_far = len(seen_topics) # Calculate Alpha for this specific moment in time current_alpha = solve_alpha(N_so_far, K_so_far) prefix_data.append({ 'filename': filename, 'turn_number': group.iloc[i]['turn_number'], 'step': N_so_far, 'alpha_at_this_step': current_alpha })df_prefix = pd.DataFrame(prefix_data)
df_prefix.to_csv('prefix_alphas.csv')
print(df_prefix.head()) Note on Mathematical Equivalence: I used the solve_alpha (derivative) method here because it is computationally cleaner for the "Prefix" implementation where we have to run the solver thousands of times (once for every row). It produces the exact same Maximum Likelihood Estimate as the minimize_scalar method you asked about previously, but avoids the overhead of defining the full Gamma function log-likelihood every time.
- User
IS this exoected to run well: """ import numpy as np import pandas as pd from scipy.special import gammaln, digamma from scipy.optimize import minimize_scalar, brentq import matplotlib.pyplot as plt import seaborn as sns sns.set(style="whitegrid") # -------------------------------------------------- # CRP log-likelihood and alpha estimation # -------------------------------------------------- # -------------------------------------------------- # Prefix-based alpha estimation # -------------------------------------------------- # def estimate_prefix_alphas(topics, min_len=2): # """ # Estimate alpha for every prefix of a topic sequence. # Returns list of dicts with prefix length and alpha. # """ # seen = {} # results = [] # # for t, topic in enumerate(topics, start=1): # seen[topic] = seen.get(topic, 0) + 1 # # if t >= min_len: # alpha_t = estimate_alpha(list(seen.values())) # results.append({ # "prefix_length": t, # "alpha": alpha_t, # "num_topics": len(seen) # }) # # return results # -------------------------------------------------- # Main analysis # -------------------------------------------------- def get_number_turns(conv_id): """ filenames are in the format: "f{subject}{stance}{num_turns}turns_{ppl}ppl.txt" :param conv_id: :return: """ return conv_id.split("")[-2].strip("turns") def get_number_participants(conv_id): return conv_id.split("")[-1].strip("ppl.txt") def get_stance(conv_id): """ filenames are in the format: "f{subject}{stance}{num_turns}turns_{ppl}ppl.txt" :param conv_id: :return: """ return conv_id.split("_")[-3] def solve_alpha_mle(N, K): """ Estimates alpha using Maximum Likelihood Estimation. Equation: K/alpha - (digamma(alpha + N) - digamma(alpha)) = 0 """ # Edge Case 1: No data if N <= 0 or K <= 0: return np.nan # Edge Case 2: Only 1 topic (Extreme Conformity) # The equation has no solution for alpha > 0, implies alpha -> 0 if K == 1: return 0.001 # Edge Case 3: All topics are unique (Extreme Novelty) # The equation implies alpha -> infinity if K == N: return 100.0 # The Objective Function we want to be zero def objective(alpha): return (K / alpha) - (digamma(alpha + N) - digamma(alpha)) # We use Brent's method (brentq) to find the root # We search for alpha in the range [0.00001, 1000] try: # Check bounds first to ensure a sign change exists low, high = 1e-5, 1000.0 if objective(low) * objective(high) > 0: return low if objective(low) < 0 else high return brentq(objective, low, high) except Exception: return np.nan def prefix_level_analysis(df): """ :return: """ prefix_data = [] for filename, group in df.groupby('filename'): # Sort chronologically group = group.sort_values('turn_number') seen_topics = set() for i in range(len(group)): # Get data for the "prefix" (rows 0 to i) current_topic = group.iloc[i]['topic'] seen_topics.add(current_topic) total_mentions = i + 1 novelty_so_far = len(seen_topics) # Calculate Alpha for this specific moment in time current_alpha = solve_alpha_mle(total_mentions, novelty_so_far) prefix_data.append({ 'filename': filename, 'turn_number': group.iloc[i]['turn_number'], 'step': total_mentions, 'alpha_at_step': current_alpha, "novelty_so_far": novelty_so_far, "num_ppl": get_number_participants(filename), "stance": get_stance(filename) }) df_prefix = pd.DataFrame(prefix_data) df_prefix.to_csv('prefix_level_alphas.csv') def main(): # Load Data df = pd.read_csv('topic_novelty.csv') # --- Implementation --- conversation_data = [] for filename, group in df.groupby('filename'): total_mentions = len(group) # Total turns in conversation unique_topics = group["novelty"].sum() # Total unique topics found estimated_alpha = solve_alpha_mle(total_mentions, unique_topics) conversation_data.append({ "filename": filename, "alpha": estimated_alpha, "num_mentions": total_mentions, "num_novel_topics": group["novelty"].sum(), "num_turns": get_number_turns(filename), "num_ppl": get_number_participants(filename), "stance": get_stance(filename) }) # Convert to DataFrames conversation_df = pd.DataFrame(conversation_data) # Save outputs conversation_df.to_csv("conversation_level_alpha.csv", index=False) prefix_level_analysis(df) print("Analysis complete.") print("Saved:") print(" - conversation_level_alpha.csv") print(" - prefix_level_alpha.csv") if name == "main": main() """
- Assistant
Pythonimport numpy as np import pandas as pd from scipy.special import gammaln, digamma from scipy.optimize import brentq import re
Load the file
df = pd.read_csv('topic_novelty.csv')
def get_number_turns(conv_id): # filenames are in the format: "f{subject}{stance}{num_turns}turns_{ppl}ppl.txt" try: return conv_id.split("_")[-2].strip("turns") except: return np.nan
def get_number_participants(conv_id): try: return conv_id.split("_")[-1].strip("ppl.txt") except: return np.nan
def get_stance(conv_id): try: return conv_id.split("_")[-3] except: return np.nan
def solve_alpha_mle(N, K): """ Estimates alpha using Maximum Likelihood Estimation. Equation: K/alpha - (digamma(alpha + N) - digamma(alpha)) = 0 """ if N <= 0 or K <= 0: return np.nan if K == 1: return 0.001 if K == N: return 100.0
def objective(alpha): return (K / alpha) - (digamma(alpha + N) - digamma(alpha)) try: low, high = 1e-5, 1000.0 if objective(low) * objective(high) > 0: return low if objective(low) < 0 else high return brentq(objective, low, high) except Exception: return np.nandef prefix_level_analysis(df): prefix_data = []
# Check if novelty matches counts # This is just for my verification, not part of user code execution per se, # but useful for answering "will this run well" for filename, group in df.groupby('filename'): group = group.sort_values('turn_number') seen_topics = set() for i in range(len(group)): current_topic = group.iloc[i]['topic'] seen_topics.add(current_topic) total_mentions = i + 1 novelty_so_far = len(seen_topics) current_alpha = solve_alpha_mle(total_mentions, novelty_so_far) prefix_data.append({ 'filename': filename, 'turn_number': group.iloc[i]['turn_number'], 'step': total_mentions, 'alpha_at_step': current_alpha, "novelty_so_far": novelty_so_far, "num_ppl": get_number_participants(filename), "stance": get_stance(filename) }) df_prefix = pd.DataFrame(prefix_data) # User code saves to prefix_level_alphas.csv # I will not actually save to disk unless necessary, but the logic holds. return df_prefixMain logic simulation
conversation_data = []
for filename, group in df.groupby('filename'): total_mentions = len(group) # User code used sum of novelty unique_topics_sum = group["novelty"].sum() # Safe check: does it match nunique? unique_topics_count = group['topic'].nunique()
estimated_alpha = solve_alpha_mle(total_mentions, unique_topics_sum) conversation_data.append({ "filename": filename, "alpha": estimated_alpha, "num_mentions": total_mentions, "num_novel_topics": unique_topics_sum, "real_unique_topics": unique_topics_count, "num_turns": get_number_turns(filename), "num_ppl": get_number_participants(filename), "stance": get_stance(filename) })conversation_df = pd.DataFrame(conversation_data)
Check for discrepancies
discrepancies = conversation_df[conversation_df['num_novel_topics'] != conversation_df['real_unique_topics']] print("Discrepancies found:", len(discrepancies)) if len(discrepancies) > 0: print(discrepancies.head())
print("Execution successful.") print(conversation_df.head())
Code outputDiscrepancies found: 4 filename alpha num_mentions num_novel_topics real_unique_topics num_turns num_ppl stance 5 affordable_care_act_in_the_uni_For_24turns_5ppl.txt 36.001832 61 36 37 24 5 For 20 imposing_taxes_on_food_product_For_24turns_2ppl.txt 22.410442 48 26 34 24 2 For 21 imposing_taxes_on_food_product_For_24turns_5ppl.txt 13.939888 43 20 21 24 5 For 23 imposing_taxes_on_food_product_For_8turns_5ppl.txt 11.613011 26 14 21 8 5 For Execution successful. filename alpha num_mentions num_novel_topics real_unique_topics num_turns num_ppl stance 0 affordable_care_act_in_the_uni_Against_24turns_2ppl.txt 45.529284 42 30 30 24 2 Against 1 affordable_care_act_in_the_uni_Against_24turns_5ppl.txt 27.247006 48 28 28 24 5 Against 2 affordable_care_act_in_the_uni_Against_8turns_2ppl.txt 22.462794 19 14 14 8 2 Against 3 affordable_care_act_in_the_uni_Against_8turns_5ppl.txt 64.712648 26 22 22 8 5 Against 4 affordable_care_act_in_the_uni_For_24turns_2ppl.txt 23.945643 46 26 26 24 2 For
The code you provided is structurally sound and will run, but there is a critical logical inconsistency between your main function and your prefix_level_analysis function that will cause your two output files to disagree for some discussions.The Major Issue: novelty column vs. Calculated NoveltyIn prefix_level_analysis: You calculate novelty dynamically by counting unique topic strings seen so far (len(seen_topics)). This is the standard CRP definition.In main: You calculate novelty by summing the pre-existing boolean column (group["novelty"].sum()).I ran a check on your data, and for 4 of your files, these two numbers do not match. (e.g., In one file, the novelty column sums to 26, but there are actually 34 unique topic strings). This means your conversation_level_alpha.csv will report a different "truth" than your prefix_level_alphas.csv.Recommendation: Modify the main function to calculate unique topics the same way the prefix function does (using .nunique()) to ensure consistency.Minor IssuesFilename Mismatch: You save the prefix file as prefix_level_alphas.csv (plural) but print that you saved prefix_level_alpha.csv (singular).Fragile String Parsing: strip("ppl.txt") works for your current filenames (e.g., "5ppl.txt" becomes "5"), but technically strip removes any character in that set from the ends, not the substring. replace(".txt", "").replace("ppl", "") is safer, though your current method works for digits.Corrected Code SnippetHere is the corrected main function loop to ensure your results are consistent:Pythondef main(): # Load Data df = pd.read_csv('topic_novelty.csv')
# --- Implementation --- conversation_data = [] for filename, group in df.groupby('filename'): total_mentions = len(group) # CORRECTED: Use nunique() instead of novelty.sum() # to match the logic in prefix_level_analysis unique_topics = group['topic'].nunique() estimated_alpha = solve_alpha_mle(total_mentions, unique_topics) conversation_data.append({ "filename": filename, "alpha": estimated_alpha, "num_mentions": total_mentions, "num_novel_topics": unique_topics, # Updated "num_turns": get_number_turns(filename), "num_ppl": get_number_participants(filename), "stance": get_stance(filename) }) # Convert to DataFrames conversation_df = pd.DataFrame(conversation_data) # Save outputs conversation_df.to_csv("conversation_level_alpha.csv", index=False) prefix_level_analysis(df) # ... rest of your print statements - User
"group["novelty"].sum()" is the correct way to calculate the number of distinct topics. since novelty is True iff a new topic is introduced. This can compensate for typos.
- Assistant
Understood. If the novelty column is the ground truth (manually annotated to account for typos or specific context), then your main() function is correct, but your prefix_level_analysis function is incorrect.Currently, your prefix_level_analysis ignores the novelty column entirely and re-calculates uniqueness based on the strings (seen_topics.add(current_topic)). This defeats the purpose of your manual annotation.You need to update prefix_level_analysis to use the cumulative sum of the novelty column.Here is the corrected and consistent implementation:Pythonimport numpy as np import pandas as pd from scipy.special import gammaln, digamma from scipy.optimize import brentq
... [Keep your existing imports and helper functions like solve_alpha_mle] ...
def prefix_level_analysis(df): """ Estimates alpha at each step using the cumulative sum of the 'novelty' column. """ prefix_data = []
for filename, group in df.groupby('filename'): # Ensure chronological order group = group.sort_values('turn_number') # CRITICAL FIX: Use the cumulative sum of the 'novelty' column # This honors your manual annotation (True = new topic) rather than # relying on raw string uniqueness. group['novelty_count_so_far'] = group['novelty'].cumsum() for i in range(len(group)): row = group.iloc[i] total_mentions = i + 1 # Use the pre-calculated cumulative sum novelty_so_far = row['novelty_count_so_far'] current_alpha = solve_alpha_mle(total_mentions, novelty_so_far) prefix_data.append({ 'filename': filename, 'turn_number': row['turn_number'], 'step': total_mentions, 'alpha_at_step': current_alpha, "novelty_so_far": novelty_so_far, "num_ppl": get_number_participants(filename), "stance": get_stance(filename) }) df_prefix = pd.DataFrame(prefix_data) df_prefix.to_csv('prefix_level_alphas.csv', index=False)def main(): # Load Data df = pd.read_csv('topic_novelty.csv')
# Ensure novelty is treated as numeric (boolean True becomes 1) for summation df['novelty'] = df['novelty'].astype(int) # --- Implementation --- conversation_data = [] for filename, group in df.groupby('filename'): total_mentions = len(group) # Consistent: We use the sum of the novelty column here too total_unique_topics = group["novelty"].sum() estimated_alpha = solve_alpha_mle(total_mentions, total_unique_topics) conversation_data.append({ "filename": filename, "alpha": estimated_alpha, "num_mentions": total_mentions, "num_novel_topics": total_unique_topics, "num_turns": get_number_turns(filename), "num_ppl": get_number_participants(filename), "stance": get_stance(filename) }) # Convert to DataFrames conversation_df = pd.DataFrame(conversation_data) # Save outputs conversation_df.to_csv("conversation_level_alpha.csv", index=False) # Run the corrected prefix analysis prefix_level_analysis(df) print("Analysis complete.")if name == "main": main() Why this is betterConsistency: Both main and prefix_level_analysis now derive K (number of tables) from the exact same source: your novelty boolean column.Efficiency: .cumsum() is vectorized and much faster than iterating through rows and updating a python set() in the prefix loop.Accuracy: It respects your manual cleaning. If "Education" and "education" are marked as the same topic in your novelty column (False), this code will correctly treat them as the same, whereas set() would have treated them as different.
- User
Is this better? Why not iterrows? import numpy as np import pandas as pd from scipy.special import gammaln, digamma from scipy.optimize import minimize_scalar, brentq import matplotlib.pyplot as plt import seaborn as sns sns.set(style="whitegrid") # -------------------------------------------------- # CRP log-likelihood and alpha estimation # -------------------------------------------------- # -------------------------------------------------- # Prefix-based alpha estimation # -------------------------------------------------- # def estimate_prefix_alphas(topics, min_len=2): # """ # Estimate alpha for every prefix of a topic sequence. # Returns list of dicts with prefix length and alpha. # """ # seen = {} # results = [] # # for t, topic in enumerate(topics, start=1): # seen[topic] = seen.get(topic, 0) + 1 # # if t >= min_len: # alpha_t = estimate_alpha(list(seen.values())) # results.append({ # "prefix_length": t, # "alpha": alpha_t, # "num_topics": len(seen) # }) # # return results # -------------------------------------------------- # Main analysis # -------------------------------------------------- def get_number_turns(conv_id): """ filenames are in the format: "f{subject}{stance}{num_turns}turns_{ppl}ppl.txt" :param conv_id: :return: """ return conv_id.split("")[-2].strip("turns") def get_number_participants(conv_id): return conv_id.split("")[-1].strip("ppl.txt") def get_stance(conv_id): """ filenames are in the format: "f{subject}{stance}{num_turns}turns_{ppl}ppl.txt" :param conv_id: :return: """ return conv_id.split("_")[-3] def solve_alpha_mle(N, K): """ Estimates alpha using Maximum Likelihood Estimation. Equation: K/alpha - (digamma(alpha + N) - digamma(alpha)) = 0 """ # Edge Case 1: No data if N <= 0 or K <= 0: return np.nan # Edge Case 2: Only 1 topic (Extreme Conformity) # The equation has no solution for alpha > 0, implies alpha -> 0 if K == 1: return 0.001 # Edge Case 3: All topics are unique (Extreme Novelty) # The equation implies alpha -> infinity if K == N: return 100.0 # The Objective Function we want to be zero def objective(alpha): return (K / alpha) - (digamma(alpha + N) - digamma(alpha)) # We use Brent's method (brentq) to find the root # We search for alpha in the range [0.00001, 1000] try: # Check bounds first to ensure a sign change exists low, high = 1e-5, 1000.0 if objective(low) * objective(high) > 0: return low if objective(low) < 0 else high return brentq(objective, low, high) except Exception: return np.nan def prefix_level_analysis(df): """ :return: """ prefix_data = [] for filename, group in df.groupby('filename'): # Sort chronologically group = group.sort_values('turn_number') group['novelty_count_so_far'] = group['novelty'].cumsum() for i in range(len(group)): row = group.iloc[i] # Get data for the "prefix" (rows 0 to i) current_topic = group.iloc[i]['topic'] total_mentions = i + 1 novelty_so_far = row['novelty_count_so_far'] # Calculate Alpha for this specific moment in time current_alpha = solve_alpha_mle(total_mentions, novelty_so_far) prefix_data.append({ 'filename': filename, 'turn_number': group.iloc[i]['turn_number'], 'step': total_mentions, 'alpha_at_step': current_alpha, "novelty_so_far": novelty_so_far, "num_ppl": get_number_participants(filename), "stance": get_stance(filename) }) df_prefix = pd.DataFrame(prefix_data) df_prefix.to_csv('prefix_level_alphas.csv') def main(): # Load Data df = pd.read_csv('topic_novelty.csv') # --- Implementation --- conversation_data = [] for filename, group in df.groupby('filename'): total_mentions = len(group) # Total turns in conversation unique_topics = group["novelty"].sum() # Total unique topics found estimated_alpha = solve_alpha_mle(total_mentions, unique_topics) conversation_data.append({ "filename": filename, "alpha": estimated_alpha, "num_mentions": total_mentions, "num_novel_topics": group["novelty"].sum(), "num_turns": get_number_turns(filename), "num_ppl": get_number_participants(filename), "stance": get_stance(filename) }) # Convert to DataFrames conversation_df = pd.DataFrame(conversation_data) # Save outputs conversation_df.to_csv("conversation_level_alpha.csv", index=False) prefix_level_analysis(df) print("Analysis complete.") print("Saved:") print(" - conversation_level_alpha.csv") print(" - prefix_level_alpha.csv") if name == "main": main()
- Assistant
Pythonimport numpy as np import pandas as pd from scipy.special import gammaln, digamma from scipy.optimize import brentq
Load the file
df = pd.read_csv('topic_novelty.csv')
def get_number_turns(conv_id): try: return conv_id.split("_")[-2].strip("turns") except: return np.nan
def get_number_participants(conv_id): try: return conv_id.split("_")[-1].strip("ppl.txt") except: return np.nan
def get_stance(conv_id): try: return conv_id.split("_")[-3] except: return np.nan
def solve_alpha_mle(N, K): if N <= 0 or K <= 0: return np.nan if K == 1: return 0.001 if K == N: return 100.0
def objective(alpha): return (K / alpha) - (digamma(alpha + N) - digamma(alpha)) try: low, high = 1e-5, 1000.0 if objective(low) * objective(high) > 0: return low if objective(low) < 0 else high return brentq(objective, low, high) except Exception: return np.nandef prefix_level_analysis(df): prefix_data = []
# Ensure novelty is int for cumsum df['novelty'] = df['novelty'].astype(int) for filename, group in df.groupby('filename'): # Sort chronologically group = group.sort_values('turn_number') # Vectorized cumsum - efficient group['novelty_count_so_far'] = group['novelty'].cumsum() # The loop in question for i in range(len(group)): # Access by position row = group.iloc[i] total_mentions = i + 1 novelty_so_far = row['novelty_count_so_far'] current_alpha = solve_alpha_mle(total_mentions, novelty_so_far) prefix_data.append({ 'filename': filename, 'turn_number': row['turn_number'], 'step': total_mentions, 'alpha_at_step': current_alpha, "novelty_so_far": novelty_so_far, "num_ppl": get_number_participants(filename), "stance": get_stance(filename) }) df_prefix = pd.DataFrame(prefix_data) return df_prefixTest run
try: result_df = prefix_level_analysis(df) print("Execution successful.") print(result_df.head()) except Exception as e: print(f"Error: {e}")
Code outputExecution successful. filename turn_number step alpha_at_step novelty_so_far num_ppl stance 0 affordable_care_act_in_the_uni_Against_24turns_2ppl.txt 1 1 0.001000 1 2 Against 1 affordable_care_act_in_the_uni_Against_24turns_2ppl.txt 1 2 100.000000 2 2 Against 2 affordable_care_act_in_the_uni_Against_24turns_2ppl.txt 2 3 1.414214 2 2 Against 3 affordable_care_act_in_the_uni_Against_24turns_2ppl.txt 2 4 3.766435 3 2 Against 4 affordable_care_act_in_the_uni_Against_24turns_2ppl.txt 2 5 7.106196 4 2 Against
Is this better?Yes, much better. This version fixes the logic error by using the cumulative sum of your manual annotations (group['novelty'].cumsum()), ensuring that the Alpha calculation respects your ground truth data.Why not iterrows()?You are right to avoid iterrows(). In the Pandas community, iterrows() is famously slow and is often considered an "anti-pattern" for two reasons:Overhead: It converts every single row into a Pandas Series object. Creating these objects thousands of times takes a lot of memory and time.Dtype Loss: When a row is converted to a Series, Pandas tries to force all data into a single type. If you have integers and floats, they might all become floats. If you have strings and ints, they might become objects. This can cause subtle bugs.The Hierarchy of Iteration Speed:Vectorization (Fastest): Applying the operation to the whole column at once (e.g., df['a'] + df['b']).itertuples() (Fast): Returns a lightweight "named tuple" for each row. It doesn't create a Series object.range(len(df)) + iloc (Slow): Accessing via index is slower than itertuples because of the lookup overhead.iterrows() (Slowest): The overhead of creating Series objects makes it the slowest option.Optimization for your specific codeWhile your current range(len(group)) loop is "okay", using itertuples() would be more Pythonic and slightly faster. However, because solve_alpha_mle (which runs a complex root-finding algorithm) is the bottleneck, the speed difference in the loop method will be negligible.If you want the "cleanest" Pythonic version, replace the loop with itertuples():Python# Instead of: for i in range(len(group)): row = group.iloc[i] ... for i, row in enumerate(group.itertuples()): # Access columns as attributes: row.novelty_count_so_far # Note: 'i' here is just the counter from enumerate, not the index
total_mentions = i + 1 novelty_so_far = row.novelty_count_so_far # faster access current_alpha = solve_alpha_mle(total_mentions, novelty_so_far) # ...Final Check on your LogicThere is one small detail in your code:Pythoncurrent_topic = group.iloc[i]['topic'] # You define this variable But you never use current_topic inside the loop. Since you are calculating Alpha based purely on the counts (total_mentions and novelty_so_far), you can delete this line to clean up the code.
- User
I now want a function that gets a filename and plots a scatter plot of P(new topic) as a function of the prefix length
- Assistant
Pythonimport numpy as np import pandas as pd import matplotlib.pyplot as plt from scipy.special import digamma from scipy.optimize import brentq
Define the solver
def solve_alpha_mle(N, K): if N <= 0 or K <= 0: return np.nan if K == 1: return 0.001 if K == N: return 100.0
def objective(alpha): return (K / alpha) - (digamma(alpha + N) - digamma(alpha)) try: low, high = 1e-5, 1000.0 if objective(low) * objective(high) > 0: return low if objective(low) < 0 else high return brentq(objective, low, high) except Exception: return np.nandef plot_p_new_topic(filename, df): # Filter for the specific file subset = df[df['filename'] == filename].copy()
# Ensure correct sorting (assuming file order/index matters) # If turn_number is reliable for sorting, use it, but preserving index is safer for lines within a turn # subset = subset.sort_values('turn_number') # Actually, we should just trust the original CSV order if it's sequential # Calculate cumulative novelty subset['novelty_int'] = subset['novelty'].astype(int) subset['novelty_cumsum'] = subset['novelty_int'].cumsum() prefix_lengths = [] probabilities = [] # Iterate through prefixes # We use enumerate to get the 0-based index, so N = i + 1 for i in range(len(subset)): N = i + 1 K = subset.iloc[i]['novelty_cumsum'] # Estimate Alpha for this prefix alpha = solve_alpha_mle(N, K) # Calculate P(New Topic) for the NEXT item # Formula: alpha / (alpha + N) # Note: Denominator for the (N+1)th customer is (N + alpha) prob_new = alpha / (alpha + N) prefix_lengths.append(N) probabilities.append(prob_new) # Plotting plt.figure(figsize=(10, 6)) plt.scatter(prefix_lengths, probabilities, alpha=0.6, c='blue', label='P(New Topic)') # Optional: Plot a smoothed line or the actual novelty events? # Let's keep it simple as requested: "scatter plot of P(new topic)" plt.title(f'Probability of New Topic vs. Prefix Length\nFile: {filename}') plt.xlabel('Prefix Length (Number of mentions so far)') plt.ylabel('P(New Topic) for Next Mention') plt.ylim(0, 1.05) plt.grid(True, linestyle='--', alpha=0.7) # Save the plot output_filename = f'p_new_topic_scatter_{filename[:20]}.png' # Truncate filename to avoid issues plt.savefig(output_filename) plt.close() return output_filenameLoad data
df = pd.read_csv('topic_novelty.csv')
Pick a sample filename to demonstrate
sample_filename = df['filename'].unique()[0] print(f"Generating plot for: {sample_filename}")
Run the function
plot_file = plot_p_new_topic(sample_filename, df) print(f"Plot saved to: {plot_file}")
Code outputGenerating plot for: imposing_taxes_on_food_product_For_24turns_5ppl.txt Plot saved to: p_new_topic_scatter_imposing_taxes_on_fo.png
Here is the Python function you requested.This function:Takes a filename (and the full dataframe) as input.Iterates through the conversation step-by-step.Estimates the CRP Alpha (α) for that specific prefix.Calculates the predictive probability that the next turn will introduce a new topic using the formula: P(new)=α+NαGenerates a scatter plot of these probabilities.Pythonimport numpy as np import pandas as pd import matplotlib.pyplot as plt from scipy.special import digamma from scipy.optimize import brentq
1. Helper: The Solver (same as before)
def solve_alpha_mle(N, K): """ Estimates alpha using Maximum Likelihood Estimation. N = Total items so far K = Total unique topics so far """ if N <= 0 or K <= 0: return np.nan if K == 1: return 0.001 # Conformity if K == N: return 100.0 # Novelty
def objective(alpha): # Derivative of the Log-Likelihood return (K / alpha) - (digamma(alpha + N) - digamma(alpha)) try: low, high = 1e-5, 1000.0 if objective(low) * objective(high) > 0: return low if objective(low) < 0 else high return brentq(objective, low, high) except: return np.nan2. The Main Plotting Function
def plot_p_new_topic(filename, df): """ Plots the probability of a new topic appearing as a function of prefix length. """ # Filter data for this file subset = df[df['filename'] == filename].copy()
# Ensure temporal order (assuming file/index order is correct) # subset = subset.sort_values('turn_number') # Calculate cumulative unique topics (K) at every step subset['novelty_int'] = subset['novelty'].astype(int) subset['K_so_far'] = subset['novelty_int'].cumsum() steps = [] probs = [] # Iterate through every prefix length N # Use itertuples for speed for i, row in enumerate(subset.itertuples()): N = i + 1 K = row.K_so_far # Estimate Alpha for this exact moment alpha = solve_alpha_mle(N, K) # Calculate Probability of NEXT item being new # Formula: alpha / (alpha + N) if np.isnan(alpha): p_new = 0.0 else: p_new = alpha / (alpha + N) steps.append(N) probs.append(p_new) # Create Plot plt.figure(figsize=(10, 6)) plt.scatter(steps, probs, color='dodgerblue', alpha=0.7, s=30) # Formatting plt.title(f'Probability of Next Topic Being New\nFile: {filename}') plt.xlabel('Prefix Length (N)') plt.ylabel('P(New Topic)') plt.ylim(-0.05, 1.05) plt.grid(True, linestyle='--', alpha=0.5) # Save safe_name = filename.replace('.txt', '').replace('.', '_') plt.savefig(f'p_new_topic_{safe_name}.png') plt.close() print(f"Plot saved: p_new_topic_{safe_name}.png")--- Example Usage ---
df = pd.read_csv('topic_novelty.csv')
target_file = "imposing_taxes_on_food_product_For_24turns_5ppl.txt"
plot_p_new_topic(target_file, df)
- User
Take the Estimates, the CRP Alpha value from the prior csv file saved
- Assistant
Pythonimport pandas as pd import matplotlib.pyplot as plt import numpy as np
Load the previously generated prefix level alphas
try: df_prefix = pd.read_csv('prefix_level_alphas.csv') except FileNotFoundError: # If the file doesn't exist (e.g. new session), we need to regenerate it using the logic from before. # But for this turn, I will assume it might be there or I'll regenerate it quickly if needed. # To be safe and self-contained, I'll regenerate the necessary dataframe in memory using the previous logic # if the file isn't found, but I'll try to read it first. # Since I cannot rely on persistent state across turns for file existence if the VM reset, # I will re-run the generation logic just to be absolutely sure I have the data. # actually, I'll just re-implement the generation to be safe and ensure the "prior csv" logic is respected. pass
Re-generating the dataframe to ensure we have the data for the plot
This mimics loading the "prior csv"
df_raw = pd.read_csv('topic_novelty.csv')
... (Previous helper functions for alpha calculation needed if I regenerate) ...
But wait, the user specifically asked to "Take the Estimates... from the prior csv file".
This implies they might have downloaded it or it's a conceptual instruction.
In this environment, files from previous turns might be available.
I will check if 'prefix_level_alphas.csv' exists.
import os if not os.path.exists('prefix_level_alphas.csv'): # Regenerate it if missing from scipy.special import digamma from scipy.optimize import brentq
def solve_alpha_mle(N, K): if N <= 0 or K <= 0: return np.nan if K == 1: return 0.001 if K == N: return 100.0 def objective(alpha): return (K / alpha) - (digamma(alpha + N) - digamma(alpha)) try: low, high = 1e-5, 1000.0 if objective(low) * objective(high) > 0: return low if objective(low) < 0 else high return brentq(objective, low, high) except: return np.nan prefix_data = [] df_raw['novelty'] = df_raw['novelty'].astype(int) for filename, group in df_raw.groupby('filename'): group = group.sort_values('turn_number') group['novelty_count_so_far'] = group['novelty'].cumsum() for i in range(len(group)): row = group.iloc[i] N = i + 1 K = row['novelty_count_so_far'] alpha = solve_alpha_mle(N, K) prefix_data.append({ 'filename': filename, 'step': N, 'alpha_at_step': alpha }) df_prefix = pd.DataFrame(prefix_data)else: df_prefix = pd.read_csv('prefix_level_alphas.csv')
def plot_p_new_from_estimates(filename, df_estimates): subset = df_estimates[df_estimates['filename'] == filename].copy() subset = subset.sort_values('step')
# Calculate P(New Topic) using the pre-calculated alpha # P(New) = alpha / (alpha + N) # subset['p_new'] = subset['alpha_at_step'] / (subset['alpha_at_step'] + subset['step']) steps = subset['step'] alphas = subset['alpha_at_step'] probs = alphas / (alphas + steps) plt.figure(figsize=(10, 6)) plt.scatter(steps, probs, color='crimson', alpha=0.7, s=30, label='P(New Topic)') plt.title(f'Probability of New Topic (using Pre-calculated Alpha)\nFile: {filename}') plt.xlabel('Prefix Length (N)') plt.ylabel('P(New Topic)') plt.ylim(-0.05, 1.05) plt.grid(True, linestyle='--', alpha=0.5) plt.legend() output_filename = f'p_new_topic_precalc_{filename[:15]}.png' plt.savefig(output_filename) plt.close() return output_filenamePlot for a sample file
sample_file = df_prefix['filename'].unique()[0] plot_output = plot_p_new_from_estimates(sample_file, df_prefix) print(f"Plot saved: {plot_output}")
Code outputPlot saved: p_new_topic_precalc_affordable_care.png
The plot has been generated using the alpha values from the previously calculated estimates.This new function plot_p_new_from_estimates:Loads the prefix_level_alphas.csv file (re-generating it if the environment reset).Filters for a specific discussion.Calculates the probability P(New) using the stored alpha for each step: αstep+Nαstep.Creates a scatter plot.This approach ensures the plot perfectly matches the "Prefix Analysis" data you generated earlier.Pythonimport pandas as pd import matplotlib.pyplot as plt
def plot_p_new_from_estimates(filename, df_estimates): """ Plots P(New Topic) using the pre-calculated Alpha values from the CSV. """ # Filter for the specific file subset = df_estimates[df_estimates['filename'] == filename].copy() subset = subset.sort_values('step')
# Calculate Probability # Formula: P(New) = alpha / (alpha + N) subset['p_new'] = subset['alpha_at_step'] / (subset['alpha_at_step'] + subset['step']) # Plotting plt.figure(figsize=(10, 6)) plt.scatter(subset['step'], subset['p_new'], color='crimson', alpha=0.7, s=30) plt.title(f'Probability of New Topic vs. Prefix Length\nFile: {filename}') plt.xlabel('Prefix Length (Mentions so far)') plt.ylabel('P(New Topic)') plt.ylim(-0.05, 1.05) plt.grid(True, linestyle='--', alpha=0.5) # Save safe_name = filename.replace('.txt', '').replace('.', '_') plt.savefig(f'p_new_topic_precalc_{safe_name}.png') plt.close()Example Usage:
df_prefix = pd.read_csv('prefix_level_alphas.csv')
target_file = "imposing_taxes_on_food_product_For_24turns_5ppl.txt"
plot_p_new_from_estimates(target_file, df_prefix)
- User
How is this? import numpy as np import pandas as pd from scipy.special import gammaln, digamma from scipy.optimize import minimize_scalar, brentq import matplotlib.pyplot as plt import seaborn as sns sns.set(style="whitegrid") # -------------------------------------------------- # CRP log-likelihood and alpha estimation # -------------------------------------------------- # -------------------------------------------------- # Prefix-based alpha estimation # -------------------------------------------------- # def estimate_prefix_alphas(topics, min_len=2): # """ # Estimate alpha for every prefix of a topic sequence. # Returns list of dicts with prefix length and alpha. # """ # seen = {} # results = [] # # for t, topic in enumerate(topics, start=1): # seen[topic] = seen.get(topic, 0) + 1 # # if t >= min_len: # alpha_t = estimate_alpha(list(seen.values())) # results.append({ # "prefix_length": t, # "alpha": alpha_t, # "num_topics": len(seen) # }) # # return results # -------------------------------------------------- # Main analysis # -------------------------------------------------- def get_number_turns(conv_id): """ filenames are in the format: "f{subject}{stance}{num_turns}turns_{ppl}ppl.txt" :param conv_id: :return: """ return conv_id.split("")[-2].strip("turns") def get_number_participants(conv_id): return conv_id.split("")[-1].strip("ppl.txt") def get_stance(conv_id): """ filenames are in the format: "f{subject}{stance}{num_turns}turns_{ppl}ppl.txt" :param conv_id: :return: """ return conv_id.split("")[-3] def solve_alpha_mle(N, K): """ Estimates alpha using Maximum Likelihood Estimation. Equation: K/alpha - (digamma(alpha + N) - digamma(alpha)) = 0 """ # Edge Case 1: No data if N <= 0 or K <= 0: return np.nan # Edge Case 2: Only 1 topic (Extreme Conformity) # The equation has no solution for alpha > 0, implies alpha -> 0 if K == 1: return 0.001 # Edge Case 3: All topics are unique (Extreme Novelty) # The equation implies alpha -> infinity if K == N: return 100.0 # The Objective Function we want to be zero def objective(alpha): return (K / alpha) - (digamma(alpha + N) - digamma(alpha)) # We use Brent's method (brentq) to find the root # We search for alpha in the range [0.00001, 1000] try: # Check bounds first to ensure a sign change exists low, high = 1e-5, 1000.0 if objective(low) * objective(high) > 0: return low if objective(low) < 0 else high return brentq(objective, low, high) except Exception: return np.nan def prefix_level_analysis(df): """ :return: """ prefix_data = [] for filename, group in df.groupby('filename'): # Sort chronologically group = group.sort_values('turn_number') group['novelty_count_so_far'] = group['novelty'].cumsum() for i in range(len(group)): row = group.iloc[i] total_mentions = i + 1 novelty_so_far = row['novelty_count_so_far'] # Calculate Alpha for this specific moment in time current_alpha = solve_alpha_mle(total_mentions, novelty_so_far) prefix_data.append({ 'filename': filename, 'turn_number': group.iloc[i]['turn_number'], 'step': total_mentions, 'alpha_at_step': current_alpha, "novelty_so_far": novelty_so_far, "num_ppl": get_number_participants(filename), "stance": get_stance(filename) }) df_prefix = pd.DataFrame(prefix_data) df_prefix.to_csv('prefix_level_alphas.csv') def main(): # Load Data df = pd.read_csv('topic_novelty.csv') # --- Implementation --- conversation_data = [] for filename, group in df.groupby('filename'): total_mentions = len(group) # Total turns in conversation unique_topics = group["novelty"].sum() # Total unique topics found estimated_alpha = solve_alpha_mle(total_mentions, unique_topics) conversation_data.append({ "filename": filename, "alpha": estimated_alpha, "num_mentions": total_mentions, "num_novel_topics": group["novelty"].sum(), "num_turns": get_number_turns(filename), "num_ppl": get_number_participants(filename), "stance": get_stance(filename) }) # Convert to DataFrames conversation_df = pd.DataFrame(conversation_data) # Save outputs conversation_df.to_csv("conversation_level_alpha.csv", index=False) prefix_level_analysis(df) print("Analysis complete.") print("Saved:") print(" - conversation_level_alpha.csv") print(" - prefix_level_alpha.csv") def plot_p_new_topic(filename, df): """ Plots the probability of a new topic appearing as a function of prefix length. """ alpha_df = pd.read_csv('prefix_level_alphas.csv') # Filter data for this file subset = df[df['filename'] == filename].copy() # Ensure temporal order (assuming file/index order is correct) # subset = subset.sort_values('turn_number') # Calculate cumulative unique topics (K) at every step subset['novelty_int'] = subset['novelty'].astype(int) subset['K_so_far'] = subset['novelty_int'].cumsum() steps = [] probs = [] # Iterate through every prefix length N # Use itertuples for speed for i, row in enumerate(subset.itertuples()): N = i + 1 K = row.K_so_far # Estimate Alpha for this exact moment alpha = alpha_df[(alpha_df['filename'] == filename) & (alpha_df['step'] == N)]['alpha_at_step'].values[0] # Calculate Probability of NEXT item being new # Formula: alpha / (alpha + N) if np.isnan(alpha): p_new = 0.0 else: p_new = alpha / (alpha + N) steps.append(N) probs.append(p_new) # Create Plot plt.figure(figsize=(10, 6)) plt.scatter(steps, probs, color='dodgerblue', alpha=0.7, s=30) # Formatting plt.title(f'Probability of Next Topic Being New\nFile: {filename}') plt.xlabel('Prefix Length (N)') plt.ylabel('P(New Topic)') plt.ylim(-0.05, 1.05) plt.grid(True, linestyle='--', alpha=0.5) # Save safe_name = filename.replace('.txt', '').replace('.', '') plt.savefig(f'p_new_topic_{safe_name}.png') plt.show() print(f"Plot saved: p_new_topic_{safe_name}.png") def prefix_plots_main(): """ :return: """ # Load Data df = pd.read_csv('topic_novelty.csv') plot_p_new_topic("imposing_taxes_on_food_product_For_24turns_5ppl.txt", df) if name == "main": main() prefix_plots_main()
- Assistant
The code provided generally implements the analysis logic correctly, but it has some redundant parts and inefficiencies, particularly in the plot_p_new_topic function.Key Improvements:Redundant Calculation: In plot_p_new_topic, you calculate K_so_far again, even though prefix_level_alphas.csv (which you load as alpha_df) already has the necessary state (step and alpha). You don't need to process the raw df again; you can just use alpha_df directly.Inefficient Lookup: The line alpha_df[(alpha_df['filename'] == filename) & (alpha_df['step'] == N)] inside the loop is extremely slow (O(N^2)). Since both dataframes are sorted or can be sorted by step, you can just merge them or iterate directly over the filtered alpha_df.Visualization: Added a line plot connecting the scatter points to better visualize the trajectory of the conversation.Here is the refined and optimized version of your code:Pythonimport numpy as np import pandas as pd from scipy.special import digamma from scipy.optimize import brentq import matplotlib.pyplot as plt import seaborn as sns
sns.set(style="whitegrid")
--------------------------------------------------
1. Core Math & Parsing Functions
--------------------------------------------------
def solve_alpha_mle(N, K): """ Estimates alpha using Maximum Likelihood Estimation. Equation: K/alpha - (digamma(alpha + N) - digamma(alpha)) = 0 """ if N <= 0 or K <= 0: return np.nan if K == 1: return 0.001 # Extreme Conformity if K == N: return 100.0 # Extreme Novelty
def objective(alpha): return (K / alpha) - (digamma(alpha + N) - digamma(alpha)) try: low, high = 1e-5, 1000.0 if objective(low) * objective(high) > 0: return low if objective(low) < 0 else high return brentq(objective, low, high) except Exception: return np.nandef get_metadata(filename): """Parses filename to extract metadata.""" parts = filename.replace('.txt', '').split('_') return { "num_turns": parts[-2].replace('turns', ''), "num_ppl": parts[-1].replace('ppl', ''), "stance": parts[-3] }
--------------------------------------------------
2. Analysis Functions
--------------------------------------------------
def run_full_analysis(df): """ Runs both conversation-level and prefix-level analysis. """ conversation_data = [] prefix_data = []
# Ensure novelty is int for calculations df['novelty'] = df['novelty'].astype(int) for filename, group in df.groupby('filename'): # 1. Conversation Level total_mentions = len(group) total_unique = group["novelty"].sum() overall_alpha = solve_alpha_mle(total_mentions, total_unique) meta = get_metadata(filename) conversation_data.append({ "filename": filename, "alpha": overall_alpha, "num_mentions": total_mentions, "num_novel_topics": total_unique, **meta }) # 2. Prefix Level group = group.sort_values('turn_number') # Fast vectorized cumulative sum group['novelty_cumsum'] = group['novelty'].cumsum() # Itertuples is faster than iterrows for i, row in enumerate(group.itertuples()): step = i + 1 novelty_so_far = row.novelty_cumsum current_alpha = solve_alpha_mle(step, novelty_so_far) prefix_data.append({ 'filename': filename, 'step': step, 'alpha_at_step': current_alpha, 'novelty_so_far': novelty_so_far, **meta }) return pd.DataFrame(conversation_data), pd.DataFrame(prefix_data)--------------------------------------------------
3. Plotting Function (Optimized)
--------------------------------------------------
def plot_p_new_topic(filename, alpha_df): """ Plots P(New Topic) vs Prefix Length using pre-calculated alphas. """ # Filter and sort - no need to join with original DF subset = alpha_df[alpha_df['filename'] == filename].sort_values('step')
if subset.empty: print(f"No data found for {filename}") return # Vectorized calculation of Probability # P(New) = alpha / (alpha + N) steps = subset['step'] alphas = subset['alpha_at_step'] probs = alphas / (alphas + steps) # Plotting plt.figure(figsize=(10, 6)) # Line + Scatter for better visibility of trends plt.plot(steps, probs, color='dodgerblue', alpha=0.4, linestyle='-') plt.scatter(steps, probs, color='dodgerblue', alpha=0.8, s=40, label='P(New Topic)') plt.title(f'Probability of Next Topic Being New\nFile: {filename}') plt.xlabel('Prefix Length (N)') plt.ylabel('P(New Topic)') plt.ylim(-0.05, 1.05) plt.legend() safe_name = filename.replace('.txt', '').replace('.', '_') plt.savefig(f'p_new_topic_{safe_name}.png') plt.close() # Close to free memory print(f"Plot saved: p_new_topic_{safe_name}.png")--------------------------------------------------
4. Main Execution
--------------------------------------------------
def main(): print("Loading data...") df = pd.read_csv('topic_novelty.csv')
print("Running analysis...") conv_df, prefix_df = run_full_analysis(df) print("Saving results...") conv_df.to_csv("conversation_level_alpha.csv", index=False) prefix_df.to_csv("prefix_level_alphas.csv", index=False) # Plotting specific file as requested target_file = "imposing_taxes_on_food_product_For_24turns_5ppl.txt" plot_p_new_topic(target_file, prefix_df) print("Done.")if name == "main": main()
- User
Great. I think it can be interesting to plot the p(new_topic) for the final alpha value (the entire discussion, not prefixes) on the same plot for camparison. What to you think?
- Assistant
Pythonimport numpy as np import pandas as pd from scipy.special import digamma from scipy.optimize import brentq import matplotlib.pyplot as plt import seaborn as sns
sns.set(style="whitegrid")
--- Helper Functions (Same as before) ---
def solve_alpha_mle(N, K): if N <= 0 or K <= 0: return np.nan if K == 1: return 0.001 if K == N: return 100.0 def objective(alpha): return (K / alpha) - (digamma(alpha + N) - digamma(alpha)) try: low, high = 1e-5, 1000.0 if objective(low) * objective(high) > 0: return low if objective(low) < 0 else high return brentq(objective, low, high) except: return np.nan
def get_metadata(filename): parts = filename.replace('.txt', '').split('_') return { "num_turns": parts[-2].replace('turns', ''), "num_ppl": parts[-1].replace('ppl', ''), "stance": parts[-3] }
def run_full_analysis(df): conversation_data = [] prefix_data = [] df['novelty'] = df['novelty'].astype(int)
for filename, group in df.groupby('filename'): # Conversation Level total_mentions = len(group) total_unique = group["novelty"].sum() overall_alpha = solve_alpha_mle(total_mentions, total_unique) meta = get_metadata(filename) conversation_data.append({ "filename": filename, "alpha": overall_alpha, "num_mentions": total_mentions, "num_novel_topics": total_unique, **meta }) # Prefix Level group = group.sort_values('turn_number') group['novelty_cumsum'] = group['novelty'].cumsum() for i, row in enumerate(group.itertuples()): step = i + 1 novelty_so_far = row.novelty_cumsum current_alpha = solve_alpha_mle(step, novelty_so_far) prefix_data.append({ 'filename': filename, 'step': step, 'alpha_at_step': current_alpha, 'novelty_so_far': novelty_so_far, **meta }) return pd.DataFrame(conversation_data), pd.DataFrame(prefix_data)--- Updated Plotting Function ---
def plot_p_new_topic_comparison(filename, prefix_df, conv_df): """ Plots P(New Topic) vs Prefix Length. Compares the dynamic prefix-based probability with the static global probability. """ # 1. Get Prefix Data (Dynamic) subset = prefix_df[prefix_df['filename'] == filename].sort_values('step') if subset.empty: print(f"No data found for {filename}") return
steps = subset['step'] alphas_dynamic = subset['alpha_at_step'] probs_dynamic = alphas_dynamic / (alphas_dynamic + steps) # 2. Get Conversation Data (Static Global Alpha) global_row = conv_df[conv_df['filename'] == filename] if global_row.empty: print("Global alpha not found.") return alpha_global = global_row.iloc[0]['alpha'] # Calculate what the probability trajectory WOULD be if alpha was constant probs_global = alpha_global / (alpha_global + steps) # 3. Plotting plt.figure(figsize=(10, 6)) # Dynamic Line (The "Real" evolution) plt.plot(steps, probs_dynamic, color='dodgerblue', alpha=0.5, linestyle='-') plt.scatter(steps, probs_dynamic, color='dodgerblue', alpha=0.8, s=40, label='Dynamic (Prefix-based)') # Static Line (The "Average" expectation) plt.plot(steps, probs_global, color='crimson', alpha=0.8, linestyle='--', linewidth=2, label=f'Global Model ($\\alpha={alpha_global:.2f}$)') plt.title(f'Probability of Next Topic Being New: Dynamic vs Global\nFile: {filename}') plt.xlabel('Prefix Length (N)') plt.ylabel('P(New Topic)') plt.ylim(-0.05, 1.05) plt.legend() plt.grid(True, linestyle='--', alpha=0.5) safe_name = filename.replace('.txt', '').replace('.', '_') output_filename = f'p_new_topic_comparison_{safe_name}.png' plt.savefig(output_filename) plt.close() print(f"Plot saved: {output_filename}")--- Main Execution ---
df = pd.read_csv('topic_novelty.csv') conv_df, prefix_df = run_full_analysis(df) target_file = "imposing_taxes_on_food_product_For_24turns_5ppl.txt" plot_p_new_topic_comparison(target_file, prefix_df, conv_df)
Code outputPlot saved: p_new_topic_comparison_imposing_taxes_on_food_product_For_24turns_5ppl.png
That is a great idea. Comparing the dynamic probability (which changes as the conversation evolves) against the static probability (what we would expect if the whole conversation had a single constant "creativity" parameter) reveals whether the group is becoming more or less creative than their average over time.If the Blue Dots (Dynamic) are above the Red Dashed Line (Global), the group is in a "hot streak" of introducing new topics (more creative than their average).If the Blue Dots are below the line, they are converging and reusing old topics (less creative than their average).Here is the updated code that plots both lines:Pythonimport numpy as np import pandas as pd import matplotlib.pyplot as plt import seaborn as sns
sns.set(style="whitegrid")
def plot_p_new_topic_comparison(filename, prefix_df, conv_df): """ Plots P(New Topic) vs Prefix Length. Compares the dynamic prefix-based probability with the static global probability. """ # 1. Get Prefix Data (The "Real" evolution) subset = prefix_df[prefix_df['filename'] == filename].sort_values('step') if subset.empty: print(f"No data found for {filename}") return
steps = subset['step'] alphas_dynamic = subset['alpha_at_step'] # Calculate Dynamic Probability: alpha_t / (alpha_t + N) probs_dynamic = alphas_dynamic / (alphas_dynamic + steps) # 2. Get Global Data (The "Average" expectation) global_row = conv_df[conv_df['filename'] == filename] if global_row.empty: print("Global alpha not found.") return alpha_global = global_row.iloc[0]['alpha'] # Calculate Global Curve: alpha_global / (alpha_global + N) # This shows what the curve WOUL look like if alpha never changed probs_global = alpha_global / (alpha_global + steps) # 3. Plotting plt.figure(figsize=(10, 6)) # Plot Dynamic (Blue) plt.plot(steps, probs_dynamic, color='dodgerblue', alpha=0.4, linestyle='-') plt.scatter(steps, probs_dynamic, color='dodgerblue', alpha=0.8, s=40, label='Dynamic (Prefix-based)') # Plot Global (Red Dashed) plt.plot(steps, probs_global, color='crimson', alpha=0.9, linestyle='--', linewidth=2, label=f'Global Model ($\\alpha={alpha_global:.2f}$)') # Formatting plt.title(f'Probability of New Topic: Dynamic vs Global\nFile: {filename}') plt.xlabel('Prefix Length (Mentions so far)') plt.ylabel('P(New Topic)') plt.ylim(-0.05, 1.05) plt.legend() plt.grid(True, linestyle='--', alpha=0.5) # Save safe_name = filename.replace('.txt', '').replace('.', '_') plt.savefig(f'p_new_topic_comparison_{safe_name}.png') plt.show()--- Example Usage ---
Assuming you have run run_full_analysis() and have the dataframes:
target_file = "imposing_taxes_on_food_product_For_24turns_5ppl.txt"
plot_p_new_topic_comparison(target_file, prefix_df, conv_df)
- User
In the same domain, I want a funtion that visualizes the overall rate of topics arriving, for a given discussion: the observed number of unique topics vs. the number of mentions (prefix length). In addition, on the same plot, visualize the expected number of topics arriving vs. the number of mentions.
- Assistant
Pythonimport numpy as np import pandas as pd import matplotlib.pyplot as plt from scipy.special import digamma, gammaln from scipy.optimize import brentq
Re-defining helper functions to ensure standalone execution
def solve_alpha_mle(N, K): if N <= 0 or K <= 0: return np.nan if K == 1: return 0.001 if K == N: return 100.0 def objective(alpha): return (K / alpha) - (digamma(alpha + N) - digamma(alpha)) try: low, high = 1e-5, 1000.0 if objective(low) * objective(high) > 0: return low if objective(low) < 0 else high return brentq(objective, low, high) except: return np.nan
def plot_topic_arrival_rate(filename, df): # 1. Prepare Data subset = df[df['filename'] == filename].copy() subset['novelty'] = subset['novelty'].astype(int)
# Ensure sorted order # Assuming turn_number is sufficient, or just index if file is sequential # For safety, we trust the file order or sort by turn_number # subset = subset.sort_values('turn_number') # Observed Data N_total = len(subset) subset['observed_K'] = subset['novelty'].cumsum() observed_K = subset['observed_K'].values steps = np.arange(1, N_total + 1) # 2. Estimate Global Alpha K_total = subset['novelty'].sum() alpha = solve_alpha_mle(N_total, K_total) # 3. Calculate Expected Data (Model) # E[K_n] = sum_{i=0}^{n-1} alpha / (alpha + i) # We can compute the probabilities for i=0 to N_total-1 indices = np.arange(N_total) # 0, 1, ..., N-1 probs_new = alpha / (alpha + indices) expected_K = np.cumsum(probs_new) # 4. Plotting plt.figure(figsize=(10, 6)) # Observed Line plt.plot(steps, observed_K, label='Observed Topics', color='black', linewidth=2) plt.scatter(steps, observed_K, color='black', s=10, alpha=0.5) # Expected Line plt.plot(steps, expected_K, label=f'Expected Topics ($\\alpha={alpha:.2f}$)', color='red', linestyle='--', linewidth=2) # Formatting plt.title(f'Topic Arrival Rate: Observed vs. Expected\nFile: {filename}') plt.xlabel('Number of Mentions (N)') plt.ylabel('Number of Unique Topics (K)') plt.legend() plt.grid(True, linestyle='--', alpha=0.5) # Save safe_name = filename.replace('.txt', '').replace('.', '_') output_filename = f'topic_arrival_{safe_name}.png' plt.savefig(output_filename) plt.close() print(f"Plot saved: {output_filename}") return output_filenameLoad Data and Run
df = pd.read_csv('topic_novelty.csv') target_file = "imposing_taxes_on_food_product_For_24turns_5ppl.txt" plot_topic_arrival_rate(target_file, df)
Code outputPlot saved: topic_arrival_imposing_taxes_on_food_product_For_24turns_5ppl.png
Here is the function plot_topic_arrival_rate. It visualizes the "growth chart" of the discussion.Black Line (Observed): The actual number of unique topics accumulating over time.Steep slope = lots of new ideas.Flat slope = rehashing old ideas.Red Dashed Line (Expected): The theoretical growth curve predicted by the CRP model using the overall α.If the black line is above the red line, the group was introducing topics faster than predicted by their average rate (a "burst" of creativity).If the black line is below, they were repeating themselves more than expected.Pythonimport numpy as np import pandas as pd import matplotlib.pyplot as plt from scipy.special import digamma from scipy.optimize import brentq
Helper to find Alpha (needed if not passed in)
def solve_alpha_mle(N, K): if N <= 0 or K <= 0: return np.nan if K == 1: return 0.001 if K == N: return 100.0 def objective(alpha): return (K / alpha) - (digamma(alpha + N) - digamma(alpha)) try: return brentq(objective, 1e-5, 1000.0) except: return np.nan
def plot_topic_arrival_rate(filename, df): """ Plots the cumulative number of unique topics vs. mentions. Compares the Observed data against the Expected curve from the CRP model. """ # 1. Prepare Observed Data subset = df[df['filename'] == filename].copy()
# Calculate cumulative observed topics subset['novelty_int'] = subset['novelty'].astype(int) observed_K = subset['novelty_int'].cumsum().values N_total = len(subset) steps = np.arange(1, N_total + 1) # 2. Estimate Global Alpha for the Model K_total = subset['novelty_int'].sum() alpha = solve_alpha_mle(N_total, K_total) # 3. Calculate Expected Curve # The expected number of topics at step n is the sum of probabilities # of a new topic at each previous step i: P(new) = alpha / (alpha + i) indices = np.arange(N_total) # 0 to N-1 probs_new = alpha / (alpha + indices) expected_K = np.cumsum(probs_new) # 4. Plotting plt.figure(figsize=(10, 6)) # Plot Observed (Actual) plt.plot(steps, observed_K, label='Observed Data', color='black', linewidth=2.5) # Plot Expected (Model) plt.plot(steps, expected_K, label=f'CRP Model Expectation ($\\alpha={alpha:.2f}$)', color='firebrick', linestyle='--', linewidth=2) # Formatting plt.title(f'Topic Arrival Rate: Observed vs. Model\nFile: {filename}') plt.xlabel('Number of Mentions (Prefix Length)') plt.ylabel('Count of Unique Topics Found') plt.legend() plt.grid(True, linestyle='--', alpha=0.5) # Save safe_name = filename.replace('.txt', '').replace('.', '_') plt.savefig(f'topic_arrival_{safe_name}.png') plt.close() print(f"Plot saved: topic_arrival_{safe_name}.png")Usage
df = pd.read_csv('topic_novelty.csv')
plot_topic_arrival_rate("imposing_taxes_on_food_product_For_24turns_5ppl.txt", df)