Skip to content
The Chat Archive
ChatGPT (model unspecified)· OpenAI ShareLM (browser-extension contributions)

computationalanalysissocial_roles.pdfPDFHelp me understand and compute LSA spaces. Here…

  1. User

    computational_analysis_social_roles.pdfPDFHelp me understand and compute LSA spaces. Here is the supplementary material:

    """ Computing LSA Spaces Each dataset was used to generate a distinct LSA space used for calculating the GCA measures on that dataset. This ensures that each corpus of chat transcripts is given an appropriate semantic representation for the material being discussed. The principal difficulty in generating an LSA space from chat transcripts is that subjects and topics referenced in natural conversations are not sufficiently defined to provide a comprehensive mapping of their semantic relationships. We take for granted that our conversational partners already have a well-developed understanding of a vast array of topics. For example, one may engage in a perfectly coherent conversation with a friend or colleague about careers, food, family or any number of other subjects, without ever needing to provide a comprehensive verbal description of any of these subjects. Therefore, we must supplement the chat transcripts with appropriate external documents in order to robustly represent the semantic space of subjects referenced in a conversation (Cai, Burkett, Morgan, & Shaffer, 2011). To this end, we analyze the frequencies of terms used in the discussion in order to identify the most significant terms (keywords), and then query publicly available databases (i.e., Wikipedia) for documents on those topics. This process of scanning for keywords can be repeated with the newly added documents until a satisfactory number of documents has been obtained to generate a reasonable mapping of the semantic space. Finally, an LSA space of 300 dimensions was computed from each expanded corpus. Table S1 provides the descriptive information for the original chat corpora, the extended corpora, and LSA spaces for each data set. Table S1 Total Terms and Unique Terms for each Data Set, Expanded Corpus, and LSA Space Dataset Chat Transcripts Expanded Corpus LSA Space

    Total Terms Unique Terms Total Terms Unique Terms Unique Terms Traditional CSCL 130,946 6,010 2,703,978 91,613 32,297 SMOC 457,639 14,207 8,024,354 149,188 56,609 Land Science 401,652 9,932 1,981,589 73,702 25,417 """

    I with also send you my code

  2. Assistant

    Quick explanation + runnable recipe for computing LSA spaces (and using them like the paper does) Short version: the authors build a domain-specific LSA for each chat dataset by (1) making a seed corpus from the chat transcripts + any assigned readings, (2) extracting important terms (keywords) from that seed, (3) expanding the corpus by fetching documents about those keywords (they used Wikipedia), and (4) building an LSA (SVD) on the expanded corpus and keeping 300 dimensions. That LSA is then used to project each chat contribution to a vector and compute cosine similarities, given/new projections, and the cohesion/responsivity measurements described in the paper.
    Below I give: (A) an explained checklist of the steps, (B) concrete Python code you can run (no web calls included — for expansion you can plug in Wikipedia fetches), and (C) the exact math for the operations the paper implements (projection, cosine, given–new, cross-cohesion, etc.). Use this as a template to reproduce the paper’s LSA spaces and the GCA measures.

    A. Conceptual checklist (what you need and why)

    Seed corpus: your chat transcripts + any course readings or documents the participants were expected to use. (paper: seed corpus → scan keywords → expand with Wikipedia).

    Keyword extraction / expansion: extract top content words (tf-idf, RAKE, or simple frequency + POS filter). Use those keywords to pull in documents (Wikipedia pages, Wikipedia sections, or other domain pages). Repeat if needed until the expanded corpus is sufficiently large and covers the domain. (paper used Wikipedia).

    Text preprocessing: tokenization, lowercasing, remove punctuation, optionally lemmatize/stem, remove stopwords but be careful — some stop-words matter in cohesion tasks (decide consistently).

    Construct term × document matrix: choose your document unit for the expanded corpus (e.g., paragraphs or full articles). For chat LSA, the paper used documents as units that make sense for LSA (sentence/paragraph).

    Weighting: TF-IDF or log-entropy. The paper notes TF-IDF and log-entropy as common choices. TF-IDF is typical and easy to reproduce.

    SVD / dimensionality reduction: run SVD on the weighted matrix and reduce to k dimensions. The paper used k = 300. (They state ~100–500 dimensions are standard; they used 300).

    Get document vectors & term vectors: after SVD you can project documents into k-dimensional LSA space (these are your document vectors). Short chat contributions (turns) are projected the same way (build a tfidf vector for each contribution, then apply the SVD transform).

    Normalize vectors: for cosine similarity and projection math you usually L2-normalize vectors. The authors add normalized term vectors to obtain contribution vectors — an equivalent practical step is to tfidf-vectorize each chat contribution and then apply the TruncatedSVD transform; finally L2-normalize.

    Use those vectors to compute: cosine similarities, given–new (project onto span of previous document vectors), communication density (norm / number of words), cross-cohesion and responsivity as in the paper.

    B. Concrete Python code (scikit-learn / numpy) — copy/paste and run This is a robust, reproducible pipeline. It uses TF-IDF + TruncatedSVD(300). It shows how to:

    build the LSA

    project chat contributions

    compute cosine similarity, given/new (newness) and communication density

    compute cross-cohesion for a lag window and responsivity averages like the paper

    Note: the code below does not fetch Wikipedia; plug in your expanded corpus (list of documents) where indicated. I kept the code explicit and commented so you can drop your own corpus and chat turns in.

    pythonCopyEdit# Python 3.9+; install: pip install scikit-learn numpy scipy nltk import numpy as np from sklearn.feature_extraction.text import TfidfVectorizer from sklearn.decomposition import TruncatedSVD from sklearn.preprocessing import normalize from scipy.spatial.distance import cosine from scipy.linalg import orth import math

    ---------- Inputs you must provide ----------

    expanded_corpus: list[str] of the documents used to build the LSA (seed + Wikipedia pages)

    chat_contributions: list[dict] with each contribution: {'participant': 'userA', 'text': '...'}

    Example:

    expanded_corpus = ["Wikipedia article 1 text ...", "Wikipedia article 2 ...", ...]

    chat_contributions = [

    {'participant':'A','text':'I think personality disorders involve...'},

    {'participant':'B','text':'Could you give an example?'},

    ...

    ]

    ---------- Step 1: build TF-IDF and truncated SVD (LSA) ----------

    n_components = 300 # the paper used 300 vectorizer = TfidfVectorizer(max_df=0.95, min_df=2, stop_words='english') # tune min_df as needed X_tfidf = vectorizer.fit_transform(expanded_corpus) # shape: (n_docs, n_terms)

    svd = TruncatedSVD(n_components=n_components, n_iter=10, random_state=42) X_lsa = svd.fit_transform(X_tfidf) # shape: (n_docs, n_components)

    We can L2-normalize document vectors for cosine use:

    X_lsa = normalize(X_lsa, norm='l2', axis=1)

    ---------- Step 2: project chat contributions into same LSA space ----------

    Build tfidf for contributions using the same vectorizer (transform only)

    contrib_texts = [c['text'] for c in chat_contributions] C_tfidf = vectorizer.transform(contrib_texts) # shape: (n_contribs, n_terms)

    transform with SVD model — gives LSA coordinates for each contribution

    C_lsa = svd.transform(C_tfidf) # shape: (n_contribs, n_components) C_lsa = normalize(C_lsa, norm='l2', axis=1)

    ---------- Utilities ----------

    def cosine_sim_matrix(mat): # mat: (n, dim) L2-normalized rows -> cosine similarity = dot product return np.dot(mat, mat.T)

    def cosine(a, b): # both vectors assumed not identically zero; returns cosine similarity denom = np.linalg.norm(a) * np.linalg.norm(b) return float(np.dot(a, b) / denom) if denom > 0 else 0.0

    ---------- Given/New decomposition (Hu et al. method used in paper) ----------

    For each contribution t, define G_t = span(d_1, ..., d_{t-1}). Project d_t onto G_t and orthogonal complement.

    def given_new_scores(C_vectors): # C_vectors: (n_contribs, dim) not necessarily normalized for projection (we'll use raw vectors) n, dim = C_vectors.shape given_scores = np.zeros(n) new_scores = np.zeros(n) for t in range(n): if t == 0: # no prior content => all new given_scores[t] = 0.0 new_scores[t] = 1.0 continue # build orthonormal basis Q for previous vectors (use QR on transpose) prev = C_vectors[:t, :].T # shape (dim, t) # if rank deficiency (fewer prev than dim), use qr on prev Q, _ = np.linalg.qr(prev) # Q shape (dim, dim) but only first t columns are meaningful # project current vector onto span(prev): g = Q Q^T d (but Q has dim columns, we only need first t) # Using tall-skinny prev: get Q_t = first t columns of Q Q_t = Q[:, :t] d = C_vectors[t] g = Q_t.dot(Q_t.T.dot(d)) nvec = d - g # norms ng = np.linalg.norm(g) nn = np.linalg.norm(nvec) # given-new as in paper: n(ct) = ||n|| / (||n|| + ||g||) denom = nn + ng n_val = nn / denom if denom > 0 else 0.0 given_scores[t] = 1.0 - n_val new_scores[t] = n_val return given_scores, new_scores

    Call it:

    For projection math it's better to use non-normalized LSA vectors to preserve magnitudes,

    but paper uses normalized term-vectors added; using the tfidf->svd outputs is acceptable.

    C_lsa_nonorm = svd.transform(C_tfidf) # raw (not normalized) given_scores, new_scores = given_new_scores(C_lsa_nonorm)

    ---------- Communication density ----------

    For each contribution i: D_i = ||d_i|| / (number_of_words_in_contrib)

    word_counts = np.array([len(c['text'].split()) for c in chat_contributions]) norms = np.linalg.norm(C_lsa_nonorm, axis=1) comm_density = np.where(word_counts > 0, norms / word_counts, 0.0)

    ---------- Cross-cohesion and responsivity (τ-lagged averages) ----------

    We'll compute ξ_ab(τ) as average cosine similarity between contributions of a at time t-τ and b at time t.

    Implementation plan: iterate through chat contributions (t from 0..n-1); find initiator author at t-τ and responder at t.

    participants = [c['participant'] for c in chat_contributions] unique_parts = sorted(set(participants)) part_to_idx = {p:i for i,p in enumerate(unique_parts)} k = len(unique_parts) n = len(chat_contributions)

    Precompute cosine similarity matrix for all pairs of contributions

    sim = cosine_sim_matrix(normalize(C_lsa_nonorm, axis=1)) # L2-normalize for cosine = dot product

    function to compute ξ_ab(τ)

    def cross_cohesion_tau(tau, sim_matrix, participants): k = len(unique_parts) num = np.zeros((k,k)) denom = np.zeros((k,k)) for t in range(tau, n): responder = participants[t] responder_idx = part_to_idx[responder] initiator = participants[t - tau] initiator_idx = part_to_idx[initiator] s = sim_matrix[t - tau, t] # cos(d_{t-tau}, d_t) num[responder_idx, initiator_idx] += s denom[responder_idx, initiator_idx] += 1 # average where denom>0 with np.errstate(divide='ignore', invalid='ignore'): xi = np.where(denom>0, num/denom, 0.0) return xi, denom

    Example: compute cross-cohesion for tau=1..w and get w-spanning responsivity R(w)

    w = 3 X_tau_sum = np.zeros((k,k)) count_tau = np.zeros((k,k)) for tau in range(1, w+1): xi, denom = cross_cohesion_tau(tau, sim, participants) X_tau_sum += xi count_tau += (denom > 0).astype(float) # just counting availability; paper averages over windows differently; this is an operational approach

    R_w = X_tau_sum / w # average of matrices (paper uses simple average across τ=1..w)

    participant-level measures:

    overall_responsivity = np.array([np.mean([R_w[i,j] for j in range(k) if j!=i]) for i in range(k)]) social_impact = np.array([np.mean([R_w[j,i] for j in range(k) if j!=i]) for i in range(k)]) internal_cohesion = np.diag(R_w) # diagonal is autocohesion averaged over w

    map results back to participant ids

    results = {} for p, idx in part_to_idx.items(): results[p] = { 'overall_responsivity': float(overall_responsivity[idx]), 'social_impact': float(social_impact[idx]), 'internal_cohesion': float(internal_cohesion[idx]), 'avg_newness': float(np.mean(new_scores[[i for i,c in enumerate(chat_contributions) if c['participant']==p]])) if any(c['participant']==p for c in chat_contributions) else 0.0, 'comm_density': float(np.mean(comm_density[[i for i,c in enumerate(chat_contributions) if c['participant']==p]])) if any(c['participant']==p for c in chat_contributions) else 0.0 }

    results now holds GCA-style measures per participant

    Notes about the code and correspondence to the paper

    The paper constructs the LSA using a term×document matrix built from the expanded corpus (seed + Wikipedia), then SVD to 300 dims; the code does the same with TF-IDF + TruncatedSVD(300).

    For given-new the paper projects the contribution vector onto the span of previous contributions (their Eq.21–26). The function given_new_scores follows that idea: build orthonormal basis of previous document vectors and compute projections (g_t and n_t).

    For cross-cohesion and responsivity the code computes ξ_ab(τ) by averaging cosine similarity for τ-lagged pairs (paper Eq.15–17). The paper averages across τ=1..w to make R(w); the code demonstrates that approach and then computes participant means for overall responsivity, social impact, and internal cohesion.

    C. Implementation tips, choices & pitfalls (from the paper + practical notes)

    Document unit for LSA: using paragraphs or full Wikipedia articles changes the term-document counts. The paper suggests using documents at a unit that captures topical structure (paragraphs or article sections often work).

    Weighting scheme: TF-IDF is fine. Log-entropy is another option (harder to implement but sometimes preferred in classic LSA literature). The paper mentions both TF-IDF and log-entropy as common.

    Dimensionality (k): they used 300 dimensions — a reasonable choice for conversational + Wikipedia expanded corpora; acceptable range 100–500. If you want to tune, compute reconstruction error or downstream stability of cosine similarities with different k.

    Short texts (chat turns): short contributions are sparse; projecting via the same TF-IDF+SVD pipeline is appropriate. The paper sums normalized term vectors as a practical alternative; transforming the tfidf vector by the SVD is equivalent and simpler.

    Expansion strategy: iterate keyword extraction → fetch Wikipedia pages → add pages → re-extract keywords etc., until the expanded corpus is large and covers the topics in the chats. The paper used that seed/expand loop.

    Normalization and magnitudes: LSA vector magnitudes can matter: communication density uses ‖d_i‖ / word_count. The paper normalizes where appropriate for cosine similarity but uses raw norms for density and given/new magnitude computations.

    D. How this ties back to Table S1 in the supplementary material Table S1 in your snippet reports total terms and unique terms for chat transcripts, the expanded corpus, and the final LSA vocabulary sizes (they report unique terms in final LSA space). To reproduce those numbers you should:

    count tokens and unique tokens in your original chat corpus (after your chosen preprocessing).

    count tokens and unique tokens in the expanded corpus (after adding Wikipedia pages).

    the final LSA unique terms equals the vocabulary size after vectorizer.fit(...) (terms that meet min_df and were not removed as stopwords). The paper reports those numbers per dataset and that they computed a 300-dimensional LSA for each expanded corpus.

    E. Next steps I can do for you (pick any, no waiting required)

    Run this pipeline on your actual transcripts and expanded corpus (you mentioned you'll send code — send your code + data and I’ll adapt/merge it into this pipeline).

    Review your code and point out mismatches vs. the paper (I can refactor your code to mirror the paper exactly).

    Add automatic Wikipedia expansion code (I can show a robust approach using the wikipedia python package or MediaWiki API for production data collection).

    Run dimension selection experiments (compute stability of similarity measures for k ∈ {100,200,300,400}).

    If you want, paste your code (you said you’ll send it) and I’ll inline-review it and adapt it to the exact computations in the paper (LSA build, 300 dims, given/new projection, cross-cohesion/responsivity). I already used your uploaded article for method details above.

  3. User

    I don't have external data. This is a Hebrew dataset. Do you have recommendations for me?

  4. Assistant

    Nice — Hebrew + no external data is totally workable. Below I’ll give focused, practical recommendations and a ready-to-run pipeline you can use without fetching any external corpora. It covers Hebrew preprocessing quirks, ways to expand your internal data into a usable LSA corpus, LSA-building choices, and checks to validate the space.

    1. High-level strategy (short)

    You don’t strictly need external documents. Create an expanded internal corpus by aggregating and re-slicing your chat data (thread/participant/keyword-based pseudo-documents, sliding windows, concatenated context).

    Use Hebrew-specific text normalization (remove niqqud, normalize final forms, handle clitics/prefixes) and character n-grams to reduce sparsity from rich morphology.

    Use TF–IDF (with word and character n-grams) → TruncatedSVD (LSA) with tuned k (start lower than 300 if your corpus is small) → project chat turns to the space.

    Validate: nearest neighbours for known terms, stability across dimensions, and sanity-check GCA measures.

    1. Hebrew-specific preprocessing recommendations

    Remove Hebrew diacritics (niqqud / cantillation) — they’re rarely present and break token matching.

    Normalize final forms: ך→כ, ם→מ, ן→נ, ף→פ, ץ→צ so tokens with final forms match root forms.

    Strip / split clitics and common single-letter prefixes (ב, כ, ל, ו, מ, ש, the definite article ה) carefully — but do it conservatively (e.g., remove only when token length > 3 after removing).

    Keep pronouns/negation if your downstream cohesion measures rely on them — sometimes stopwords carry discourse info. Decide consistently.

    Use char n-grams (e.g., char_wb 3–5) in addition to word tokens to capture rich inflectional patterns.

    Lowercase (Hebrew is case-insensitive but if you have mixed Latin tokens, normalize them too).

    Tokenization: whitespace + punctuation, but expect that punctuation and emoticons are useful signals in chats — consider keeping them as tokens or at least record them.

    1. How to expand the corpus without external data You can create a strong LSA domain from the chat data itself by generating pseudo-documents: A. Aggregate by conversation unit:

    Treat each conversation/thread or time window (e.g., sliding 5-minute windows) as a “document”.

    B. Aggregate by participant:

    For each participant, concat their messages into documents (maybe split into chunks if large).

    C. Keyword-driven pseudo-documents:

    Extract top N keywords (tf-idf on the chat corpus). For each keyword, make a document by concatenating all messages that contain that keyword — that raises term co-occurrence counts and yields richer documents.

    D. Contextual expansion:

    For each short turn, create a document = (turn −1, turn, turn +1) — that gives fuller topic coverage.

    E. Sub-sampling + paragraphization:

    Break long concatenations into paragraph-sized docs (e.g., 200–500 tokens) to increase document count.

    Do several of these and combine their documents into one expanded_corpus for the LSA builder. 4) Weighting & LSA dimensionality choices

    Weighting: TF–IDF with sublinear tf (e.g., tf = 1 + log(tf)) works well for short text. Include both word n-grams (1–2) and character n-grams (3–5).

    k (dimensions): If your expanded corpus is small (<100k tokens), 300 may be too large. Start with 100 or 150 and test 100–300. Keep k less than the effective rank of your term-doc matrix.

    SVDImpl: use randomized TruncatedSVD (scikit-learn) for speed and stability.

    Normalization: L2-normalize vectors before cosine similarity.

    1. Quick evaluation checks (do these to ensure your LSA is sensible)

    Nearest neighbors: pick a few Hebrew seed words and inspect top nearest words — check for semantic coherence.

    Term overlap & coverage: compare vocab size vs unique tokens; ensure common terms are present (after preprocessing).

    Dimensional stability: compute average pairwise cosine for held-out chat turns across different k. If results vary wildly, adjust k or aggregation strategy.

    Downstream sanity: compute a few GCA measures (given/new, cohesion) and inspect top/bottom contributors — do they match your domain knowledge?

    1. Ready-to-run Python pipeline (no external downloads) This pipeline:

    Preprocesses Hebrew text (diacritics removal, final form normalization, simple clitic stripping),

    Builds internal expanded corpus using keyword-driven pseudo-documents and sliding windows,

    Constructs TF–IDF (word + char n-grams) → TruncatedSVD → projects chat turns,

    Computes given/new and a simple cross-cohesion.

    Paste your chat data into chat_contributions and run. pythonCopyEdit# Requires: pip install scikit-learn numpy regex import re, math, numpy as np from sklearn.feature_extraction.text import TfidfVectorizer from sklearn.decomposition import TruncatedSVD from sklearn.preprocessing import normalize from collections import defaultdict, Counter

    ---------- Hebrew preprocessing helpers ----------

    Remove Hebrew diacritics (niqqud/cantillation): Unicode range U+0591..U+05C7

    DIACRITICS_RE = re.compile(r'[\u0591-\u05C7]') PUNCT_RE = re.compile(r'[^\w\u0590-\u05FF]', flags=re.UNICODE) # removes punctuation but keeps Hebrew letters

    final_map = str.maketrans({'ך':'כ','ם':'מ','ן':'נ','ף':'פ','ץ':'צ'})

    def normalize_hebrew(text, strip_diacritics=True, strip_punct=False, lower=True): if strip_diacritics: text = DIACRITICS_RE.sub('', text) text = text.translate(final_map) # normalize final forms if lower: text = text.lower() if strip_punct: text = PUNCT_RE.sub(' ', text) # optional: collapse repeated whitespace text = re.sub(r'\s+', ' ', text).strip() return text

    Conservative clitic stripping: remove single-letter Hebrew prefixes if token len > 3 after stripping

    PREFIXES = ('ב','כ','ל','ו','מ','ש','ה') # careful heuristics def strip_prefixes_token(tok): if len(tok) > 3 and tok[0] in PREFIXES: return tok[1:] return tok

    def tokenize_and_normalize(text): text = normalize_hebrew(text, strip_diacritics=True, strip_punct=False) # naive tokenization on whitespace + punctuation toks = re.findall(r'[\u0590-\u05ff]+|\w+', text) toks = [strip_prefixes_token(t) for t in toks] return ' '.join(toks)

    ---------- Example inputs (replace these) ----------

    chat_contributions: list of dicts {'participant': 'A', 'text': '...'}

    Example minimal:

    chat_contributions = [{'participant':'A','text':'שלום, מה נשמע?'}, ...]

    chat_contributions = [] # <-- fill in

    if not chat_contributions: raise SystemExit("Put your chat_contributions list into the script and re-run.")

    texts = [tokenize_and_normalize(c['text']) for c in chat_contributions]

    ---------- Internal expansion: several strategies ----------

    1) Sliding-window documents

    window = 5 sliding_docs = [] for i in range(0, len(texts), window): block = ' '.join(texts[i:i+window]) sliding_docs.append(block)

    2) Keyword-driven documents: collect top keywords from chats and gather messages containing them

    Build a quick word-frequency / tf-idf proxy to pick keywords

    all_text = ' '.join(texts) freq = Counter(all_text.split()) top_keywords = [w for w,count in freq.most_common(200) if len(w) > 2][:100]

    kw_docs = [] for kw in top_keywords: collected = [] for t in texts: if kw in t.split(): collected.append(t) if collected: # join and chunk if too large joined = ' '.join(collected) # chunk into ~300 token chunks toks = joined.split() for i in range(0, len(toks), 300): kw_docs.append(' '.join(toks[i:i+300]))

    3) participant-aggregates (chunked)

    participant_texts = defaultdict(list) for c, t in zip(chat_contributions, texts): participant_texts[c['participant']].append(t) part_docs = [] for p, segs in participant_texts.items(): joined = ' '.join(segs) toks = joined.split() for i in range(0, len(toks), 400): part_docs.append(' '.join(toks[i:i+400]))

    Combine expanded corpus

    expanded_corpus = sliding_docs + kw_docs + part_docs print(f"Expanded corpus size (docs): {len(expanded_corpus)}")

    ---------- Build TF-IDF with word + char n-grams ----------

    vectorizer = TfidfVectorizer( analyzer='word', ngram_range=(1,2), min_df=2, # tune low for small corpora sublinear_tf=True, max_df=0.95 )

    Fit on expanded_corpus

    X_tfidf = vectorizer.fit_transform(expanded_corpus)

    Also create a combined vectorizer that uses char n-grams by unioning features (simple approach)

    char_vectorizer = TfidfVectorizer(analyzer='char_wb', ngram_range=(3,5), min_df=2) X_char = char_vectorizer.fit_transform(expanded_corpus)

    Simple feature concat (horizontal stacking)

    from scipy.sparse import hstack X_combined = hstack([X_tfidf, X_char])

    ---------- Truncated SVD (LSA) ----------

    n_components = 150 # start smaller; increase to 300 if you have many docs/tokens svd = TruncatedSVD(n_components=n_components, n_iter=12, random_state=0) X_lsa = svd.fit_transform(X_combined) X_lsa = normalize(X_lsa, axis=1)

    ---------- Project chat contributions into same space ----------

    transform contributions via the two vectorizers and svd (note: must transform each separately then stack)

    C_tfidf = vectorizer.transform(texts) C_char = char_vectorizer.transform(texts) C_combined = hstack([C_tfidf, C_char]) C_lsa_raw = svd.transform(C_combined) C_lsa = normalize(C_lsa_raw, axis=1)

    ---------- Given / New (simple projection method) ----------

    def given_new_scores(C): n, d = C.shape given = np.zeros(n) new = np.zeros(n) basis = np.zeros((0, d)) for t in range(n): if t == 0: given[t] = 0.0 new[t] = 1.0 else: # form matrix of previous vectors Prev = C[:t].T # dim x t # compute orthonormal basis via QR Q, _ = np.linalg.qr(Prev) Qt = Q[:, :t] dvec = C[t] g = Qt.dot(Qt.T.dot(dvec)) nvec = dvec - g ng = np.linalg.norm(g) nn = np.linalg.norm(nvec) denom = ng + nn n_val = nn / denom if denom>0 else 0.0 given[t] = 1 - n_val new[t] = n_val return given, new

    given, new = given_new_scores(C_lsa_raw)

    print("Example given/new (first 10):") for i in range(min(10, len(given))): print(i, given[i], new[i])

    Now you have LSA vectors and given/new per-turn. Compute cross-cohesion like in the paper using cosine on C_lsa.

    1. Practical tips & pitfalls

    If the corpus is tiny, aggressive min_df (like min_df=2) or char n-grams will help; but you may still have a noisy space. Use participant or topic aggregation to increase document sizes.

    Don’t over-strip prefixes — overly aggressive stemming/clitic removal can destroy meanings. Use conservative heuristics first, inspect output, then iterate.

    Dimension tuning: measure stability of pairwise similarities as you increase k and pick where gains flatten.

    If you later can use external data: add Hebrew Wikipedia, news corpora, or parallel corpora — they’re very helpful. But the pseudo-document techniques above often get you surprisingly far.

    1. Next steps — what I can do right now

    If you paste your code (you said you’d send it) I will adapt it to the Hebrew preprocessing + internal expansion methods above and produce runnable code for your dataset.

    Or paste a small sample of your data (10–50 turns) and I’ll run the pipeline above on it and show the LSA sanity checks (nearest neighbors, example given/new, sample cohesion).

    Which would you like me to do: (A) adapt your code now, or (B) run the pipeline on a small sample you paste here?

  5. User

    I don't need this step of Preprocesses Hebrew text since I have a model that lemmatizes the text and I have a list of stopwords. I will load the data already lemmatized

  6. Assistant

    Perfect — since your data is already lemmatized and you have a stopword list, I’ll skip preprocessing and give you a compact, ready-to-run pipeline that:

    builds an internal expanded corpus (no external data),

    produces an LSA (TruncatedSVD),

    projects chat turns,

    computes given/new, communication density, and cross-cohesion / responsivity,

    includes quick sanity checks (nearest neighbors, dimension sweep).

    I picked conservative defaults that work well for Hebrew short-text datasets but tuned so you can run this immediately. Paste your lemmatized turns into chat_contributions and your Hebrew stopwords into stopwords_list. Pipeline — copy / paste and run pythonCopyEdit# Requires: pip install scikit-learn numpy scipy import numpy as np from collections import defaultdict, Counter from sklearn.feature_extraction.text import TfidfVectorizer from sklearn.decomposition import TruncatedSVD from sklearn.preprocessing import normalize from scipy.sparse import hstack import math

    ############### ---------- INPUTS (replace) ----------

    chat_contributions: list of dicts {'participant': 'A', 'text': 'lemmatized hebrew text here'}

    stopwords_list: Python list of stopwords (lemmatized forms)

    Example:

    chat_contributions = [{'participant':'A','text':'ברוך הבא'}, {'participant':'B','text':'תודה רבה'}]

    stopwords_list = ['ה', 'ו', 'על', ...] # your list

    chat_contributions = [] # <- replace with your data stopwords_list = [] # <- replace with your stopword list (lemmatized)

    if not chat_contributions: raise SystemExit("Please fill chat_contributions with your lemmatized data and re-run.")

    texts = [c['text'] for c in chat_contributions] participants = [c['participant'] for c in chat_contributions]

    ############### ---------- 1) Internal expansion strategies (no external data) ---------- def build_expanded_corpus(texts, participants, top_kw_count=150, window_size=5, chunk_tokens=400): # 1. sliding-window documents (consecutive turns grouped) sliding_docs = [] for i in range(0, len(texts), window_size): sliding_docs.append(' '.join(texts[i:i+window_size]))

    # 2. keyword-driven documents: keywords from internal tf-idf-style frequency
    all_toks = ' '.join(texts).split()
    freq = Counter(all_toks)
    # choose top content words (exclude provided stopwords)
    top_keywords = [w for w,_ in freq.most_common( top_kw_count*3 ) if w not in stopwords_list and len(w)>1]
    top_keywords = top_keywords[:top_kw_count]
    
    kw_docs = []
    for kw in top_keywords:
        hits = [t for t in texts if kw in t.split()]
        if hits:
            joined = ' '.join(hits)
            toks = joined.split()
            for i in range(0, len(toks), chunk_tokens):
                kw_docs.append(' '.join(toks[i:i+chunk_tokens]))
    
    # 3. participant aggregated docs (chunked)
    participant_texts = defaultdict(list)
    for p, t in zip(participants, texts):
        participant_texts[p].append(t)
    part_docs = []
    for p, segs in participant_texts.items():
        joined = ' '.join(segs)
        toks = joined.split()
        for i in range(0, len(toks), chunk_tokens):
            part_docs.append(' '.join(toks[i:i+chunk_tokens]))
    
    # combine and deduplicate short or empty docs
    expanded = [d for d in (sliding_docs + kw_docs + part_docs) if len(d.split())>5]
    # optional dedupe while preserving order
    seen=set(); expanded_uniq=[]
    for d in expanded:
        if d not in seen:
            expanded_uniq.append(d); seen.add(d)
    return expanded_uniq
    

    expanded_corpus = build_expanded_corpus(texts, participants, top_kw_count=120, window_size=5, chunk_tokens=400) print(f"Expanded corpus documents: {len(expanded_corpus)}")

    ############### ---------- 2) Vectorization (word + char n-grams) ----------

    Word n-grams + your stopwords

    word_vectorizer = TfidfVectorizer( analyzer='word', ngram_range=(1,2), stop_words=stopwords_list if stopwords_list else None, min_df=2, # tune down if corpus very small sublinear_tf=True, max_df=0.95 ) X_word = word_vectorizer.fit_transform(expanded_corpus)

    Char n-grams to help with morphology (char_wb works well for languages with clitics)

    char_vectorizer = TfidfVectorizer( analyzer='char_wb', ngram_range=(3,5), min_df=2 ) X_char = char_vectorizer.fit_transform(expanded_corpus)

    combine feature sets

    X_combined = hstack([X_word, X_char])

    ############### ---------- 3) Truncated SVD (LSA) ----------

    Choose n_components relative to data size:

    vocab_size = X_combined.shape[1] n_docs = X_combined.shape[0]

    heuristic default:

    n_components = 300

    If corpus small, reduce dims

    if n_docs < 500 or vocab_size < 5000: n_components = 150 if n_docs < 200: n_components = 100 n_components = min(n_components, n_docs-1) if n_docs>1 else 1

    print(f"Using n_components={n_components} (documents={n_docs}, vocab≈{vocab_size})")

    svd = TruncatedSVD(n_components=n_components, n_iter=12, random_state=0) X_lsa = svd.fit_transform(X_combined) # (n_docs, n_components) X_lsa_norm = normalize(X_lsa, axis=1)

    ############### ---------- 4) Project chat contributions into LSA ---------- C_word = word_vectorizer.transform(texts) C_char = char_vectorizer.transform(texts) C_combined = hstack([C_word, C_char]) C_lsa_raw = svd.transform(C_combined) # raw projected vectors C_lsa = normalize(C_lsa_raw, axis=1) # normalized for cosine

    ############### ---------- 5) Given / New decomposition ---------- def given_new_scores(C): # C: (n_turns, dim) raw (not necessary normalized) — we use raw for magnitude math n, d = C.shape given = np.zeros(n) new = np.zeros(n) for t in range(n): if t == 0: given[t] = 0.0 new[t] = 1.0 continue Prev = C[:t].T # dim x t # orthonormal basis via QR (works even if t>dim) try: Q, _ = np.linalg.qr(Prev) except Exception: # fallback small-SVD approach U,, = np.linalg.svd(Prev, full_matrices=False) Q = U Qt = Q[:, :min(Q.shape[1], t)] dvec = C[t] g = Qt.dot(Qt.T.dot(dvec)) nvec = dvec - g ng = np.linalg.norm(g) nn = np.linalg.norm(nvec) denom = ng + nn n_val = nn / denom if denom > 0 else 0.0 given[t] = 1.0 - n_val new[t] = n_val return given, new

    given_scores, new_scores = given_new_scores(C_lsa_raw)

    ############### ---------- 6) Communication density ---------- word_counts = np.array([len(t.split()) for t in texts]) norms = np.linalg.norm(C_lsa_raw, axis=1) comm_density = np.where(word_counts>0, norms / word_counts, 0.0)

    ############### ---------- 7) Cross-cohesion & responsivity ----------

    precompute cosine similarity matrix using normalized vectors (dot product)

    sim_matrix = np.dot(C_lsa, C_lsa.T) # (n_turns, n_turns)

    unique_parts = sorted(set(participants)) part_idx = {p:i for i,p in enumerate(unique_parts)} k = len(unique_parts) n = len(participants)

    def cross_cohesion_tau(tau): num = np.zeros((k,k)) denom = np.zeros((k,k)) for t in range(tau, n): initiator = participants[t - tau] responder = participants[t] i_idx = part_idx[initiator] r_idx = part_idx[responder] s = float(sim_matrix[t - tau, t]) num[r_idx, i_idx] += s denom[r_idx, i_idx] += 1 xi = np.where(denom>0, num/denom, 0.0) return xi, denom

    compute average responsivity across tau=1..w

    w = 3 R_sum = np.zeros((k,k)) for tau in range(1, w+1): xi, denom = cross_cohesion_tau(tau) R_sum += xi R_w = R_sum / w

    per-participant metrics

    overall_responsivity = np.array([np.mean([R_w[i,j] for j in range(k) if j!=i]) if k>1 else 0.0 for i in range(k)]) social_impact = np.array([np.mean([R_w[j,i] for j in range(k) if j!=i]) if k>1 else 0.0 for i in range(k)]) internal_cohesion = np.diag(R_w)

    ############### ---------- 8) Quick output structuring ---------- results_by_participant = {} for p, idx in part_idx.items(): idxs = [i for i,pp in enumerate(participants) if pp==p] results_by_participant[p] = { 'n_turns': len(idxs), 'avg_given': float(np.mean(given_scores[idxs])) if idxs else 0.0, 'avg_new': float(np.mean(new_scores[idxs])) if idxs else 0.0, 'avg_comm_density': float(np.mean(comm_density[idxs])) if idxs else 0.0, 'overall_responsivity': float(overall_responsivity[idx]), 'social_impact': float(social_impact[idx]), 'internal_cohesion': float(internal_cohesion[idx]) }

    Print summary for quick inspection

    print("Participants summary (top-level):") for p, vals in results_by_participant.items(): print(p, vals)

    ############### ---------- 9) Sanity checks / utilities ----------

    Nearest neighbors in LSA for a query token (uses word_vectorizer vocab)

    def nearest_terms(term, topn=10): if term not in word_vectorizer.vocabulary_: return [] # get term vector in original SVD space: approximate via term -> column in X_word term_idx = word_vectorizer.vocabulary_[term] # Build a one-hot column vector for term across feature space: # We can approximate term vector by extracting svd.components_ weights for feature index # svd.components_ shape: (n_components, n_features) feat_vec = np.zeros(word_vectorizer.transform([term]).shape[1] + char_vectorizer.transform([term]).shape[1]) # easier approach: find docs containing term and average their LSA vectors doc_ids = [i for i,d in enumerate(expanded_corpus) if term in d.split()] if not doc_ids: return [] avg_vec = np.mean(X_lsa_norm[doc_ids], axis=0) sims = np.dot(X_lsa_norm, avg_vec) top_idx = np.argsort(-sims)[:topn] # convert top docs back to most frequent terms in those docs as proxies candidates = Counter() for di in top_idx: for tok in expanded_corpus[di].split(): if tok not in stopwords_list and len(tok)>1: candidates[tok]+=1 return [t for t,_ in candidates.most_common(topn)]

    example nearest terms for a few frequent words (inspect manually)

    sample_terms = [w for w,_ in Counter(' '.join(texts).split()).most_common(10) if w not in stopwords_list][:5] nn = {t: nearest_terms(t, topn=8) for t in sample_terms} print("Nearest-term proxies (sample):") print(nn)

    Short notes, recommendations and defaults I used

    No preprocessing: code expects lemmatized texts and stopwords; it does conservative min_df=2 for small corpora — you can lower to 1 if needed (but careful: very noisy).

    Expansion: I combine sliding windows, keyword-aggregates and participant-concat. This often produces a well-shaped term–document matrix without Wikipedia.

    n_components: default auto-scaling: 100–300 depending on n_docs. If you have many documents (≥1k) use 300; otherwise 100–150 is more stable.

    Char n-grams: included to reduce sparsity because Hebrew morphology is rich — they help even when you lemmatize.

    Given/New: uses QR on the matrix of previous vectors (stable and interpretable). Uses raw SVD-projected vectors for magnitude computations.

    Responsivity: averaged across τ=1..w where w=3 by default — change w to a longer window if conversations have longer lags.

    Sanity checks: inspect nearest-term proxies, participant summaries, and sample high/low newness contributions to validate.

    If you want immediate extra help (I will do it right now)

    I can adapt the script to accept your data format (if you paste a small sample, e.g., 20–100 turns) and run the pipeline with the outputs (nearest neighbors, a small table of participant metrics, sample high-newness turns). Paste that sample and your stopword list and I’ll run the pipeline on it in this session.