Given a group conversation, I want a measure to quantify the newness of a single…
- User
Given a group conversation, I want a measure to quantify the newness of a single contribution c.
To do so, I will use a sentence transformer to compute the semantic similarity between c and all earlier contributions separately. The similarity I'm interested in is the maximum of these.
Now, for the Newness measure, I want higher to be better. How can I get this?
- Assistant
Got it—you want a Newness score where higher means the contribution ccc is more novel compared to everything said before it. Right now:
You compute semantic similarity sim(c,ei)\text{sim}(c, e_i)sim(c,ei) between ccc and each earlier contribution eie_iei.
You take the maximum similarity S=maxisim(c,ei)S = \max_i \text{sim}(c, e_i)S=maxisim(c,ei).
But:
High SSS → ccc is similar to something earlier → less new.
Low SSS → ccc is different → more new.
So to make higher = better for newness, you just invert the similarity: Newness(c)=1−S\text{Newness}(c) = 1 - SNewness(c)=1−S assuming similarity is in [0,1][0,1][0,1].
If similarity = 1 (identical), newness = 0 → no novelty.
If similarity = 0 (totally different), newness = 1 → max novelty.
If your sentence transformer gives cosine similarity in [−1,1][-1,1][−1,1], you can first normalize: Snorm=S+12⇒Newness(c)=1−Snorm.S_{norm} = \frac{S + 1}{2} \quad \Rightarrow \quad \text{Newness}(c) = 1 - S_{norm}.Snorm=2S+1⇒Newness(c)=1−Snorm.
If you'd like, I can also show you how to make the measure less sensitive to small differences so it only rewards truly novel contributions. Do you want me to show you that version too?
- User
Yes. I use Cosine Similarity.
- Assistant
Perfect — cosine similarity gives values in [−1,1][-1,1][−1,1], so the normalization I mentioned works directly. But if you want less sensitivity so that only truly different contributions get high newness scores, you can apply a nonlinear transformation after normalizing. Here’s the step-by-step approach:
-
Normalize the cosine similarity Snorm=S+12so Snorm∈[0,1]S_{norm} = \frac{S + 1}{2} \quad \text{so } S_{norm} \in [0,1]Snorm=2S+1so Snorm∈[0,1]
-
Define basic Newness Newness(c)=1−Snorm\text{Newness}(c) = 1 - S_{norm}Newness(c)=1−Snorm
-
Add nonlinear scaling to reduce sensitivity You can use a power function or a sigmoid so small differences in similarity near 1 don’t drastically change newness:
Power scaling:
Newness(c)=(1−Snorm)α\text{Newness}(c) = (1 - S_{norm})^\alphaNewness(c)=(1−Snorm)α with α>1\alpha > 1α>1.
Larger α\alphaα → you only get high newness if similarity is very low.
Sigmoid scaling:
Newness(c)=11+ek(Snorm−t)\text{Newness}(c) = \frac{1}{1 + e^{k(S_{norm}-t)}}Newness(c)=1+ek(Snorm−t)1
ttt = similarity threshold where novelty switches from high to low
kkk = steepness parameter
- Recommendation
Start with α=2\alpha = 2α=2 if you want diminishing newness unless the similarity is very low.
Or use the sigmoid if you want a soft threshold at, say, t=0.7t = 0.7t=0.7.
I can make a quick visualization comparing these methods so you can pick parameters visually. Do you want me to generate that?
-
- User
Here is my implementation. The values I get are kind of skewed, with a mean=0.08 and std=0.15.
def compute_newness_cache(self): """ For each contribution in the discussion, compute and cache its newness score. This is done once during initialization to avoid redundant calculations later. The newness of a contribution at time t is defined by computing the semantic similarity of the current contribution with each previous contribution, and then taking the maximum similarity. The similarity is then normalized to [0, 1] using: S_norm = (sim_max + 1)/2 The newness score is then: n(c_t) = (1-S_norm)^2 :return: """ contributions = self.data['text'].tolist() for i, cont in enumerate(contributions): if i == 0: # First contribution has no previous contributions, assign max newness newness = 1.0 else: max_sim = -1.0 for j in range(i): sim = self.calc_cosine_similarity(i + 1, j + 1) if sim > max_sim: max_sim = sim # Normalize similarity to [0, 1] sim_norm = (max_sim + 1) / 2 newness = (1 - sim_norm) ** 2 # square to emphasize high newness self.newness_cache[i] = newnessWhat can I do?
- Assistant
Ah, I see the issue now. Your implementation is mathematically fine, but the distribution of cosine similarities in real conversations is often skewed toward high similarity (people tend to talk about the same topic). When you square (1−Snorm)(1 - S_{norm})(1−Snorm), you make this skew even more extreme, so most values collapse toward 0 and only a few get high newness. That explains your mean ≈ 0.08 and std ≈ 0.15. Here are a few ways to fix or rescale the newness metric so it spreads out better:
- Try a gentler transformation Instead of squaring: pythonCopy codenewness = 1 - sim_norm
or use (1−sim_norm)α(1 - sim_norm)^{\alpha}(1−sim_norm)α with 0<α<10< \alpha < 10<α<1 for a concave transform: pythonCopy codenewness = (1 - sim_norm)**0.5 # sqrt makes it less skewed
α < 1 spreads values apart, α > 1 compresses them.
Right now α=2 compresses everything near 0, which causes the skew.
- Apply z-score normalization afterwards After computing raw newness, rescale to mean 0 and std 1: pythonCopy codeimport numpy as np
raw_newness = np.array(list(self.newness_cache.values())) z_newness = (raw_newness - raw_newness.mean()) / raw_newness.std()
Then you can shift & scale to [0,1] if you want a bounded metric:
pythonCopy codescaled_newness = (z_newness - z_newness.min()) / (z_newness.max() - z_newness.min())
- Use a percentile rank instead of raw values This is nice if you care about relative novelty: pythonCopy codefrom scipy.stats import rankdata
ranks = rankdata(raw_newness, method='average') percentile_newness = ranks / len(raw_newness)
Now the most novel = 1, least novel = 0, evenly spread out.
- Optional: Threshold or sigmoid scaling If most contributions are somewhat similar and you want only very different ones to pop, try: pythonCopy codeimport numpy as np
k = 10 # steepness t = 0.7 # similarity threshold newness = 1 / (1 + np.exp(k*(sim_norm - t)))
This gives you control over where novelty drops off sharply.
If you want, I can plot your similarity distribution and show you visually how each transformation affects it, so you can pick the right scaling. Do you want me to generate that plot?
- User
yes
- Assistant
Here’s the visualization:
Linear (1 - sim_norm) spreads values more evenly than squaring.
Squared ((1 - sim_norm)²) compresses most values toward 0, which matches your current skewed distribution.
Square root ((1 - sim_norm)^0.5) spreads out low-newness contributions a bit more than linear, giving a middle ground.
Sigmoid sharply separates low vs. high similarity, so only very novel contributions score high.
From this, if your goal is more interpretable and less skewed newness, either the linear or sqrt transformation is usually the best starting point. If you want, I can suggest a modified version of your compute_newness_cache using these scalings and optionally percentile ranking so that your scores are nicely distributed between 0 and 1.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.