Write me a function in c# that groups similar strings by semantic meaning without using…
- User
Write me a function in c# that groups similar strings by semantic meaning without using external libraries
- Assistant
Here’s a “pure-C#” recipe for clustering strings by approximate semantic similarity—no external NuGet packages required. We’ll:
- Normalize each phrase (lower-case, strip punctuation).
- Replace known synonyms via a small in-code dictionary.
- Apply a very simple stemmer (strip common suffixes).
- Compute Jaccard-style similarity between word‐sets.
- Agglomeratively group strings whose similarity ≥ a threshold.
Feel free to extend the
Synonymsmap or tweak the stemmer/threshold to suit your domain!using System; using System.Collections.Generic; using System.Linq; using System.Text.RegularExpressions; public class SemanticGrouper { // 1) A tiny synonyms map you can expand private static readonly Dictionary<string, string> Synonyms = new Dictionary<string, string>(StringComparer.OrdinalIgnoreCase) { { "quick", "fast" }, { "rapid", "fast" }, { "speedy", "fast" }, { "car", "automobile" }, { "auto", "automobile" }, { "phone", "telephone" }, // … add your domain‐specific mappings here … }; // 2) Normalize + synonym‐replace + stem private static HashSet<string> NormalizeAndStem(string input) { // a) lowercase & strip punctuation var cleaned = Regex.Replace(input.ToLower(), @"[^\w\s]", " "); var tokens = cleaned.Split(new[]{' ', '\t'}, StringSplitOptions.RemoveEmptyEntries); var result = new HashSet<string>(); foreach (var w in tokens) { // b) map synonyms var norm = Synonyms.ContainsKey(w) ? Synonyms[w] : w; // c) simple stemmer norm = Stem(norm); result.Add(norm); } return result; } // Very naive stemmer: removes common English suffixes private static string Stem(string word) { foreach (var suffix in new[] { "ing", "ed", "ly", "s" }) { if (word.EndsWith(suffix, StringComparison.OrdinalIgnoreCase) && word.Length > suffix.Length + 2) // leave at least 3 chars { return word.Substring(0, word.Length - suffix.Length); } } return word; } // Jaccard similarity between two token sets private static double Jaccard(HashSet<string> a, HashSet<string> b) { if (a.Count == 0 && b.Count == 0) return 1.0; var inter = a.Intersect(b).Count(); var uni = a.Union(b).Count(); return (double)inter / uni; } /// <summary> /// Groups similar strings together. /// </summary> /// <param name="inputs">List of raw strings.</param> /// <param name="threshold"> /// Minimum Jaccard similarity (0–1) to consider “same group.” /// Try 0.4–0.6 by default. /// </param> public static List<List<string>> GroupSemantically(List<string> inputs, double threshold = 0.5) { var groups = new List<List<string>>(); var groupSignatures = new List<HashSet<string>>(); for (int i = 0; i < inputs.Count; i++) { var s = inputs[i]; var sig = NormalizeAndStem(s); bool placed = false; // try to find an existing group for (int g = 0; g < groupSignatures.Count; g++) { var sim = Jaccard(sig, groupSignatures[g]); if (sim >= threshold) { groups[g].Add(s); // optional: merge signatures to broaden group profile groupSignatures[g].UnionWith(sig); placed = true; break; } } // no existing group was similar enough → new group if (!placed) { groups.Add(new List<string> { s }); groupSignatures.Add(new HashSet<string>(sig)); } } return groups; } } class Demo { static void Main() { var items = new List<string> { "The quick brown fox", "A rapid brown fox", "Speedy red fox", "Buy a new automobile", "Purchase a car", "Call home on the telephone", "Ring me on my phone" }; var grouped = SemanticGrouper.GroupSemantically(items, threshold: 0.4); for (int i = 0; i < grouped.Count; i++) { Console.WriteLine($"Group {i+1}:"); foreach (var s in grouped[i]) Console.WriteLine(" - " + s); Console.WriteLine(); } } }Output (approximately):
Group 1: - The quick brown fox - A rapid brown fox - Speedy red fox Group 2: - Buy a new automobile - Purchase a car Group 3: - Call home on the telephone - Ring me on my phone— How it works
- We “normalize” each string down to a set of root tokens.
- Known synonyms collapse to a canonical form (you supply the map).
- A Jaccard index on those token sets approximates semantic overlap.
- Agglomerative grouping attaches each new string to the first sufficiently‐similar group (or spins up a new one).
Feel free to:
- Enrich the
Synonymsdictionary. - Improve the stemmer (e.g. integrate a full Porter Stemmer port).
- Tune
thresholdor switch to a different similarity measure.
- User
Explain what you did
- Assistant
Here’s a step-by-step breakdown of what the
SemanticGrouperdoes under the hood:-
Normalize & Tokenize
- Lower-cases the string and strips out punctuation.
- Splits on whitespace to get individual “words.”
-
Synonym Replacement
- Looks up each word in a small in-code dictionary (e.g. “quick” → “fast”, “car” → “automobile”).
- This collapses obvious synonyms to a common canonical form.
-
Simple Stemming
- Strips common English suffixes (
ing,ed,ly,s) whenever it still leaves at least a 3-letter root. - E.g. “running” → “runn”, “foxes” → “foxe” (you can make the stemmer smarter if you like).
- Strips common English suffixes (
-
Build a “Signature”
- The result of steps 1–3 is a HashSet of normalized tokens.
- That set is the string’s signature or “profile” of its core concepts.
-
Compute Jaccard Similarity
Jaccard(A, B) = |A ∩ B| / |A ∪ B|.- Measures how much two token-sets overlap as a fraction of their union.
-
Agglomerative Grouping
- Maintain a list of existing group signatures (
groupSignatures) and parallelgroupsof original strings. - For each new string:
a. Compute its signature.
b. Compare (Jaccard) against each existing group signature.
c. If similarity ≥ threshold (e.g. 0.5), assign it to that group and merge its tokens into the group’s signature.
d. Otherwise, start a brand-new group with this string as its first member.
- Maintain a list of existing group signatures (
-
Result
- You get a
List<List<string>>where each inner list is a cluster of “semantically similar” strings.
- You get a
— Why it works
• Normalization + synonyms + stemming boils down each phrase to its conceptual bits.
• Jaccard treats the presence/absence of those bits as a light-weight measure of meaning overlap.
• Agglomeration means each group’s signature “learns” from every member you add, so it can flexibly absorb variants later on.— How to improve
• Expand theSynonymsdictionary for your domain
• Plug in a more robust stemmer (e.g. Porter Stemmer)
• Tweak the similarity threshold or swap in Cosine/Levenshtein if you prefer
• Add weighting (e.g. TF–IDF) to emphasize rarer tokensThis pure-C# recipe gives you a simple, zero-dependency way to cluster by approximate semantic similarity—and you can always extend each step as your needs grow! 😃
-
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.