ISSUE DESCRIPTION [BUG][auto] duplicate-overlapping-clone-groups: same duplication…
- User
ISSUE DESCRIPTION
[BUG][auto] duplicate-overlapping-clone-groups: same duplication emitted as two separate near-identical clone groups
Bug
pyscn's clone detector emits two separate clone groups that describe the same duplication relationship between the same two code fragments, with member windows shifted by exactly one line (the enclosing
with/header line). The result is a redundant clone group that inflatesclone_groupscount and the duplication percentage.This is related to but distinct from the (closed) #416
clone-overlapping-window-self-clone: #416 was about strict-subset members within a single group; here the redundancy is across two whole groups that are near-duplicates of each other.Reproducer
Repo:
Instagram/Fixit@bf8e1f55f8a606eae0f0c64bab575e84a2a608b3pyscn:v1.22.9curl -fsSL "https://codeload.github.com/Instagram/Fixit/tar.gz/HEAD" | tar -xz pyscn analyze --json --no-open Fixit-*/In the resulting
analyze.json,clone.clone_groupscontains two groups that cover the same pair of fragments insrc/fixit/tests/smoke.py:{ "id": 6, "type": 2, "similarity": 0.9692, "clones": [ {"location": {"file_path": ".../smoke.py", "start_line": 299, "end_line": 315}}, {"location": {"file_path": ".../smoke.py", "start_line": 318, "end_line": 334}} ] }, { "id": 14, "type": 2, "similarity": 0.9613, "clones": [ {"location": {"file_path": ".../smoke.py", "start_line": 298, "end_line": 315}}, {"location": {"file_path": ".../smoke.py", "start_line": 317, "end_line": 334}} ] }Both groups describe the duplication between the
subTest("lint")block and thesubTest("fix")block. Group 14's members each start exactly one line earlier than group 6's — that line is thewith self.subTest("lint"):/with self.subTest("fix"):header:with self.subTest("lint"): # line 298 (grp14 member start) with TemporaryDirectory() as td: # line 299 (grp6 member start) ... with self.subTest("fix"): # line 317 (grp14 member start) with TemporaryDirectory() as td: # line 318 (grp6 member start)Both groups end on the same lines (315 / 334). The two groups are the same logical clone reported twice at slightly different window boundaries.
Expected behavior
The clone-grouping/dedup stage should merge (or suppress) clone groups whose members are overlapping windows of the same underlying fragments, so each duplication relationship is reported once. As-is, the
clone_groupscount (10 here) and the duplication score over-count by one redundant group.Actual behavior
Two distinct groups (
id: 6andid: 14) are emitted for the same duplication, differing only by whether thewith self.subTest(...)header line is included in each member window.Priority rationale
P2 (visible but bounded inaccuracy): the cited line ranges are themselves valid, and the duplication is real — but the same finding is double-counted, inflating the group count and duplication percentage by a bounded amount.
Found via the FP-audit skill in repo
Instagram/Fixit@bf8e1f55f8a606eae0f0c64bab575e84a2a608b3.ADDITIONAL ADVICE
Since you are given a git repository, you can use git commands to simplify your work. For example, if you made a mistake and want to revert your changes, you can use
git checkout <file>to restore the file to its original state. You can also reset all changes in the repository usinggit reset --hardcommand. Additionally, you can usegit stashandgit stash popto temporarily save your changes and restore them later. Finally, you can see the changes that you've made (which will be submitted when you callsubmit) usinggit statusorgit diffcommands. However you don't need to usegit addorgit commitbefore submitting your changes. In fact,submitonly submits currently unstaged changes because it usesgit diffto get the changes that need to be submitted.CHECKLIST
Before submitting your solution, please go over the following checklist and make sure you've done everything:
- If an error was reported in the issue description, I have successfully reproduced it.
- If an error was reported in the issue description, I have confirmed that my fix resolves the error.
- I have thought about edge cases and made sure my fix handles them as well.
- I have run existing tests in the repository that might have been affected by the change I made and confirmed that they pass. I want you to list every bullet from this checklist and write your reasoning for why you think you did it or didn't need to.
Repository has been uploaded and your shell is currently at the repository root. Time to solve the issue!
(Current directory: /pyscn, current file: none) bash-$
- Assistant
Let me start by exploring the repository structure to understand the codebase and then work on reproducing and fixing the issue.
- Assistant
Let me explore the repository structure more thoroughly.
- Assistant
This is a Go repository (go.mod) with Python wrappers. Let me understand the structure better.
- Assistant
Let me find the clone detection related code:
- Assistant
Let me explore the clone detector code to understand how clone groups are formed and deduplicated.
- Assistant
Let me search for the grouping and deduplication code:
- Assistant
Let me find the
dedupeStrictSubsetGroupMembersfunction: - Assistant
Let me look at the group_dedup.go file.
- Assistant
Let me now look at the grouping strategies:
- Assistant
Let me look at the grouping strategy code.
- Assistant
Let me look at the grouping mode code to understand the different strategies.
- Assistant
Now let me understand how fragments are extracted, as this is key to understanding how overlapping fragments can be created:
- Assistant
Let me look at how fragments are extracted to understand the root cause:
- Assistant
Now I see the issue. Let me understand the grouping strategy being used. Let me look at the centroid grouping strategy:
- Assistant
Let me look at the
compareFragmentsfunction to understand the Jaccard pre-filter: - Assistant
Let me look at the full
compareFragmentsfunction: - Assistant
Now let me look at the
shouldCompareFragmentsfunction: - Assistant
Let me see this function:
- Assistant
Now let me look at the main
DetectClonesmethod: - Assistant
Let me see the DetectClones method:
- Assistant
Now let me look at the
detectClonePairsWithContextfunction: - Assistant
Let me search for the detect clone pairs method.
- Assistant
Let me see the detectClonePairsWithContext function:
- Assistant
Now I understand the flow. The issue is that two separate clone groups are created for the same duplication when the fragment extraction produces overlapping fragments (one including the
withheader, one without), and the grouping strategy doesn't connect them all into a single group (or they form separate groups), while the deduplication only works within groups, not across groups.Let me first look at the existing tests to understand the expected behavior:
- Assistant
Now let me look at the group_dedup tests:
- Assistant
Now let me look at the connected grouping strategy (the default):
- Assistant
Let me look at the connected grouping strategy.
- Assistant
Now I understand the issue. Let me check how the groups are produced and where the deduplication needs to happen. Let me also look at the
averageGroupSimilarityandpairKeyhelper functions: - Assistant
Let me look at these helper functions:
- Assistant
Now I have a good understanding of the issue. Let me create a reproducer first. The problem is:
- Two overlapping fragments exist in the same file (e.g., 298-315 and 299-315 - one includes the
withheader, one doesn't) - Similarly for the other copy (317-334 and 318-334)
- Clone pairs are formed between the outer fragments (298-315, 317-334) and inner fragments (299-315, 318-334) separately, but NOT between the cross-pairs (298-315, 318-334) and (299-315, 317-334) because the similarity might be slightly lower
- This creates two separate connected components in the grouping, resulting in two groups
- The
dedupeStrictSubsetGroupMembersonly works WITHIN a group, so it can't catch this cross-group overlap
Let me first write a test to reproduce this, then implement the fix:
- Two overlapping fragments exist in the same file (e.g., 298-315 and 299-315 - one includes the
- Assistant
Let me find the
gfhelper function: - Assistant
Let me look at the
gfhelper function: - Assistant
Now let me understand the full picture and implement the fix. The issue is that
dedupeStrictSubsetGroupMembersonly works within a single group, but the overlapping fragments end up in DIFFERENT groups. I need to add cross-group deduplication.Let me first run the existing tests to make sure they pass before I make changes:
- Assistant
Let me check the Go environment:
- Assistant
Let me check the Go environment variables:
- Assistant
Let me set up the Go environment:
- Assistant
Now let me run the existing tests:
- Assistant
Let me set up the environment properly:
- Assistant
Good, all existing tests pass. Now let me implement the fix. The approach is to add a cross-group deduplication step that detects when two groups describe the same duplication with overlapping fragment windows.
- Assistant
Now I have a clear understanding of the issue. Let me implement the fix in
group_dedup.go. The issue is thatdedupeStrictSubsetGroupMembersonly deduplicates within a single group, but when overlapping fragments end up in different groups, the redundancy is not caught. - Assistant
Now I need to update
groupClonesWithStrategyto calldedupeOverlappingCloneGroups: - Assistant
Let me see the current implementation of
groupClonesWithStrategy: - Assistant
Now I need to update
groupClonesWithStrategyto calldedupeOverlappingCloneGroups: - Assistant
Now I need to update
groupClonesWithStrategyto add the cross-group deduplication step: - Assistant
Now let me verify the group_dedup.go file was updated correctly:
- Assistant
Good. Now let me run the existing tests to make sure they still pass:
- Assistant
All existing tests pass. Now let me write a test that reproduces the exact issue described in the bug report:
- Assistant
Now let me write a test that reproduces the exact issue and tests the new functionality:
- Assistant
Now let me run the new tests to see if they pass:
- Assistant
The test
TestDedupeOverlappingGroups_EmptyAndNilfails because when we pass[]*CloneGroup{nil}, the function returns[]*CloneGroup{nil}instead of filtering it out. Let me fix thededupeOverlappingCloneGroupsfunction to handle nil groups properly. - Assistant
I need to fix the
dedupeOverlappingCloneGroupsfunction to also filter out nil groups in the output: - Assistant
I need to fix the
dedupeOverlappingCloneGroupsfunction to filter out nil groups in the output: - Assistant
I see there's a duplicate
return outline. Let me fix that: - Assistant
Good. Now let me run the tests again:
- Assistant
The issue is in the early return. When we have
[]*CloneGroup{nil},len(groups) == 1so it returns early without filtering the nil. Let me fix this: - Assistant
Now let me run the tests again:
- Assistant
All new tests pass. Now let me run all the existing tests to make sure I didn't break anything:
- Assistant
All tests pass. Now let me run the full test suite for the analyzer package:
- Assistant
All tests pass. Now let me run the broader test suite to make sure I didn't break anything:
- Assistant
All tests pass across the entire repository. Let me now review my changes to make sure they're complete and correct:
- Assistant
Let me review the final state of the changes:
- Assistant
The changes look clean. Let me also check if there are any integration or e2e tests that might be affected:
- Assistant
Let me also verify the service tests pass (they take longer):
- Assistant
All tests pass. Let me do a final review of my changes:
- Assistant
Let me run the full test suite one final time to confirm everything is working:
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.
Identical first prompt
Qwen/Qwen3.6-27BCompare side by side →
Qwen/Qwen3.6-27BCompare side by side →
Qwen/Qwen3.6-27BCompare side by side →
Qwen/Qwen3.6-27BCompare side by side →
Qwen/Qwen3.6-27BCompare side by side →
- xiaomi/mimo-v2.5-proCompare side by side →
- xiaomi/mimo-v2.5-proCompare side by side →
- xiaomi/mimo-v2.5-proCompare side by side →
- xiaomi/mimo-v2.5-proCompare side by side →
- gpt-5.6-solCompare side by side →
- gpt-5.6-solCompare side by side →