ISSUE DESCRIPTION [BUG][auto] duplicate-overlapping-clone-groups: same duplication…
- User
ISSUE DESCRIPTION
[BUG][auto] duplicate-overlapping-clone-groups: same duplication emitted as two separate near-identical clone groups
Bug
pyscn's clone detector emits two separate clone groups that describe the same duplication relationship between the same two code fragments, with member windows shifted by exactly one line (the enclosing
with/header line). The result is a redundant clone group that inflatesclone_groupscount and the duplication percentage.This is related to but distinct from the (closed) #416
clone-overlapping-window-self-clone: #416 was about strict-subset members within a single group; here the redundancy is across two whole groups that are near-duplicates of each other.Reproducer
Repo:
Instagram/Fixit@bf8e1f55f8a606eae0f0c64bab575e84a2a608b3pyscn:v1.22.9curl -fsSL "https://codeload.github.com/Instagram/Fixit/tar.gz/HEAD" | tar -xz pyscn analyze --json --no-open Fixit-*/In the resulting
analyze.json,clone.clone_groupscontains two groups that cover the same pair of fragments insrc/fixit/tests/smoke.py:{ "id": 6, "type": 2, "similarity": 0.9692, "clones": [ {"location": {"file_path": ".../smoke.py", "start_line": 299, "end_line": 315}}, {"location": {"file_path": ".../smoke.py", "start_line": 318, "end_line": 334}} ] }, { "id": 14, "type": 2, "similarity": 0.9613, "clones": [ {"location": {"file_path": ".../smoke.py", "start_line": 298, "end_line": 315}}, {"location": {"file_path": ".../smoke.py", "start_line": 317, "end_line": 334}} ] }Both groups describe the duplication between the
subTest("lint")block and thesubTest("fix")block. Group 14's members each start exactly one line earlier than group 6's — that line is thewith self.subTest("lint"):/with self.subTest("fix"):header:with self.subTest("lint"): # line 298 (grp14 member start) with TemporaryDirectory() as td: # line 299 (grp6 member start) ... with self.subTest("fix"): # line 317 (grp14 member start) with TemporaryDirectory() as td: # line 318 (grp6 member start)Both groups end on the same lines (315 / 334). The two groups are the same logical clone reported twice at slightly different window boundaries.
Expected behavior
The clone-grouping/dedup stage should merge (or suppress) clone groups whose members are overlapping windows of the same underlying fragments, so each duplication relationship is reported once. As-is, the
clone_groupscount (10 here) and the duplication score over-count by one redundant group.Actual behavior
Two distinct groups (
id: 6andid: 14) are emitted for the same duplication, differing only by whether thewith self.subTest(...)header line is included in each member window.Priority rationale
P2 (visible but bounded inaccuracy): the cited line ranges are themselves valid, and the duplication is real — but the same finding is double-counted, inflating the group count and duplication percentage by a bounded amount.
Found via the FP-audit skill in repo
Instagram/Fixit@bf8e1f55f8a606eae0f0c64bab575e84a2a608b3.ADDITIONAL ADVICE
Since you are given a git repository, you can use git commands to simplify your work. For example, if you made a mistake and want to revert your changes, you can use
git checkout <file>to restore the file to its original state. You can also reset all changes in the repository usinggit reset --hardcommand. Additionally, you can usegit stashandgit stash popto temporarily save your changes and restore them later. Finally, you can see the changes that you've made (which will be submitted when you callsubmit) usinggit statusorgit diffcommands. However you don't need to usegit addorgit commitbefore submitting your changes. In fact,submitonly submits currently unstaged changes because it usesgit diffto get the changes that need to be submitted.CHECKLIST
Before submitting your solution, please go over the following checklist and make sure you've done everything:
- If an error was reported in the issue description, I have successfully reproduced it.
- If an error was reported in the issue description, I have confirmed that my fix resolves the error.
- I have thought about edge cases and made sure my fix handles them as well.
- I have run existing tests in the repository that might have been affected by the change I made and confirmed that they pass. I want you to list every bullet from this checklist and write your reasoning for why you think you did it or didn't need to.
Repository has been uploaded and your shell is currently at the repository root. Time to solve the issue!
(Current directory: /pyscn, current file: none) bash-$
Assistant
I'll start by exploring the repository structure to understand the codebase and locate the clone detection logic.
Assistant
This is a Go-based project with some Python components. The clone detection logic is likely in the Go code. Let me explore the structure more.
Assistant
Let me start by reading the clone detector code and understanding the structure:
Assistant
The open command needs a file path, not a key-value argument. Let me fix that.
Assistant
Let me continue reading the file to see the grouping logic:
Assistant
Let me look for the grouping logic and the
dedupeStrictSubsetGroupMembersfunction:Assistant
Now let me look at the grouping strategy and the deduplication function:
Assistant
Let me read the group dedup logic first:
Assistant
Now let me look at the grouping strategy to understand how groups are formed:
Assistant
Let me look for the connected grouping implementation:
Assistant
Now I understand the issue better. The problem is that two separate clone groups are being emitted that are near-duplicates of each other. The current
dedupeStrictSubsetGroupMembersfunction only deduplicates members within a single group, but it does not deduplicate across different groups.Let me look at the test file to understand the expected behavior:
Assistant
Now I understand the existing tests. The issue described in the bug report is different - it's about two separate clone groups that are near-duplicates of each other (not members within the same group). Let me understand the problem better by looking at how pairs are formed and groups created.
Let me first reproduce the issue to understand it better:
Assistant
I need to set up the Go environment properly before building.
Assistant
Good, now let me download the Fixit repo and reproduce the issue:
Assistant
Now let me run the clone analysis on the Fixit repo:
Assistant
Let me look for the duplicate clone groups mentioned in the issue:
Assistant
Let me check what's in the file:
Assistant
The analyze.json file is not valid JSON, it seems to have been written to a different location. Let me find the actual JSON report file.
Assistant
commands_devstr长度为1
Assistant
Perfect! I can reproduce the issue. There are two groups (17 and 10) that are near-duplicates:
- Group 17: smoke.py: 299-315 and 318-334 (sim=0.9692)
- Group 10: smoke.py: 298-315 and 317-334 (sim=0.9613)
These are the same duplication but with group 10 starting one line earlier. Let me verify by looking at the actual source code:
Assistant
Now I understand the issue clearly. The two groups report the same duplication but with different window boundaries. The fix needs to add inter-group deduplication logic that detects when two groups are essentially reporting the same duplication relationship.
Let me analyze this more:
- Both groups have 2 members from the same file (smoke.py)
- Group 17 members: [299-315, 318-334]
- Group 10 members: [298-315, 317-334]
- Each member in Group 10 overlaps with the corresponding member in Group 17 (one covers the other)
The fix should merge or suppress such duplicate groups. Let me look at the code flow more carefully:
Assistant
Now I need to implement inter-group deduplication. The key insight is:
- Two groups are duplicates if they describe the same duplication relationships
- For 2-member groups, this means both members in one group overlap with corresponding members in the other group (from the same files)
Let me design the solution:
- After
dedupeStrictSubsetGroupMembers, add a new function to deduplicate overlapping groups - Two groups are "overlapping duplicates" if, after normalizing member order by file, each member in one group overlaps with the corresponding member in the other group
Let me implement this:
Assistant
Now let me look at some helper functions used in this file:
Assistant
Continuing to explore the codebase to understand the helper functions.
Assistant
Search for helper function definitions to understand how to compare fragments and groups.
Assistant
Now I understand the codebase structure. Let me implement the fix. The issue is that two separate groups can represent the same duplication relationship (overlapping groups). I need to add inter-group deduplication logic.
Let me create the fix by adding a new deduplication function:
Assistant
I notice there's a bug in my code - I referenced
sortedBbefore defining it in the first sort. Let me fix that:Assistant
I see the bug - the first sort references
sortedB[j]beforesortedBis defined. Let me fix this:Assistant
Let me verify the fix was applied correctly:
Assistant
Good. Now I need to integrate the new function into the clone detector. Let me update the
groupClonesWithStrategyfunction to call the new dedup function:Assistant
Now let me update this function to also call the new inter-group deduplication:
Assistant
Let me verify the change was applied correctly:
Assistant
Now let me rebuild and test:
Assistant
Great! The build succeeded. Let me run the tests first to make sure I didn't break anything:
Assistant
All existing dedup tests pass. Now let me test the fix with the Fixit repository to confirm it resolves the issue:
Assistant
Excellent! The fix works! The duplicate group has been removed:
- Before: 9 clone groups, with 2 overlapping groups for smoke.py (Group 17 and Group 10)
- After: 8 clone groups, with only 1 group for that duplication (Group 5, which has the higher similarity of 0.9692)
The group with higher similarity (0.9692) was kept, and the redundant one (0.9613) was removed.
Now let me write a test case to ensure this fix is properly tested:
Assistant
Now let me run the new tests to make sure they pass:
Assistant
All new tests pass. Let me also run all the analyzer tests to make sure nothing is broken:
Assistant
All tests pass. Now let me run the broader test suite to make sure nothing is broken in other packages:
Assistant
All tests pass. Let me verify the fix one more time by confirming the output with the Fixit repository:
Assistant
The fix is working correctly. Let me verify by comparing with the original output that showed the bug:
Before fix (original output):
- Total clone groups: 9
- Two overlapping groups for the same duplication:
- Group 17: 299-315, 318-334 (sim=0.9692)
- Group 10: 298-315, 317-334 (sim=0.9613)
After fix:
- Total clone groups: 8 (one redundant group removed)
- Only one group for that duplication:
- Group 5: 299-315, 318-334 (sim=0.9692) - the higher similarity one
The fix correctly:
- Detects that two groups are overlapping (describing the same duplication with slightly different window boundaries)
- Keeps the group with higher similarity (0.9692 > 0.9613)
- Removes the redundant group
- Reassigns IDs sequentially
Let me also check edge cases to make sure my solution is robust. Let me view the final code changes:
Assistant
Now let me also review the full implementation of the new functions I added:
Assistant
A
<tool_call> <function=bash_terminal> <parameter=command> cd /pyscn && git diff internal/analyzer/group_dedup_test.go | head -100 </tool_call>
Assistant
I need to use the bash_terminal function properly.
Assistant
Let me run the full test suite one final time to confirm everything works:
Assistant
All tests pass. Let me provide a summary of what I did:
Summary
I successfully fixed the bug where pyscn's clone detector emitted two separate clone groups that describe the same duplication relationship between the same two code fragments, with member windows shifted by exactly one line.
Root Cause
The clone grouping/dedup stage only handled deduplication within groups (via
dedupeStrictSubsetGroupMembers) but not across different groups. When two groups reported the same duplication with slightly different window boundaries (e.g., one including an enclosingwith/header line and the other not), both were kept.Solution
I added inter-group deduplication logic:
-
dedupeOverlappingCloneGroups: Removes clone groups that are near-duplicates of each other. When two groups are overlapping, it keeps the one with higher similarity. -
areOverlappingGroups: Determines if two groups describe the same duplication relationship. The heuristic is:- Both groups must have the same number of members
- After sorting members by file path, each corresponding pair must be from the same file with overlapping line ranges
Changes Made
internal/analyzer/group_dedup.go: Added the two new functions (~96 lines)internal/analyzer/clone_detector.go: Integrated the new function intogroupClonesWithStrategy(1 line change)internal/analyzer/group_dedup_test.go: Added comprehensive tests for edge cases (~181 lines)
Verification
- Before fix: 9 clone groups, including two overlapping groups for the same duplication (similarity 0.9692 and 0.9613)
- After fix: 8 clone groups, only one group for that duplication (similarity 0.9692, the higher one)
Checklist
- ✅ Reproduced the error: Confirmed the bug with Instagram/Fixit repository
- ✅ Confirmed the fix resolves the error: Duplicate group is now removed
- ✅ Thought about edge cases: Added tests for different files, different sizes, disjoint ranges, multiple overlapping groups, empty/single groups
- ✅ Ran existing tests: All tests in
./...pass (38+ seconds of test execution)
-
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.
Identical first prompt
Qwen/Qwen3.6-27BCompare side by side →
Qwen/Qwen3.6-27BCompare side by side →
Qwen/Qwen3.6-27BCompare side by side →
Qwen/Qwen3.6-27BCompare side by side →
- xiaomi/mimo-v2.5-proCompare side by side →
- xiaomi/mimo-v2.5-proCompare side by side →
- xiaomi/mimo-v2.5-proCompare side by side →
- xiaomi/mimo-v2.5-proCompare side by side →
- xiaomi/mimo-v2.5-proCompare side by side →
- gpt-5.6-solCompare side by side →
- gpt-5.6-solCompare side by side →