ISSUE DESCRIPTION [BUG][auto] duplicate-overlapping-clone-groups: same duplication…
- User
ISSUE DESCRIPTION
[BUG][auto] duplicate-overlapping-clone-groups: same duplication emitted as two separate near-identical clone groups
Bug
pyscn's clone detector emits two separate clone groups that describe the same duplication relationship between the same two code fragments, with member windows shifted by exactly one line (the enclosing
with/header line). The result is a redundant clone group that inflatesclone_groupscount and the duplication percentage.This is related to but distinct from the (closed) #416
clone-overlapping-window-self-clone: #416 was about strict-subset members within a single group; here the redundancy is across two whole groups that are near-duplicates of each other.Reproducer
Repo:
Instagram/Fixit@bf8e1f55f8a606eae0f0c64bab575e84a2a608b3pyscn:v1.22.9curl -fsSL "https://codeload.github.com/Instagram/Fixit/tar.gz/HEAD" | tar -xz pyscn analyze --json --no-open Fixit-*/In the resulting
analyze.json,clone.clone_groupscontains two groups that cover the same pair of fragments insrc/fixit/tests/smoke.py:{ "id": 6, "type": 2, "similarity": 0.9692, "clones": [ {"location": {"file_path": ".../smoke.py", "start_line": 299, "end_line": 315}}, {"location": {"file_path": ".../smoke.py", "start_line": 318, "end_line": 334}} ] }, { "id": 14, "type": 2, "similarity": 0.9613, "clones": [ {"location": {"file_path": ".../smoke.py", "start_line": 298, "end_line": 315}}, {"location": {"file_path": ".../smoke.py", "start_line": 317, "end_line": 334}} ] }Both groups describe the duplication between the
subTest("lint")block and thesubTest("fix")block. Group 14's members each start exactly one line earlier than group 6's — that line is thewith self.subTest("lint"):/with self.subTest("fix"):header:with self.subTest("lint"): # line 298 (grp14 member start) with TemporaryDirectory() as td: # line 299 (grp6 member start) ... with self.subTest("fix"): # line 317 (grp14 member start) with TemporaryDirectory() as td: # line 318 (grp6 member start)Both groups end on the same lines (315 / 334). The two groups are the same logical clone reported twice at slightly different window boundaries.
Expected behavior
The clone-grouping/dedup stage should merge (or suppress) clone groups whose members are overlapping windows of the same underlying fragments, so each duplication relationship is reported once. As-is, the
clone_groupscount (10 here) and the duplication score over-count by one redundant group.Actual behavior
Two distinct groups (
id: 6andid: 14) are emitted for the same duplication, differing only by whether thewith self.subTest(...)header line is included in each member window.Priority rationale
P2 (visible but bounded inaccuracy): the cited line ranges are themselves valid, and the duplication is real — but the same finding is double-counted, inflating the group count and duplication percentage by a bounded amount.
Found via the FP-audit skill in repo
Instagram/Fixit@bf8e1f55f8a606eae0f0c64bab575e84a2a608b3.ADDITIONAL ADVICE
Since you are given a git repository, you can use git commands to simplify your work. For example, if you made a mistake and want to revert your changes, you can use
git checkout <file>to restore the file to its original state. You can also reset all changes in the repository usinggit reset --hardcommand. Additionally, you can usegit stashandgit stash popto temporarily save your changes and restore them later. Finally, you can see the changes that you've made (which will be submitted when you callsubmit) usinggit statusorgit diffcommands. However you don't need to usegit addorgit commitbefore submitting your changes. In fact,submitonly submits currently unstaged changes because it usesgit diffto get the changes that need to be submitted.CHECKLIST
Before submitting your solution, please go over the following checklist and make sure you've done everything:
- If an error was reported in the issue description, I have successfully reproduced it.
- If an error was reported in the issue description, I have confirmed that my fix resolves the error.
- I have thought about edge cases and made sure my fix handles them as well.
- I have run existing tests in the repository that might have been affected by the change I made and confirmed that they pass. I want you to list every bullet from this checklist and write your reasoning for why you think you did it or didn't need to.
Repository has been uploaded and your shell is currently at the repository root. Time to solve the issue!
(Current directory: /pyscn, current file: none) bash-$
- Assistant
I’ll first inspect the repository structure and locate the clone detection/grouping implementation and relevant tests, then reproduce the overlap behavior with the smallest practical fixture before modifying the deduplication logic.
- Assistant
The project is Go-based, with clone behavior likely split between
internal,service, anddomain. I’ll enumerate clone-specific files and inspect the detector pipeline, especially any existing handling for overlapping/subset windows related to issue #416. - Assistant
There is already a dedicated
group_dedup.goand tests for strict-subset members within a group. I’ll inspect its data model, grouping strategy, and invocation to determine where cross-group near-duplicate suppression belongs without conflating distinct clone relationships. - Assistant
The current post-pass only compares members inside each group, so it cannot catch the reported shape: two groups each containing the same number of corresponding, nearly nested windows. I’ll now inspect group construction and fragment identity/keying so the new pass can conservatively identify groups whose members admit a one-to-one same-file containment mapping, retain the maximal-window group, and keep pair output/statistics consistent.
- Assistant
A conservative rule should require equal group cardinality and a bijection where every member of one group contains its counterpart in the same file, with at least one strict containment; partial overlap alone is too broad. Before implementing, I’ll verify how detector groups become domain output and how duplication percentage is calculated, because suppressing a redundant group may also require suppressing only the redundant clone-pair edge while preserving fragments used by unrelated findings.
- Assistant
The analysis score is directly based on
TotalCloneGroups, so removing the duplicate group addresses both symptoms. Pair filtering should be exact: remove edges wholly represented by a suppressed group only when their endpoints are not members of any retained group, avoiding damage to shared fragments in overlapping clustering modes. I’ll inspect K-core/star behavior and current test helpers, then implement the cross-group pass as a second stage ingroup_dedup.go. - Assistant
A pure containment rule across groups could incorrectly erase a legitimate nested clone (for example, duplicated functions and duplicated inner loops). I’ll therefore use a near-identity criterion: corresponding ranges must be in the same file, one direction must consistently contain the other, and each inner range must cover at least 80% of its outer range. This handles boundary shifts proportionally rather than hardcoding one line, while excluding substantially nested findings. I’ll implement deterministic one-to-one matching, prefer the maximal group, and feed suppressed fragments through the existing pair-filter mechanism.
- Assistant
Reviewing the first implementation exposed an important edge case: two near-duplicate groups may share an exact fragment pointer. Marking every member of the removed group as globally suppressed would then remove valid pairs supporting the retained group. I’ll refine the pass to compute retained membership first and suppress only fragments not used by any retained group. I’ll also collapse fully identical cross-groups, which are an even clearer duplicate case, and rely on actual fragment cardinality rather than potentially stale
Sizemetadata. - Assistant
The combined shell command had a quoting typo and did not execute, so no partial edit occurred. I’ll rerun the same refinement cleanly, then run the focused analyzer/service/app tests.
- Assistant
The focused test command was blocked by the environment because both
GOMODCACHEandGOPATHare unset, not by a code failure. Before rerunning with an isolated/tmpGo cache, I’ll replace the recursive matching with a polynomial-time augmenting-path matcher so large clone groups cannot trigger factorial behavior. Then I’ll run focused and full tests. - Assistant
Dependencies downloaded successfully, but this minimal environment also lacks
HOME/GOCACHE; I’ll explicitly setGOCACHE=/tmp/gocacheand rerun. After tests pass, I’ll build the CLI and attempt the exact Fixit revision reproducer in/tmp, comparing the reportedsmoke.pygroups before and after the fix if network access is available. - Assistant
All focused and full Go tests pass. The exact Fixit analysis also completed and reports 8 groups, but my glob omitted the hidden
.pyscndirectory. I’ll inspect that report now. To satisfy the reproduction checklist rigorously, I’ll also temporarily build the unmodified baseline from Git (restoring the working file immediately), run it against the same checkout, and compare the specificsmoke.pygroups and totals. - Assistant
The current
HEADbaseline no longer reproduces the v1.22.9 count exactly because intervening grouping changes already combine these fragments into a larger group; both current binaries report 8 groups. Since the issue names v1.22.9 explicitly, I’ll reproduce against that tag using a separate Git worktree, leaving the submitted tree untouched. I’ll also tighten “near-identical” coverage to 90%, which still captures a one-line shift on 17–18-line windows while more safely preserving deeply nested clone findings. - Assistant
The uploaded clone does not contain the v1.22.9 tag locally, so the detached worktree command failed before changing anything. I’ll check the remote and fetch only that tag if available, then rerun the historical binary.
- Assistant
No Git remote is configured in the uploaded repository. I can fetch the tag directly from the public repository URL without modifying Git configuration, so I’ll do that and repeat the historical reproduction.
- Assistant
Even the exact public v1.22.9 tag in this environment produces a single broad group containing both outer windows (7 total), rather than the reported 10-group output—likely due configuration/environment differences in the audit run. The historical command nevertheless confirms the relevant outer ranges. I’ll now perform a careful code review, run static analysis, rerun all tests after the final 90% threshold change, and inspect the final diff for correctness and unintended repository changes.
- Assistant
Static analysis and the full suite pass. To directly verify the new private helper without adding any repository test file (as prohibited), I’ll use Go’s
-overlayfacility to inject a temporary/tmptest at build time. It will cover the exact 298/299 and 317/318 ranges, deeply nested groups that must remain distinct, and shared-fragment pair-suppression safety. I’ll remove the temporary overlay afterward and do one final status check. - Assistant
Final review is complete, with only
internal/analyzer/group_dedup.gomodified.Checklist:
- If an error was reported, successfully reproduced it. I reproduced the exact reported post-group shape using the same
smoke.pyranges (298–315/317–334 and 299–315/318–334) through a temporary Go overlay test. I also ran the full Fixit revision with both current HEAD and public v1.22.9; those builds now/defaulted to broader connected groups in this environment, so they did not independently emit the historical 10-group layout. - Confirmed the fix resolves the error. The overlay test verifies the two near-identical groups collapse to the maximal outer-window group and the inner fragments are suppressed from pair output.
- Thought about edge cases. The implementation requires equal cardinality, a complete one-to-one same-file containment mapping, consistent containment direction, and at least 90% line coverage. It preserves substantially nested clones, handles identical groups, nil locations, arbitrary member ordering, and shared fragments that remain in another retained group. Matching is polynomial-time.
- Ran affected existing tests.
go test ./...passes, including analyzer, service, app, integration, and all other packages. Focused tests,go vet,gofmt, andgit diff --checkalso pass.
I’ll now submit the single-file fix.
- If an error was reported, successfully reproduced it. I reproduced the exact reported post-group shape using the same
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.
Identical first prompt
Qwen/Qwen3.6-27BCompare side by side →
Qwen/Qwen3.6-27BCompare side by side →
Qwen/Qwen3.6-27BCompare side by side →
Qwen/Qwen3.6-27BCompare side by side →
Qwen/Qwen3.6-27BCompare side by side →
- xiaomi/mimo-v2.5-proCompare side by side →
- xiaomi/mimo-v2.5-proCompare side by side →
- xiaomi/mimo-v2.5-proCompare side by side →
- xiaomi/mimo-v2.5-proCompare side by side →
- xiaomi/mimo-v2.5-proCompare side by side →
- gpt-5.6-solCompare side by side →
- gpt-5.6-solCompare side by side →