Skip to content
TheChatArchiveThe Chat Archive
Claudeclaude-opus-5· anthropicHugging Face ibragim-bad/swe_rebench_07_2026_trajectories

ISSUE DESCRIPTION [BUG][auto] duplicate-overlapping-clone-groups: same duplication…

  1. User

    ISSUE DESCRIPTION

    [BUG][auto] duplicate-overlapping-clone-groups: same duplication emitted as two separate near-identical clone groups

    Bug

    pyscn's clone detector emits two separate clone groups that describe the same duplication relationship between the same two code fragments, with member windows shifted by exactly one line (the enclosing with/header line). The result is a redundant clone group that inflates clone_groups count and the duplication percentage.

    This is related to but distinct from the (closed) #416 clone-overlapping-window-self-clone: #416 was about strict-subset members within a single group; here the redundancy is across two whole groups that are near-duplicates of each other.

    Reproducer

    Repo: Instagram/Fixit @ bf8e1f55f8a606eae0f0c64bab575e84a2a608b3 pyscn: v1.22.9

    curl -fsSL "https://codeload.github.com/Instagram/Fixit/tar.gz/HEAD" | tar -xz
    pyscn analyze --json --no-open Fixit-*/
    

    In the resulting analyze.json, clone.clone_groups contains two groups that cover the same pair of fragments in src/fixit/tests/smoke.py:

    {
      "id": 6,  "type": 2, "similarity": 0.9692,
      "clones": [
        {"location": {"file_path": ".../smoke.py", "start_line": 299, "end_line": 315}},
        {"location": {"file_path": ".../smoke.py", "start_line": 318, "end_line": 334}}
      ]
    },
    {
      "id": 14, "type": 2, "similarity": 0.9613,
      "clones": [
        {"location": {"file_path": ".../smoke.py", "start_line": 298, "end_line": 315}},
        {"location": {"file_path": ".../smoke.py", "start_line": 317, "end_line": 334}}
      ]
    }
    

    Both groups describe the duplication between the subTest("lint") block and the subTest("fix") block. Group 14's members each start exactly one line earlier than group 6's — that line is the with self.subTest("lint"): / with self.subTest("fix"): header:

            with self.subTest("lint"):          # line 298  (grp14 member start)
                with TemporaryDirectory() as td: # line 299  (grp6 member start)
                    ...
            with self.subTest("fix"):           # line 317  (grp14 member start)
                with TemporaryDirectory() as td: # line 318  (grp6 member start)
    

    Both groups end on the same lines (315 / 334). The two groups are the same logical clone reported twice at slightly different window boundaries.

    Expected behavior

    The clone-grouping/dedup stage should merge (or suppress) clone groups whose members are overlapping windows of the same underlying fragments, so each duplication relationship is reported once. As-is, the clone_groups count (10 here) and the duplication score over-count by one redundant group.

    Actual behavior

    Two distinct groups (id: 6 and id: 14) are emitted for the same duplication, differing only by whether the with self.subTest(...) header line is included in each member window.

    Priority rationale

    P2 (visible but bounded inaccuracy): the cited line ranges are themselves valid, and the duplication is real — but the same finding is double-counted, inflating the group count and duplication percentage by a bounded amount.

    Found via the FP-audit skill in repo Instagram/Fixit@bf8e1f55f8a606eae0f0c64bab575e84a2a608b3.

    Interface

    No new interfaces are introduced.

    ADDITIONAL ADVICE

    Since you are given a git repository, you can use git commands to simplify your work. For example, if you made a mistake and want to revert your changes, you can use git checkout <file> to restore the file to its original state. You can also reset all changes in the repository using git reset --hard command. Additionally, you can use git stash and git stash pop to temporarily save your changes and restore them later. Finally, you can see the changes that you've made (which will be submitted when you call submit) using git status or git diff commands. However you don't need to use git add or git commit before submitting your changes. In fact, submit only submits currently unstaged changes because it uses git diff to get the changes that need to be submitted.

    CHECKLIST

    Before submitting your solution, please go over the following checklist and make sure you've done everything:

    • If an error was reported in the issue description, I have successfully reproduced it.
    • If an error was reported in the issue description, I have confirmed that my fix resolves the error.
    • I have thought about edge cases and made sure my fix handles them as well.
    • I have run existing tests in the repository that might have been affected by the change I made and confirmed that they pass. I want you to list every bullet from this checklist and write your reasoning for why you think you did it or didn't need to.

    Repository has been uploaded and your shell is currently at the repository root. Time to solve the issue!

    (Current directory: /pyscn, current file: none) bash-$

  2. Claude Assistant

    Now let me implement the cross-group dedup pass.

  3. Claude Assistant

    I've completed the fix. Let me do a final review of the change and then go through the checklist.

  4. Claude Assistant

    Summary of the change

    internal/analyzer/group_dedup.go — the clone-group dedup stage (already called from CloneDetector.groupClonesWithStrategy) got a second pass, filterRedundantOverlappingGroups:

    • Groups are visited in "most informative first" order (groupMorePreferred: most members → widest line coverage → highest similarity → deterministic location tiebreak).
    • A group is dropped when each of its members can be matched one-to-one (Kuhn bipartite matching) with a distinct member of an already-kept group that covers essentially the same code window (sameCodeWindow: same file, shared lines ≥ 80 % of both windows). That is exactly the reported shape: {smoke.py:299-315, 318-334} vs {smoke.py:298-315, 317-334} — differing only by the enclosing with self.subTest(...) header line.
    • Fragments that only lived in dropped groups are added to the existing suppressed set, so filterClonePairsWithSuppressedMembers also drops the now-redundant clone pairs (keeps clone_pairs, clones and the duplication score consistent).
    • Kept groups are re-sorted with the existing sortCloneGroups ordering, so output stays deterministic; candidate lookup is indexed by file path plus a cheap fast-reject, so there is no measurable slowdown.

    Deliberately conservative: nested-but-differently-sized clones (a small clone inside a big one), disjoint same-file clones and cross-file relationships with a non-matching member are untouched, so real distinct findings are never merged.

    Checklist

    • “If an error was reported in the issue description, I have successfully reproduced it.” — Partly, and here is exactly how: I downloaded Instagram/Fixit@bf8e1f5 and ran the built CLI. With the current code the two ids 6/14 no longer appear (similarity computation changed since the reported v1.22.9 — the pair (299-315, 318-334) isn’t even produced any more), so the CLI could not reproduce the literal JSON. I therefore reproduced the reported shape at the analyzer layer with a temporary test: feeding the two real clone pairs (299-315~318-334 @0.9692 and 298-315~317-334 @0.9613) through the real NewConnectedGrouping and CloneDetector.groupClonesWithStrategy emitted two near-identical groups before my change (log output confirmed pre-dedup groups: 2).
    • “If an error was reported in the issue description, I have confirmed that my fix resolves the error.” — Yes. After the change the same input yields one group (smoke.py:298-315, smoke.py:317-334) and one clone pair, both through dedupeStrictSubsetGroupMembers and through the detector entry point groupClonesWithStrategy (so clone_groups count and the duplication percentage are no longer inflated). I also re-ran the real Fixit and Scrapy repos end-to-end: group sets are byte-identical to the pre-change binary (no false merges), timings unchanged.
    • “I have thought about edge cases and made sure my fix handles them as well.” — Covered with temporary tests and code guards: distinct duplications in the same files stay separate; a small clone nested in a much bigger clone stays separate (mutual-coverage criterion, not plain containment); groups of different member counts (2 vs 3) subsume correctly via injective matching; identical duplicate groups collapse to one; nil groups/fragments/locations, zero/negative spans, groups sharing a fragment (fragment is not suppressed if it survives in a kept group), single-group and empty input, deterministic tiebreaks, and O(G²) blow-up avoided via a per-file index plus fast reject.
    • “I have run existing tests in the repository that might have been affected by the change I made and confirmed that they pass.” — Yes: go test ./... (whole repo, incl. internal/analyzer clone/dedup/grouping suites, service, app, integration) passes, plus go vet ./... and gofmt -l clean. No existing test was modified; my temporary reproduction test file and scratch directories were deleted (git status shows only internal/analyzer/group_dedup.go modified).

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.