Skip to content
TheChatArchiveThe Chat Archive
DeepSeekdeepseek/deepseek-v4-pro· deepseekHugging Face ibragim-bad/swe_rebench_07_2026_trajectories

ISSUE DESCRIPTION [BUG][auto] duplicate-overlapping-clone-groups: same duplication…

  1. User

    ISSUE DESCRIPTION

    [BUG][auto] duplicate-overlapping-clone-groups: same duplication emitted as two separate near-identical clone groups

    Bug

    pyscn's clone detector emits two separate clone groups that describe the same duplication relationship between the same two code fragments, with member windows shifted by exactly one line (the enclosing with/header line). The result is a redundant clone group that inflates clone_groups count and the duplication percentage.

    This is related to but distinct from the (closed) #416 clone-overlapping-window-self-clone: #416 was about strict-subset members within a single group; here the redundancy is across two whole groups that are near-duplicates of each other.

    Reproducer

    Repo: Instagram/Fixit @ bf8e1f55f8a606eae0f0c64bab575e84a2a608b3 pyscn: v1.22.9

    curl -fsSL "https://codeload.github.com/Instagram/Fixit/tar.gz/HEAD" | tar -xz
    pyscn analyze --json --no-open Fixit-*/
    

    In the resulting analyze.json, clone.clone_groups contains two groups that cover the same pair of fragments in src/fixit/tests/smoke.py:

    {
      "id": 6,  "type": 2, "similarity": 0.9692,
      "clones": [
        {"location": {"file_path": ".../smoke.py", "start_line": 299, "end_line": 315}},
        {"location": {"file_path": ".../smoke.py", "start_line": 318, "end_line": 334}}
      ]
    },
    {
      "id": 14, "type": 2, "similarity": 0.9613,
      "clones": [
        {"location": {"file_path": ".../smoke.py", "start_line": 298, "end_line": 315}},
        {"location": {"file_path": ".../smoke.py", "start_line": 317, "end_line": 334}}
      ]
    }
    

    Both groups describe the duplication between the subTest("lint") block and the subTest("fix") block. Group 14's members each start exactly one line earlier than group 6's — that line is the with self.subTest("lint"): / with self.subTest("fix"): header:

            with self.subTest("lint"):          # line 298  (grp14 member start)
                with TemporaryDirectory() as td: # line 299  (grp6 member start)
                    ...
            with self.subTest("fix"):           # line 317  (grp14 member start)
                with TemporaryDirectory() as td: # line 318  (grp6 member start)
    

    Both groups end on the same lines (315 / 334). The two groups are the same logical clone reported twice at slightly different window boundaries.

    Expected behavior

    The clone-grouping/dedup stage should merge (or suppress) clone groups whose members are overlapping windows of the same underlying fragments, so each duplication relationship is reported once. As-is, the clone_groups count (10 here) and the duplication score over-count by one redundant group.

    Actual behavior

    Two distinct groups (id: 6 and id: 14) are emitted for the same duplication, differing only by whether the with self.subTest(...) header line is included in each member window.

    Priority rationale

    P2 (visible but bounded inaccuracy): the cited line ranges are themselves valid, and the duplication is real — but the same finding is double-counted, inflating the group count and duplication percentage by a bounded amount.

    Found via the FP-audit skill in repo Instagram/Fixit@bf8e1f55f8a606eae0f0c64bab575e84a2a608b3.

    ADDITIONAL ADVICE

    Since you are given a git repository, you can use git commands to simplify your work. For example, if you made a mistake and want to revert your changes, you can use git checkout <file> to restore the file to its original state. You can also reset all changes in the repository using git reset --hard command. Additionally, you can use git stash and git stash pop to temporarily save your changes and restore them later. Finally, you can see the changes that you've made (which will be submitted when you call submit) using git status or git diff commands. However you don't need to use git add or git commit before submitting your changes. In fact, submit only submits currently unstaged changes because it uses git diff to get the changes that need to be submitted.

    CHECKLIST

    Before submitting your solution, please go over the following checklist and make sure you've done everything:

    • If an error was reported in the issue description, I have successfully reproduced it.
    • If an error was reported in the issue description, I have confirmed that my fix resolves the error.
    • I have thought about edge cases and made sure my fix handles them as well.
    • I have run existing tests in the repository that might have been affected by the change I made and confirmed that they pass. I want you to list every bullet from this checklist and write your reasoning for why you think you did it or didn't need to.

    Repository has been uploaded and your shell is currently at the repository root. Time to solve the issue!

    (Current directory: /pyscn, current file: none) bash-$

  2. DeepSeek Assistant

    Let me start by understanding the repository structure and the issue. The bug is about duplicate overlapping clone groups being emitted for the same duplication relationship. Let me explore the repository first.

    The issue describes:

    1. pyscn emits two separate clone groups that describe the same duplication between the same two code fragments
    2. The member windows are shifted by exactly one line (the enclosing with/header line)
    3. Group 14's members each start exactly one line earlier than group 6's

    This is a deduplication/merging issue in the clone detection code.

    Let me start by exploring the repository structure.

  3. DeepSeek Assistant

    This looks like a Go project. Let me explore the internal structure, particularly the clone detection logic

  4. DeepSeek Assistant

    Let me look at the key files related to clone detection and deduplication.

  5. DeepSeek Assistant

    Let me look at the key files: group_dedup.go (deduplication logic) and clone_detector.go (clone detection).

  6. DeepSeek Assistant

    The open command might not be working. Let me just use cat or head to read the file.

  7. DeepSeek Assistant

    Now let me look at the clone_detector.go to understand how groups are created:

  8. DeepSeek Assistant

    Let me see the rest of the file and also look at the grouping strategies:

  9. DeepSeek Assistant

    Let me look at the connected grouping strategy (default) and the dedup test:

  10. DeepSeek Assistant

    Now let me look at the group_dedup_test.go to understand what's tested:

  11. DeepSeek Assistant

    Let me understand how fragments are created and how clone pairs are formed. Let me look at the file_analyzer.go and also find relevant code about fragment extraction:

  12. DeepSeek Assistant

    Let me look at the fragment extraction code in clone_detector.go:

  13. DeepSeek Assistant

    Let me look at isFragmentCandidate:

  14. DeepSeek Assistant

    Let me see these two functions:

  15. DeepSeek Assistant

    Now let me see the main detection flow to understand where the dedup is called:

  16. DeepSeek Assistant

    Now let me understand the full picture. Let me also look at the test file for clone_detector to see if there are related tests:

  17. DeepSeek Assistant

    Let me now look at the module_analyzer to understand how the results flow to the final output:

  18. DeepSeek Assistant

    No results. Let me search more broadly:

  19. DeepSeek Assistant

    Let me look at the test helper functions for creating fragments:

  20. DeepSeek Assistant

    Let me look at these helper functions:

  21. DeepSeek Assistant

    Let me check testdata for smoke test files:

  22. DeepSeek Assistant

    Now I have a good understanding of the code. Let me implement the fix. I'll add a new function to group_dedup.go and call it from clone_detector.go.

    Let me first re-read the full group_dedup.go:

  23. DeepSeek Assistant

    Let me find where utility functions like pairKey and fragmentLess are defined

  24. DeepSeek Assistant

    Now let me look at the groupClonesWithStrategy to see where to add the new dedup call

  25. DeepSeek Assistant

    Now I have a clear plan. Let me implement the fix:

    1. Add dedupeOverlappingCloneGroups to group_dedup.go to suppress groups whose members are all covered by another group
    2. Call it from groupClonesWithStrategy in clone_detector.go

    Let me first add the new function to group_dedup.go:

  26. DeepSeek Assistant

    Now let me update groupClonesWithStrategy in clone_detector.go to call the new dedup

  27. DeepSeek Assistant

    Now let me update the groupClonesWithStrategy function to add the new dedup call

  28. DeepSeek Assistant

    Let me verify the change was applied correctly

  29. DeepSeek Assistant

    Good. Now let me run the existing tests to make sure nothing is broken:

  30. DeepSeek Assistant

    Let me set up the Go environment:

  31. DeepSeek Assistant

    Let me set up all Go environment

  32. DeepSeek Assistant

    All existing dedup tests pass. Let me run all analyzer tests to make sure nothing else is

  33. DeepSeek Assistant

    All tests pass. Let me also run the full test suite:

  34. DeepSeek Assistant

    Now let me check how duplication percentage is calculated to see if we also need to filter pairs:

  35. DeepSeek Assistant

    Let me look at how duplication is computed:

  36. DeepSeek Assistant

    Let me look at how CodeDuplication is computed:

  37. DeepSeek Assistant

    Let me check countUniqueCloneFragments

  38. DeepSeek Assistant

    Now let me write a quick test script to verify the fix handles the exact scenario from the issue:

  39. DeepSeek Assistant

    Let me check all places where grouping is done to make sure my fix covers all paths:

  40. DeepSeek Assistant

    Let me try a simpler grep

  41. DeepSeek Assistant

    Good - groupClonesWithStrategy is called from both DetectClonesWithContext (line 641) and DetectClonesWithLSH (line 810), so my fix covers all paths.

    Let me verify the final state of my changes:

  42. DeepSeek Assistant

    Let me run the full test suite once more to confirm everything passes

  43. DeepSeek Assistant

    All tests pass. Let me also run all tests across the project

  44. DeepSeek Assistant

    Let me clean up the temp

  45. DeepSeek Assistant

    Now let me go through the checklist:

    Checklist

    [x] Reproduced the error

    I simulated the exact scenario from the issue using a Go script that creates two clone groups matching the reported pattern (Group 6 with lines 299-315/318-334, Group 14 with lines 298-315/317-334). The script confirmed that without the fix, both groups would be emitted.

    [x] Confirmed the fix resolves the error

    My test script verified that dedupeOverlappingCloneGroups correctly suppresses the inner/narrower group (Group 6) and keeps the outer/wider group (Group 14). The surviving group has the larger line ranges (298-315, 317-334).

    [x] Edge cases considered

    • Groups with different numbers of members: groupCovers returns false when fragment counts differ, so both groups survive. Correct.
    • Groups in different files: groupCovers checks file path equality, so cross-file mismatches prevent false suppression.
    • Chain of 3+ nested groups: The algorithm iterates and suppresses all but the widest group. E.g., A covers B, B covers C → only A survives.
    • Identical groups (same ranges): The second loop handles this by suppressing the later-index group deterministically.
    • Non-containment overlap: If fragments overlap but neither contains the other, groupCovers returns false and both survive.
    • Nil fragments/locations: Both functions have nil checks to prevent panics.
    • Empty groups: Already filtered out by dedupeStrictSubsetGroupMembers (drops groups with < 2 members).

    [x] Existing tests pass

    All analyzer tests pass, and all project-wide tests pass. No existing tests were modified.

    The fix adds a new post-dedup step dedupeOverlappingCloneGroups that runs after the existing dedupeStrictSubsetGroupMembers in groupClonesWithStrategy. This ensures that clone groups describing the same duplication relationship (with one group's member windows shifted by one or more lines due to recursive fragment extraction of parent/child AST nodes) are merged, keeping only the group with the widest window boundaries

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.