Skip to content
TheChatArchiveThe Chat Archive
QwenQwen/Qwen3.6-27B· alibabaHugging Face ibragim-bad/swe_rebench_07_2026_trajectories

ISSUE DESCRIPTION [BUG][auto] duplicate-overlapping-clone-groups: same duplication…

  1. User

    ISSUE DESCRIPTION

    [BUG][auto] duplicate-overlapping-clone-groups: same duplication emitted as two separate near-identical clone groups

    Bug

    pyscn's clone detector emits two separate clone groups that describe the same duplication relationship between the same two code fragments, with member windows shifted by exactly one line (the enclosing with/header line). The result is a redundant clone group that inflates clone_groups count and the duplication percentage.

    This is related to but distinct from the (closed) #416 clone-overlapping-window-self-clone: #416 was about strict-subset members within a single group; here the redundancy is across two whole groups that are near-duplicates of each other.

    Reproducer

    Repo: Instagram/Fixit @ bf8e1f55f8a606eae0f0c64bab575e84a2a608b3 pyscn: v1.22.9

    curl -fsSL "https://codeload.github.com/Instagram/Fixit/tar.gz/HEAD" | tar -xz
    pyscn analyze --json --no-open Fixit-*/
    

    In the resulting analyze.json, clone.clone_groups contains two groups that cover the same pair of fragments in src/fixit/tests/smoke.py:

    {
      "id": 6,  "type": 2, "similarity": 0.9692,
      "clones": [
        {"location": {"file_path": ".../smoke.py", "start_line": 299, "end_line": 315}},
        {"location": {"file_path": ".../smoke.py", "start_line": 318, "end_line": 334}}
      ]
    },
    {
      "id": 14, "type": 2, "similarity": 0.9613,
      "clones": [
        {"location": {"file_path": ".../smoke.py", "start_line": 298, "end_line": 315}},
        {"location": {"file_path": ".../smoke.py", "start_line": 317, "end_line": 334}}
      ]
    }
    

    Both groups describe the duplication between the subTest("lint") block and the subTest("fix") block. Group 14's members each start exactly one line earlier than group 6's — that line is the with self.subTest("lint"): / with self.subTest("fix"): header:

            with self.subTest("lint"):          # line 298  (grp14 member start)
                with TemporaryDirectory() as td: # line 299  (grp6 member start)
                    ...
            with self.subTest("fix"):           # line 317  (grp14 member start)
                with TemporaryDirectory() as td: # line 318  (grp6 member start)
    

    Both groups end on the same lines (315 / 334). The two groups are the same logical clone reported twice at slightly different window boundaries.

    Expected behavior

    The clone-grouping/dedup stage should merge (or suppress) clone groups whose members are overlapping windows of the same underlying fragments, so each duplication relationship is reported once. As-is, the clone_groups count (10 here) and the duplication score over-count by one redundant group.

    Actual behavior

    Two distinct groups (id: 6 and id: 14) are emitted for the same duplication, differing only by whether the with self.subTest(...) header line is included in each member window.

    Priority rationale

    P2 (visible but bounded inaccuracy): the cited line ranges are themselves valid, and the duplication is real — but the same finding is double-counted, inflating the group count and duplication percentage by a bounded amount.

    Found via the FP-audit skill in repo Instagram/Fixit@bf8e1f55f8a606eae0f0c64bab575e84a2a608b3.

    ADDITIONAL ADVICE

    Since you are given a git repository, you can use git commands to simplify your work. For example, if you made a mistake and want to revert your changes, you can use git checkout <file> to restore the file to its original state. You can also reset all changes in the repository using git reset --hard command. Additionally, you can use git stash and git stash pop to temporarily save your changes and restore them later. Finally, you can see the changes that you've made (which will be submitted when you call submit) using git status or git diff commands. However you don't need to use git add or git commit before submitting your changes. In fact, submit only submits currently unstaged changes because it uses git diff to get the changes that need to be submitted.

    CHECKLIST

    Before submitting your solution, please go over the following checklist and make sure you've done everything:

    • If an error was reported in the issue description, I have successfully reproduced it.
    • If an error was reported in the issue description, I have confirmed that my fix resolves the error.
    • I have thought about edge cases and made sure my fix handles them as well.
    • I have run existing tests in the repository that might have been affected by the change I made and confirmed that they pass. I want you to list every bullet from this checklist and write your reasoning for why you think you did it or didn't need to.

    Repository has been uploaded and your shell is currently at the repository root. Time to solve the issue!

    (Current directory: /pyscn, current file: none) bash-$

  2. Qwen Assistant

    I'll start by exploring the repository structure to understand the codebase and locate the clone detection logic.

  3. Qwen Assistant

    This is a Go-based project with some Python components. The clone detection logic is likely in the Go code. Let me explore the structure more.

  4. Qwen Assistant

    Let me start by reading the clone detector code and understanding the structure:

  5. Qwen Assistant

    The open command needs a file path, not a key-value argument. Let me fix that.

  6. Qwen Assistant

    Let me continue reading the file to see the grouping logic:

  7. Qwen Assistant

    Let me look for the grouping logic and the dedupeStrictSubsetGroupMembers function:

  8. Qwen Assistant

    Now let me look at the grouping strategy and the deduplication function:

  9. Qwen Assistant

    Let me read the group dedup logic first:

  10. Qwen Assistant

    Now let me look at the grouping strategy to understand how groups are formed:

  11. Qwen Assistant

    Let me look for the connected grouping implementation:

  12. Qwen Assistant

    Now I understand the issue better. The problem is that two separate clone groups are being emitted that are near-duplicates of each other. The current dedupeStrictSubsetGroupMembers function only deduplicates members within a single group, but it does not deduplicate across different groups.

    Let me look at the test file to understand the expected behavior:

  13. Qwen Assistant

    Now I understand the existing tests. The issue described in the bug report is different - it's about two separate clone groups that are near-duplicates of each other (not members within the same group). Let me understand the problem better by looking at how pairs are formed and groups created.

    Let me first reproduce the issue to understand it better:

  14. Qwen Assistant

    I need to set up the Go environment properly before building.

  15. Qwen Assistant

    Good, now let me download the Fixit repo and reproduce the issue:

  16. Qwen Assistant

    Now let me run the clone analysis on the Fixit repo:

  17. Qwen Assistant

    Let me look for the duplicate clone groups mentioned in the issue:

  18. Qwen Assistant

    Let me check what's in the file:

  19. Qwen Assistant

    The analyze.json file is not valid JSON, it seems to have been written to a different location. Let me find the actual JSON report file.

  20. Qwen Assistant

    commands_devstr长度为1

  21. Qwen Assistant

    Perfect! I can reproduce the issue. There are two groups (17 and 10) that are near-duplicates:

    • Group 17: smoke.py: 299-315 and 318-334 (sim=0.9692)
    • Group 10: smoke.py: 298-315 and 317-334 (sim=0.9613)

    These are the same duplication but with group 10 starting one line earlier. Let me verify by looking at the actual source code:

  22. Qwen Assistant

    Now I understand the issue clearly. The two groups report the same duplication but with different window boundaries. The fix needs to add inter-group deduplication logic that detects when two groups are essentially reporting the same duplication relationship.

    Let me analyze this more:

    • Both groups have 2 members from the same file (smoke.py)
    • Group 17 members: [299-315, 318-334]
    • Group 10 members: [298-315, 317-334]
    • Each member in Group 10 overlaps with the corresponding member in Group 17 (one covers the other)

    The fix should merge or suppress such duplicate groups. Let me look at the code flow more carefully:

  23. Qwen Assistant

    Now I need to implement inter-group deduplication. The key insight is:

    1. Two groups are duplicates if they describe the same duplication relationships
    2. For 2-member groups, this means both members in one group overlap with corresponding members in the other group (from the same files)

    Let me design the solution:

    1. After dedupeStrictSubsetGroupMembers, add a new function to deduplicate overlapping groups
    2. Two groups are "overlapping duplicates" if, after normalizing member order by file, each member in one group overlaps with the corresponding member in the other group

    Let me implement this:

  24. Qwen Assistant

    Now let me look at some helper functions used in this file:

  25. Qwen Assistant

    Continuing to explore the codebase to understand the helper functions.

  26. Qwen Assistant

    Search for helper function definitions to understand how to compare fragments and groups.

  27. Qwen Assistant

    Now I understand the codebase structure. Let me implement the fix. The issue is that two separate groups can represent the same duplication relationship (overlapping groups). I need to add inter-group deduplication logic.

    Let me create the fix by adding a new deduplication function:

  28. Qwen Assistant

    I notice there's a bug in my code - I referenced sortedB before defining it in the first sort. Let me fix that:

  29. Qwen Assistant

    I see the bug - the first sort references sortedB[j] before sortedB is defined. Let me fix this:

  30. Qwen Assistant

    Let me verify the fix was applied correctly:

  31. Qwen Assistant

    Good. Now I need to integrate the new function into the clone detector. Let me update the groupClonesWithStrategy function to call the new dedup function:

  32. Qwen Assistant

    Now let me update this function to also call the new inter-group deduplication:

  33. Qwen Assistant

    Let me verify the change was applied correctly:

  34. Qwen Assistant

    Now let me rebuild and test:

  35. Qwen Assistant

    Great! The build succeeded. Let me run the tests first to make sure I didn't break anything:

  36. Qwen Assistant

    All existing dedup tests pass. Now let me test the fix with the Fixit repository to confirm it resolves the issue:

  37. Qwen Assistant

    Excellent! The fix works! The duplicate group has been removed:

    • Before: 9 clone groups, with 2 overlapping groups for smoke.py (Group 17 and Group 10)
    • After: 8 clone groups, with only 1 group for that duplication (Group 5, which has the higher similarity of 0.9692)

    The group with higher similarity (0.9692) was kept, and the redundant one (0.9613) was removed.

    Now let me write a test case to ensure this fix is properly tested:

  38. Qwen Assistant

    Now let me run the new tests to make sure they pass:

  39. Qwen Assistant

    All new tests pass. Let me also run all the analyzer tests to make sure nothing is broken:

  40. Qwen Assistant

    All tests pass. Now let me run the broader test suite to make sure nothing is broken in other packages:

  41. Qwen Assistant

    All tests pass. Let me verify the fix one more time by confirming the output with the Fixit repository:

  42. Qwen Assistant

    The fix is working correctly. Let me verify by comparing with the original output that showed the bug:

    Before fix (original output):

    • Total clone groups: 9
    • Two overlapping groups for the same duplication:
      • Group 17: 299-315, 318-334 (sim=0.9692)
      • Group 10: 298-315, 317-334 (sim=0.9613)

    After fix:

    • Total clone groups: 8 (one redundant group removed)
    • Only one group for that duplication:
      • Group 5: 299-315, 318-334 (sim=0.9692) - the higher similarity one

    The fix correctly:

    1. Detects that two groups are overlapping (describing the same duplication with slightly different window boundaries)
    2. Keeps the group with higher similarity (0.9692 > 0.9613)
    3. Removes the redundant group
    4. Reassigns IDs sequentially

    Let me also check edge cases to make sure my solution is robust. Let me view the final code changes:

  43. Qwen Assistant

    Now let me also review the full implementation of the new functions I added:

  44. Qwen Assistant

    A

    <tool_call> <function=bash_terminal> <parameter=command> cd /pyscn && git diff internal/analyzer/group_dedup_test.go | head -100 </tool_call>

  45. Qwen Assistant

    I need to use the bash_terminal function properly.

  46. Qwen Assistant

    Let me run the full test suite one final time to confirm everything works:

  47. Qwen Assistant

    All tests pass. Let me provide a summary of what I did:

    Summary

    I successfully fixed the bug where pyscn's clone detector emitted two separate clone groups that describe the same duplication relationship between the same two code fragments, with member windows shifted by exactly one line.

    Root Cause

    The clone grouping/dedup stage only handled deduplication within groups (via dedupeStrictSubsetGroupMembers) but not across different groups. When two groups reported the same duplication with slightly different window boundaries (e.g., one including an enclosing with/header line and the other not), both were kept.

    Solution

    I added inter-group deduplication logic:

    1. dedupeOverlappingCloneGroups: Removes clone groups that are near-duplicates of each other. When two groups are overlapping, it keeps the one with higher similarity.

    2. areOverlappingGroups: Determines if two groups describe the same duplication relationship. The heuristic is:

      • Both groups must have the same number of members
      • After sorting members by file path, each corresponding pair must be from the same file with overlapping line ranges

    Changes Made

    • internal/analyzer/group_dedup.go: Added the two new functions (~96 lines)
    • internal/analyzer/clone_detector.go: Integrated the new function into groupClonesWithStrategy (1 line change)
    • internal/analyzer/group_dedup_test.go: Added comprehensive tests for edge cases (~181 lines)

    Verification

    • Before fix: 9 clone groups, including two overlapping groups for the same duplication (similarity 0.9692 and 0.9613)
    • After fix: 8 clone groups, only one group for that duplication (similarity 0.9692, the higher one)

    Checklist

    • ✅ Reproduced the error: Confirmed the bug with Instagram/Fixit repository
    • ✅ Confirmed the fix resolves the error: Duplicate group is now removed
    • ✅ Thought about edge cases: Added tests for different files, different sizes, disjoint ranges, multiple overlapping groups, empty/single groups
    • ✅ Ran existing tests: All tests in ./... pass (38+ seconds of test execution)

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.