Skip to content
TheChatArchiveThe Chat Archive
xiaomi/mimo-v2.5-pro· unknownHugging Face ibragim-bad/swe_rebench_07_2026_trajectories

ISSUE DESCRIPTION [BUG][auto] duplicate-overlapping-clone-groups: same duplication…

  1. User

    ISSUE DESCRIPTION

    [BUG][auto] duplicate-overlapping-clone-groups: same duplication emitted as two separate near-identical clone groups

    Bug

    pyscn's clone detector emits two separate clone groups that describe the same duplication relationship between the same two code fragments, with member windows shifted by exactly one line (the enclosing with/header line). The result is a redundant clone group that inflates clone_groups count and the duplication percentage.

    This is related to but distinct from the (closed) #416 clone-overlapping-window-self-clone: #416 was about strict-subset members within a single group; here the redundancy is across two whole groups that are near-duplicates of each other.

    Reproducer

    Repo: Instagram/Fixit @ bf8e1f55f8a606eae0f0c64bab575e84a2a608b3 pyscn: v1.22.9

    curl -fsSL "https://codeload.github.com/Instagram/Fixit/tar.gz/HEAD" | tar -xz
    pyscn analyze --json --no-open Fixit-*/
    

    In the resulting analyze.json, clone.clone_groups contains two groups that cover the same pair of fragments in src/fixit/tests/smoke.py:

    {
      "id": 6,  "type": 2, "similarity": 0.9692,
      "clones": [
        {"location": {"file_path": ".../smoke.py", "start_line": 299, "end_line": 315}},
        {"location": {"file_path": ".../smoke.py", "start_line": 318, "end_line": 334}}
      ]
    },
    {
      "id": 14, "type": 2, "similarity": 0.9613,
      "clones": [
        {"location": {"file_path": ".../smoke.py", "start_line": 298, "end_line": 315}},
        {"location": {"file_path": ".../smoke.py", "start_line": 317, "end_line": 334}}
      ]
    }
    

    Both groups describe the duplication between the subTest("lint") block and the subTest("fix") block. Group 14's members each start exactly one line earlier than group 6's — that line is the with self.subTest("lint"): / with self.subTest("fix"): header:

            with self.subTest("lint"):          # line 298  (grp14 member start)
                with TemporaryDirectory() as td: # line 299  (grp6 member start)
                    ...
            with self.subTest("fix"):           # line 317  (grp14 member start)
                with TemporaryDirectory() as td: # line 318  (grp6 member start)
    

    Both groups end on the same lines (315 / 334). The two groups are the same logical clone reported twice at slightly different window boundaries.

    Expected behavior

    The clone-grouping/dedup stage should merge (or suppress) clone groups whose members are overlapping windows of the same underlying fragments, so each duplication relationship is reported once. As-is, the clone_groups count (10 here) and the duplication score over-count by one redundant group.

    Actual behavior

    Two distinct groups (id: 6 and id: 14) are emitted for the same duplication, differing only by whether the with self.subTest(...) header line is included in each member window.

    Priority rationale

    P2 (visible but bounded inaccuracy): the cited line ranges are themselves valid, and the duplication is real — but the same finding is double-counted, inflating the group count and duplication percentage by a bounded amount.

    Found via the FP-audit skill in repo Instagram/Fixit@bf8e1f55f8a606eae0f0c64bab575e84a2a608b3.

    ADDITIONAL ADVICE

    Since you are given a git repository, you can use git commands to simplify your work. For example, if you made a mistake and want to revert your changes, you can use git checkout <file> to restore the file to its original state. You can also reset all changes in the repository using git reset --hard command. Additionally, you can use git stash and git stash pop to temporarily save your changes and restore them later. Finally, you can see the changes that you've made (which will be submitted when you call submit) using git status or git diff commands. However you don't need to use git add or git commit before submitting your changes. In fact, submit only submits currently unstaged changes because it uses git diff to get the changes that need to be submitted.

    CHECKLIST

    Before submitting your solution, please go over the following checklist and make sure you've done everything:

    • If an error was reported in the issue description, I have successfully reproduced it.
    • If an error was reported in the issue description, I have confirmed that my fix resolves the error.
    • I have thought about edge cases and made sure my fix handles them as well.
    • I have run existing tests in the repository that might have been affected by the change I made and confirmed that they pass. I want you to list every bullet from this checklist and write your reasoning for why you think you did it or didn't need to.

    Repository has been uploaded and your shell is currently at the repository root. Time to solve the issue!

    (Current directory: /pyscn, current file: none) bash-$

  2. Assistant

    Let me start by exploring the repository structure to understand the codebase and then work on reproducing and fixing the issue.

  3. Assistant

    Let me explore the repository structure more thoroughly.

  4. Assistant

    This is a Go repository (go.mod) with Python wrappers. Let me understand the structure better.

  5. Assistant

    Let me find the clone detection related code:

  6. Assistant

    Let me explore the clone detector code to understand how clone groups are formed and deduplicated.

  7. Assistant

    Let me search for the grouping and deduplication code:

  8. Assistant

    Let me find the dedupeStrictSubsetGroupMembers function:

  9. Assistant

    Let me look at the group_dedup.go file.

  10. Assistant

    Let me now look at the grouping strategies:

  11. Assistant

    Let me look at the grouping strategy code.

  12. Assistant

    Let me look at the grouping mode code to understand the different strategies.

  13. Assistant

    Now let me understand how fragments are extracted, as this is key to understanding how overlapping fragments can be created:

  14. Assistant

    Let me look at how fragments are extracted to understand the root cause:

  15. Assistant

    Now I see the issue. Let me understand the grouping strategy being used. Let me look at the centroid grouping strategy:

  16. Assistant

    Let me look at the compareFragments function to understand the Jaccard pre-filter:

  17. Assistant

    Let me look at the full compareFragments function:

  18. Assistant

    Now let me look at the shouldCompareFragments function:

  19. Assistant

    Let me see this function:

  20. Assistant

    Now let me look at the main DetectClones method:

  21. Assistant

    Let me see the DetectClones method:

  22. Assistant

    Now let me look at the detectClonePairsWithContext function:

  23. Assistant

    Let me search for the detect clone pairs method.

  24. Assistant

    Let me see the detectClonePairsWithContext function:

  25. Assistant

    Now I understand the flow. The issue is that two separate clone groups are created for the same duplication when the fragment extraction produces overlapping fragments (one including the with header, one without), and the grouping strategy doesn't connect them all into a single group (or they form separate groups), while the deduplication only works within groups, not across groups.

    Let me first look at the existing tests to understand the expected behavior:

  26. Assistant

    Now let me look at the group_dedup tests:

  27. Assistant

    Now let me look at the connected grouping strategy (the default):

  28. Assistant

    Let me look at the connected grouping strategy.

  29. Assistant

    Now I understand the issue. Let me check how the groups are produced and where the deduplication needs to happen. Let me also look at the averageGroupSimilarity and pairKey helper functions:

  30. Assistant

    Let me look at these helper functions:

  31. Assistant

    Now I have a good understanding of the issue. Let me create a reproducer first. The problem is:

    1. Two overlapping fragments exist in the same file (e.g., 298-315 and 299-315 - one includes the with header, one doesn't)
    2. Similarly for the other copy (317-334 and 318-334)
    3. Clone pairs are formed between the outer fragments (298-315, 317-334) and inner fragments (299-315, 318-334) separately, but NOT between the cross-pairs (298-315, 318-334) and (299-315, 317-334) because the similarity might be slightly lower
    4. This creates two separate connected components in the grouping, resulting in two groups
    5. The dedupeStrictSubsetGroupMembers only works WITHIN a group, so it can't catch this cross-group overlap

    Let me first write a test to reproduce this, then implement the fix:

  32. Assistant

    Let me find the gf helper function:

  33. Assistant

    Let me look at the gf helper function:

  34. Assistant

    Now let me understand the full picture and implement the fix. The issue is that dedupeStrictSubsetGroupMembers only works within a single group, but the overlapping fragments end up in DIFFERENT groups. I need to add cross-group deduplication.

    Let me first run the existing tests to make sure they pass before I make changes:

  35. Assistant

    Let me check the Go environment:

  36. Assistant

    Let me check the Go environment variables:

  37. Assistant

    Let me set up the Go environment:

  38. Assistant

    Now let me run the existing tests:

  39. Assistant

    Let me set up the environment properly:

  40. Assistant

    Good, all existing tests pass. Now let me implement the fix. The approach is to add a cross-group deduplication step that detects when two groups describe the same duplication with overlapping fragment windows.

  41. Assistant

    Now I have a clear understanding of the issue. Let me implement the fix in group_dedup.go. The issue is that dedupeStrictSubsetGroupMembers only deduplicates within a single group, but when overlapping fragments end up in different groups, the redundancy is not caught.

  42. Assistant

    Now I need to update groupClonesWithStrategy to call dedupeOverlappingCloneGroups:

  43. Assistant

    Let me see the current implementation of groupClonesWithStrategy:

  44. Assistant

    Now I need to update groupClonesWithStrategy to call dedupeOverlappingCloneGroups:

  45. Assistant

    Now I need to update groupClonesWithStrategy to add the cross-group deduplication step:

  46. Assistant

    Now let me verify the group_dedup.go file was updated correctly:

  47. Assistant

    Good. Now let me run the existing tests to make sure they still pass:

  48. Assistant

    All existing tests pass. Now let me write a test that reproduces the exact issue described in the bug report:

  49. Assistant

    Now let me write a test that reproduces the exact issue and tests the new functionality:

  50. Assistant

    Now let me run the new tests to see if they pass:

  51. Assistant

    The test TestDedupeOverlappingGroups_EmptyAndNil fails because when we pass []*CloneGroup{nil}, the function returns []*CloneGroup{nil} instead of filtering it out. Let me fix the dedupeOverlappingCloneGroups function to handle nil groups properly.

  52. Assistant

    I need to fix the dedupeOverlappingCloneGroups function to also filter out nil groups in the output:

  53. Assistant

    I need to fix the dedupeOverlappingCloneGroups function to filter out nil groups in the output:

  54. Assistant

    I see there's a duplicate return out line. Let me fix that:

  55. Assistant

    Good. Now let me run the tests again:

  56. Assistant

    The issue is in the early return. When we have []*CloneGroup{nil}, len(groups) == 1 so it returns early without filtering the nil. Let me fix this:

  57. Assistant

    Now let me run the tests again:

  58. Assistant

    All new tests pass. Now let me run all the existing tests to make sure I didn't break anything:

  59. Assistant

    All tests pass. Now let me run the full test suite for the analyzer package:

  60. Assistant

    All tests pass. Now let me run the broader test suite to make sure I didn't break anything:

  61. Assistant

    All tests pass across the entire repository. Let me now review my changes to make sure they're complete and correct:

  62. Assistant

    Let me review the final state of the changes:

  63. Assistant

    The changes look clean. Let me also check if there are any integration or e2e tests that might be affected:

  64. Assistant

    Let me also verify the service tests pass (they take longer):

  65. Assistant

    All tests pass. Let me do a final review of my changes:

  66. Assistant

    Let me run the full test suite one final time to confirm everything is working:

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.