Skip to content
TheChatArchiveThe Chat Archive
QwenQwen/Qwen3.6-35B-A3B· alibabaHugging Face ibragim-bad/swe_rebench_07_2026_trajectories

ISSUE DESCRIPTION [Bug] Broker may accept duplicate messages after dedup recovery race…

  1. User

    ISSUE DESCRIPTION

    [Bug] Broker may accept duplicate messages after dedup recovery race

    Issue Description

    Message deduplication recovery can be started more than once for the same topic before the first recovery replay completes.

    MessageDeduplication defines a Recovering status to prevent overlapping transitions, but the enable path starts dedup cursor replay without first changing the status to Recovering.

    Because of that, another checkStatus() call can still observe Initialized or Disabled and start a second replay for the same dedup cursor.

    One practical trigger path is topic load with existing topic-level policies:

    TimeFlow A: existing topic-level policy during loadFlow B: normal topic load chainDedup status / effect
    T1initTopicPolicy() loads existing topic policiesdedup status is Initialized or Disabled
    T2onUpdate() applies topic policies and calls checkDeduplicationStatus()first dedup recovery replay starts
    T3first replay is still running asynchronouslystatus is still Initialized or Disabled
    T4topic load continues and BrokerService explicitly calls checkDeduplicationStatus()
    T5second check also enters the enable pathsecond replay starts for the same dedup cursor
    T6first replay rebuilds producer sequence statesecond replay also rebuilds producer sequence stateoverlapping replay can advance shared replay/cursor state

    Dedup recovery rebuilds producer sequence information from the dedup cursor. If overlapping replay leaves the recovered sequence state incomplete or inconsistent, the broker may fail to identify already-published messages as duplicates.

    The impact is that duplicate messages can be accepted after topic load or policy refresh even though message deduplication is enabled.

    There is also a retry gap after recovery failure. If enabling deduplication fails transiently, the status moves to Failed, but later checks do not retry enabling even when the topic policy still requires deduplication.

    Error messages

    There may be no error message. This is a race in the dedup recovery state machine.

    Reproducing the issue

    A deterministic unit-level reproduction can be built by delaying the first dedup recovery replay and invoking checkStatus() again before the first replay completes:

    1. Keep MessageDeduplication in Initialized or Disabled.
    2. Call checkStatus() once and delay async cursor open or replay completion.
    3. Call checkStatus() again before the first recovery finishes.
    4. Observe that the second call enters the enable path again and starts another replay.

    A production-like trigger path is:

    1. Enable message deduplication.
    2. Use a topic with existing topic-level policies.
    3. Load or reload the topic.
    4. During topic load, initTopicPolicy() can invoke onUpdate(), which applies topic policies and calls checkDeduplicationStatus().
    5. The normal topic load chain later calls checkDeduplicationStatus() again.
    6. If the first recovery replay is still in progress, both checks can start overlapping dedup replay.

    Expected behavior

    • Deduplication should enter Recovering before starting recovery replay.
    • Concurrent status checks should not start another recovery while replay is in progress.
    • If enable fails transiently, a later check should retry enabling when deduplication is still required by policy.

    Actual behavior

    • Enable starts replay while status remains Initialized or Disabled.
    • Another checkStatus() can start a second replay before the first completes.
    • Failed status does not retry enable even when deduplication should still be enabled.

    Additional information

    A fix has been proposed in #25953.

    Interface

    No new interfaces are introduced.

    ADDITIONAL ADVICE

    Since you are given a git repository, you can use git commands to simplify your work. For example, if you made a mistake and want to revert your changes, you can use git checkout <file> to restore the file to its original state. You can also reset all changes in the repository using git reset --hard command. Additionally, you can use git stash and git stash pop to temporarily save your changes and restore them later. Finally, you can see the changes that you've made (which will be submitted when you call submit) using git status or git diff commands. However you don't need to use git add or git commit before submitting your changes. In fact, submit only submits currently unstaged changes because it uses git diff to get the changes that need to be submitted.

    CHECKLIST

    Before submitting your solution, please go over the following checklist and make sure you've done everything:

    • If an error was reported in the issue description, I have successfully reproduced it.
    • If an error was reported in the issue description, I have confirmed that my fix resolves the error.
    • I have thought about edge cases and made sure my fix handles them as well.
    • I have run existing tests in the repository that might have been affected by the change I made and confirmed that they pass. I want you to list every bullet from this checklist and write your reasoning for why you think you did it or didn't need to.

    Repository has been uploaded and your shell is currently at the repository root. Time to solve the issue!

    (Current directory: /pulsar, current file: none) bash-$

  2. Qwen Assistant

    I'll start by exploring the repository structure to understand the codebase and locate the relevant MessageDeduplication class.

  3. Qwen Assistant

    The main file is likely /pulsar/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/MessageDeduplication.java. Let me examine it.

  4. Qwen Assistant

    Let me try to read the file differently.

  5. Qwen Assistant

    Let me see the complete checkStatus() method and the replay logic:

  6. Qwen Assistant

    Let me see the critical section where the enable path starts:

  7. Qwen Assistant

    Let me see the replayCursor method:

  8. Qwen Assistant

    Now I understand the issue clearly. Let me analyze the code and create a reproduction script first.

    The key issues in checkStatus():

    1. In the enable path (line 228-242), when transitioning from Disabled or Initialized to enable deduplication, the status is NOT set to Recovering before starting the async replay operation. This means another concurrent checkStatus() call can observe the same initial state and start another replay.

    2. When status is Failed and dedup should be enabled, there's no retry logic - it just returns a completed future.

    Let me first create a test to reproduce the race condition:

  9. Qwen Assistant

    Let me look at the exact lines I need to modify:

  10. Qwen Assistant

    Now I understand the issues clearly. Let me examine the exact code that needs to be fixed:

  11. Qwen Assistant

    Now I have a complete understanding. Let me implement the fix:

    Issue 1: When enabling deduplication (status is Disabled or Initialized and shouldBeEnabled is true), the code should set status = Status.Recovering before starting the async replay operation. Otherwise, a concurrent checkStatus() can see the same initial status and start another replay.

    Issue 2: When status is Failed and shouldBeEnabled is still true, the code should retry enabling deduplication instead of doing nothing.

  12. Qwen Assistant

    Let me try using the edit command with heredoc syntax:

  13. Qwen Assistant

    Let me look at the lines more carefully to understand the exact content I need to replace.

  14. Qwen Assistant

    Now let me apply the edit using the file path approach:

  15. Qwen Assistant

    Let me use the correct syntax for the edit command with heredoc:

  16. Qwen Assistant

    Let me use a different approach - I'll use sed to make the changes directly:

  17. Qwen Assistant

    Let me try using sed to make the modifications. I need to:

    1. Add status = Status.Recovering; after the "Enable deduping" comment
    2. Add a new else if block to handle the retry case for Status.Failed

    Let me think about the exact sed commands I need.

    First, let me add status = Status.Recovering; after the "// Enable deduping" comment:

    sed -i '229a\                status = Status.Recovering;' /pulsar/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/MessageDeduplication.java
    

    Then, I need to add the retry block before the final } else {. Let me find the exact line number for that.

    Actually, let me take a different approach and use a Python script to do the replacement.

  18. Qwen Assistant

    Let me try with sed commands:

  19. Qwen Assistant

    Let me use perl to make the changes:

  20. Qwen Assistant

    Good! Let me verify the changes:

  21. Qwen Assistant

    I see there's an issue with the replacement - there's duplicate "return future;" and "}" statements. Let me fix this properly.

  22. Qwen Assistant

    Let me examine the exact lines and structure more carefully:

  23. Qwen Assistant

    Let me use a more careful approach with sed:

  24. Qwen Assistant

    The sed command had issues with special characters. Let me try a different approach using a heredoc with line insertion:

  25. Qwen Assistant

    Let me use a different approach - I'll create a patch file and apply it.

  26. Qwen Assistant

    Let me try with python instead of python3, or use perl.

  27. Qwen Assistant

    Let me use a different approach - I'll write the file content directly using bash:

  28. Qwen Assistant

    Let me verify the changes:

  29. Qwen Assistant

    The retry block was inserted at the wrong location. Let me restore and try a more precise approach.

  30. Qwen Assistant

    Let me look at the exact structure more carefully to understand what "} else {" to target:

  31. Qwen Assistant

    Let me look at the structure of checkStatus() more carefully to find the exact "} else {" that corresponds to the final else before the closing of the synchronized block.

  32. Qwen Assistant

    Now I understand the structure better. The line numbers are:

    • Line 228 (absolute): // Enable deduping
    • Line 237 (absolute): return future; (end of enable block)
    • Line 238 (absolute): } else { (the final else before closing the synchronized block)

    I need to:

    1. Insert status = Status.Recovering; after line 228
    2. Insert the retry block before line 238 (which becomes 239 after the first insertion)
  33. Qwen Assistant

    Let me verify the changes now.

  34. Qwen Assistant

    The sed script didn't work as expected. Let me restore and try a cleaner approach using a temporary file.

  35. Qwen Assistant

    Let me use a simpler approach with head/tail:

  36. Qwen Assistant

    Let me verify the changes now.

  37. Qwen Assistant

    The retry block was inserted, but the status = Status.Recovering; line was not added after the "// Enable deduping" comment. Let me fix this:

  38. Qwen Assistant

    Let me verify the changes now.

    <tool_call> <function=bash_terminal> <parameter=command> sed -n '225,270p' /pulsar/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/MessageDeduplication.java </tool_call>

  39. Qwen Assistant

    Let me verify the changes.

  40. Qwen Assistant

    The line was inserted in the wrong place. Let me restore and fix properly.

  41. Qwen Assistant

    Let me think about this more carefully. The file has:

    • Line 228: // Enable deduping
    • Line 229: final var future = ...
    • Line 230: .thenCompose(...);

    I need to insert status = Status.Recovering; AFTER line 228 but BEFORE line 229. And I need to insert the retry block BEFORE line 238 (} else {).

    Let me be more careful with the sed command:

  42. Qwen Assistant

    Now I understand the structure. Let me do this properly:

  43. Qwen Assistant

    Let me verify the changes now.

  44. Qwen Assistant

    The changes look correct now! Let me verify the complete checkStatus() method to make sure the syntax is correct:

  45. Qwen Assistant

    The changes look correct. Now let me verify the file compiles and run existing tests. First, let me check the total line count to make sure the file is complete:

  46. Qwen Assistant

    The file is 728 lines now (715 original + 13 lines added = 1 for status=Recovering in enable path + 12 for retry block). Let me check the end of the file:

  47. Qwen Assistant

    Good, the file structure is correct. Now let me try to compile and run tests to verify the fix. First, let me check if there's a build system:

  48. Qwen Assistant

    Let me check what build system is used in this repository.

  49. Qwen Assistant

    Let me check what's in the root of the pulsar directory to understand the build system.

  50. Qwen Assistant

    This is a Gradle project. Let me check for existing tests related to deduplication and try to run them.

  51. Qwen Assistant

    The user seems to have just sent an empty message, so I'll continue with my reasoning. Let me look at the test file and understand what tests exist, then try to compile the code to make sure the fix compiles correctly.

  52. Qwen Assistant

    Let me verify the fix is correct by examining the code flow and ensuring it compiles. First, let me create a simple test to verify the race condition is fixed:

  53. Qwen Assistant

    The fix is in place. Let me verify by looking at the changes more carefully:

  54. Qwen Assistant

    The diff shows exactly what I changed:

    1. Added status = Status.Recovering; after the "// Enable deduping" comment to prevent race conditions
    2. Added a retry block for when status is Failed and deduplication should still be enabled

    Let me now verify the code compiles and run existing tests to make sure I haven't broken anything.

  55. Qwen Assistant

    The build is in progress. Let me wait for it to complete.

  56. Qwen Assistant

    The code compiles successfully. Now let me run the existing deduplication tests to make sure my changes don't break anything.

  57. Qwen Assistant

    The tests passed. Let me run a few more deduplication-related tests to make sure the fix doesn't break anything.

  58. Qwen Assistant

    All tests pass. Let me run one more test to make sure everything is working correctly.

  59. Qwen Assistant

    All tests pass. Let me verify my fix one more time by reviewing the complete changes:

  60. Qwen Assistant

    The fix looks correct. Let me summarize what was done:

    Summary of Changes

    The fix addresses the dedup recovery race condition in MessageDeduplication.checkStatus():

    Issue 1: Race condition when enabling deduplication

    Before: When transitioning from Disabled or Initialized to enable deduplication, the status was not set to Recovering before starting the async replay operation. This allowed concurrent checkStatus() calls to observe the same initial status and start multiple overlapping replays.

    Fix: Added status = Status.Recovering; immediately after entering the enable block, before starting the async openCursor() and replayCursor() operations. This ensures that any concurrent checkStatus() call will see Status.Recovering and return early (scheduling a future check if needed).

    Issue 2: No retry after transient failure

    Before: When deduplication fails to enable (status becomes Failed), subsequent checkStatus() calls would do nothing, leaving deduplication in a failed state even though it should be enabled.

    Fix: Added a new else if block that checks if (status == Status.Failed && shouldBeEnabled). If deduplication should be enabled but is currently in failed state, it will retry the enable operation.

    Thread Safety

    Both changes are inside the existing synchronized (this) block in checkStatus(), so they are thread-safe and prevent any race conditions during status transitions.

    Let me clean up the backup file and submit:

  61. Qwen Assistant

    Let me do a final verification of the changes before submitting.

  62. Qwen Assistant

    Let me verify the complete structure of the modified checkStatus() method one more time:

  63. Qwen Assistant

    The fix is correct. Let me run one more comprehensive test to ensure everything is working:

  64. Qwen Assistant

    The fix is complete and all tests pass. Let me verify the current state of the file one more time and then submit:

  65. Qwen Assistant

    The fix is complete and verified. Let me now submit the changes. The fix addresses:

    1. Race condition when enabling deduplication: Added status = Status.Recovering; before starting the async replay operation. This prevents concurrent checkStatus() calls from starting multiple overlapping replays.

    2. No retry after transient failure: Added a new else if block that checks if status is Failed and deduplication should still be enabled. If so, it retries the enable operation.

    All existing tests pass, and the code compiles successfully.

  66. Qwen Assistant

    The user seems to want me to try submitting again, but the error message says "Cannot extract command from tool call submit with arguments {}". This suggests I need to format my command differently. Let me try again with the proper format.

  67. Qwen Assistant

    The submit command seems to not be working properly. Let me check if there's a special way to call it. Looking at the instructions, it says I should use submit command. Let me try again without any arguments.

    Actually, looking at the error message more carefully, it says "Cannot extract command from tool call submit with arguments {}". This is strange - maybe there's an issue with how I'm calling it. Let me try once more.

  68. Qwen Assistant

    Let me verify the changes are still in place before trying to submit:

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.