ISSUE DESCRIPTION [Bug] Broker may accept duplicate messages after dedup recovery race…
- User
ISSUE DESCRIPTION
[Bug] Broker may accept duplicate messages after dedup recovery race
Issue Description
Message deduplication recovery can be started more than once for the same topic before the first recovery replay completes.
MessageDeduplicationdefines aRecoveringstatus to prevent overlapping transitions, but the enable path starts dedup cursor replay without first changing the status toRecovering.Because of that, another
checkStatus()call can still observeInitializedorDisabledand start a second replay for the same dedup cursor.One practical trigger path is topic load with existing topic-level policies:
Time Flow A: existing topic-level policy during load Flow B: normal topic load chain Dedup status / effect T1 initTopicPolicy()loads existing topic policiesdedup status is InitializedorDisabledT2 onUpdate()applies topic policies and callscheckDeduplicationStatus()first dedup recovery replay starts T3 first replay is still running asynchronously status is still InitializedorDisabledT4 topic load continues and BrokerServiceexplicitly callscheckDeduplicationStatus()T5 second check also enters the enable path second replay starts for the same dedup cursor T6 first replay rebuilds producer sequence state second replay also rebuilds producer sequence state overlapping replay can advance shared replay/cursor state Dedup recovery rebuilds producer sequence information from the dedup cursor. If overlapping replay leaves the recovered sequence state incomplete or inconsistent, the broker may fail to identify already-published messages as duplicates.
The impact is that duplicate messages can be accepted after topic load or policy refresh even though message deduplication is enabled.
There is also a retry gap after recovery failure. If enabling deduplication fails transiently, the status moves to
Failed, but later checks do not retry enabling even when the topic policy still requires deduplication.Error messages
There may be no error message. This is a race in the dedup recovery state machine.
Reproducing the issue
A deterministic unit-level reproduction can be built by delaying the first dedup recovery replay and invoking
checkStatus()again before the first replay completes:- Keep
MessageDeduplicationinInitializedorDisabled. - Call
checkStatus()once and delay async cursor open or replay completion. - Call
checkStatus()again before the first recovery finishes. - Observe that the second call enters the enable path again and starts another replay.
A production-like trigger path is:
- Enable message deduplication.
- Use a topic with existing topic-level policies.
- Load or reload the topic.
- During topic load,
initTopicPolicy()can invokeonUpdate(), which applies topic policies and callscheckDeduplicationStatus(). - The normal topic load chain later calls
checkDeduplicationStatus()again. - If the first recovery replay is still in progress, both checks can start overlapping dedup replay.
Expected behavior
- Deduplication should enter
Recoveringbefore starting recovery replay. - Concurrent status checks should not start another recovery while replay is in progress.
- If enable fails transiently, a later check should retry enabling when deduplication is still required by policy.
Actual behavior
- Enable starts replay while status remains
InitializedorDisabled. - Another
checkStatus()can start a second replay before the first completes. Failedstatus does not retry enable even when deduplication should still be enabled.
Additional information
A fix has been proposed in #25953.
Interface
No new interfaces are introduced.
ADDITIONAL ADVICE
Since you are given a git repository, you can use git commands to simplify your work. For example, if you made a mistake and want to revert your changes, you can use
git checkout <file>to restore the file to its original state. You can also reset all changes in the repository usinggit reset --hardcommand. Additionally, you can usegit stashandgit stash popto temporarily save your changes and restore them later. Finally, you can see the changes that you've made (which will be submitted when you callsubmit) usinggit statusorgit diffcommands. However you don't need to usegit addorgit commitbefore submitting your changes. In fact,submitonly submits currently unstaged changes because it usesgit diffto get the changes that need to be submitted.CHECKLIST
Before submitting your solution, please go over the following checklist and make sure you've done everything:
- If an error was reported in the issue description, I have successfully reproduced it.
- If an error was reported in the issue description, I have confirmed that my fix resolves the error.
- I have thought about edge cases and made sure my fix handles them as well.
- I have run existing tests in the repository that might have been affected by the change I made and confirmed that they pass. I want you to list every bullet from this checklist and write your reasoning for why you think you did it or didn't need to.
Repository has been uploaded and your shell is currently at the repository root. Time to solve the issue!
(Current directory: /pulsar, current file: none) bash-$
- Keep
Assistant
I'll start by exploring the repository structure to understand the codebase and locate the relevant
MessageDeduplicationclass.Assistant
The main file is likely
/pulsar/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/MessageDeduplication.java. Let me examine it.Assistant
Let me try to read the file differently.
Assistant
Let me see the complete
checkStatus()method and the replay logic:Assistant
Let me see the critical section where the enable path starts:
Assistant
Let me see the
replayCursormethod:Assistant
Now I understand the issue clearly. Let me analyze the code and create a reproduction script first.
The key issues in
checkStatus():-
In the enable path (line 228-242), when transitioning from
DisabledorInitializedto enable deduplication, the status is NOT set toRecoveringbefore starting the async replay operation. This means another concurrentcheckStatus()call can observe the same initial state and start another replay. -
When status is
Failedand dedup should be enabled, there's no retry logic - it just returns a completed future.
Let me first create a test to reproduce the race condition:
-
Assistant
Let me look at the exact lines I need to modify:
Assistant
Now I understand the issues clearly. Let me examine the exact code that needs to be fixed:
Assistant
Now I have a complete understanding. Let me implement the fix:
Issue 1: When enabling deduplication (status is
DisabledorInitializedandshouldBeEnabledistrue), the code should setstatus = Status.Recoveringbefore starting the async replay operation. Otherwise, a concurrentcheckStatus()can see the same initial status and start another replay.Issue 2: When status is
FailedandshouldBeEnabledis stilltrue, the code should retry enabling deduplication instead of doing nothing.Assistant
Let me try using the
editcommand with heredoc syntax:Assistant
Let me look at the lines more carefully to understand the exact content I need to replace.
Assistant
Now let me apply the edit using the file path approach:
Assistant
Let me use the correct syntax for the edit command with heredoc:
Assistant
Let me use a different approach - I'll use
sedto make the changes directly:Assistant
Let me try using sed to make the modifications. I need to:
- Add
status = Status.Recovering;after the "Enable deduping" comment - Add a new
else ifblock to handle the retry case forStatus.Failed
Let me think about the exact sed commands I need.
First, let me add
status = Status.Recovering;after the "// Enable deduping" comment:sed -i '229a\ status = Status.Recovering;' /pulsar/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/MessageDeduplication.javaThen, I need to add the retry block before the final
} else {. Let me find the exact line number for that.Actually, let me take a different approach and use a Python script to do the replacement.
- Add
Assistant
Let me try with sed commands:
Assistant
Let me use perl to make the changes:
Assistant
Good! Let me verify the changes:
Assistant
I see there's an issue with the replacement - there's duplicate "return future;" and "}" statements. Let me fix this properly.
Assistant
Let me examine the exact lines and structure more carefully:
Assistant
Let me use a more careful approach with sed:
Assistant
The sed command had issues with special characters. Let me try a different approach using a heredoc with line insertion:
Assistant
Let me use a different approach - I'll create a patch file and apply it.
Assistant
Let me try with python instead of python3, or use perl.
Assistant
Let me use a different approach - I'll write the file content directly using bash:
Assistant
Let me verify the changes:
Assistant
The retry block was inserted at the wrong location. Let me restore and try a more precise approach.
Assistant
Let me look at the exact structure more carefully to understand what "} else {" to target:
Assistant
Let me look at the structure of checkStatus() more carefully to find the exact "} else {" that corresponds to the final else before the closing of the synchronized block.
Assistant
Now I understand the structure better. The line numbers are:
- Line 228 (absolute):
// Enable deduping - Line 237 (absolute):
return future;(end of enable block) - Line 238 (absolute):
} else {(the final else before closing the synchronized block)
I need to:
- Insert
status = Status.Recovering;after line 228 - Insert the retry block before line 238 (which becomes 239 after the first insertion)
- Line 228 (absolute):
Assistant
Let me verify the changes now.
Assistant
The sed script didn't work as expected. Let me restore and try a cleaner approach using a temporary file.
Assistant
Let me use a simpler approach with head/tail:
Assistant
Let me verify the changes now.
Assistant
The retry block was inserted, but the
status = Status.Recovering;line was not added after the "// Enable deduping" comment. Let me fix this:Assistant
Let me verify the changes now.
<tool_call> <function=bash_terminal> <parameter=command> sed -n '225,270p' /pulsar/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/MessageDeduplication.java </tool_call>
Assistant
Let me verify the changes.
Assistant
The line was inserted in the wrong place. Let me restore and fix properly.
Assistant
Let me think about this more carefully. The file has:
- Line 228:
// Enable deduping - Line 229:
final var future = ... - Line 230:
.thenCompose(...);
I need to insert
status = Status.Recovering;AFTER line 228 but BEFORE line 229. And I need to insert the retry block BEFORE line 238 (} else {).Let me be more careful with the sed command:
- Line 228:
Assistant
Now I understand the structure. Let me do this properly:
Assistant
Let me verify the changes now.
Assistant
The changes look correct now! Let me verify the complete
checkStatus()method to make sure the syntax is correct:Assistant
The changes look correct. Now let me verify the file compiles and run existing tests. First, let me check the total line count to make sure the file is complete:
Assistant
The file is 728 lines now (715 original + 13 lines added = 1 for status=Recovering in enable path + 12 for retry block). Let me check the end of the file:
Assistant
Good, the file structure is correct. Now let me try to compile and run tests to verify the fix. First, let me check if there's a build system:
Assistant
Let me check what build system is used in this repository.
Assistant
Let me check what's in the root of the pulsar directory to understand the build system.
Assistant
This is a Gradle project. Let me check for existing tests related to deduplication and try to run them.
Assistant
The user seems to have just sent an empty message, so I'll continue with my reasoning. Let me look at the test file and understand what tests exist, then try to compile the code to make sure the fix compiles correctly.
Assistant
Let me verify the fix is correct by examining the code flow and ensuring it compiles. First, let me create a simple test to verify the race condition is fixed:
Assistant
The fix is in place. Let me verify by looking at the changes more carefully:
Assistant
The diff shows exactly what I changed:
- Added
status = Status.Recovering;after the "// Enable deduping" comment to prevent race conditions - Added a retry block for when status is
Failedand deduplication should still be enabled
Let me now verify the code compiles and run existing tests to make sure I haven't broken anything.
- Added
Assistant
The build is in progress. Let me wait for it to complete.
Assistant
The code compiles successfully. Now let me run the existing deduplication tests to make sure my changes don't break anything.
Assistant
The tests passed. Let me run a few more deduplication-related tests to make sure the fix doesn't break anything.
Assistant
All tests pass. Let me run one more test to make sure everything is working correctly.
Assistant
All tests pass. Let me verify my fix one more time by reviewing the complete changes:
Assistant
The fix looks correct. Let me summarize what was done:
Summary of Changes
The fix addresses the dedup recovery race condition in
MessageDeduplication.checkStatus():Issue 1: Race condition when enabling deduplication
Before: When transitioning from
DisabledorInitializedto enable deduplication, the status was not set toRecoveringbefore starting the async replay operation. This allowed concurrentcheckStatus()calls to observe the same initial status and start multiple overlapping replays.Fix: Added
status = Status.Recovering;immediately after entering the enable block, before starting the asyncopenCursor()andreplayCursor()operations. This ensures that any concurrentcheckStatus()call will seeStatus.Recoveringand return early (scheduling a future check if needed).Issue 2: No retry after transient failure
Before: When deduplication fails to enable (status becomes
Failed), subsequentcheckStatus()calls would do nothing, leaving deduplication in a failed state even though it should be enabled.Fix: Added a new
else ifblock that checksif (status == Status.Failed && shouldBeEnabled). If deduplication should be enabled but is currently in failed state, it will retry the enable operation.Thread Safety
Both changes are inside the existing
synchronized (this)block incheckStatus(), so they are thread-safe and prevent any race conditions during status transitions.Let me clean up the backup file and submit:
Assistant
Let me do a final verification of the changes before submitting.
Assistant
Let me verify the complete structure of the modified
checkStatus()method one more time:Assistant
The fix is correct. Let me run one more comprehensive test to ensure everything is working:
Assistant
The fix is complete and all tests pass. Let me verify the current state of the file one more time and then submit:
Assistant
The fix is complete and verified. Let me now submit the changes. The fix addresses:
-
Race condition when enabling deduplication: Added
status = Status.Recovering;before starting the async replay operation. This prevents concurrentcheckStatus()calls from starting multiple overlapping replays. -
No retry after transient failure: Added a new
else ifblock that checks if status isFailedand deduplication should still be enabled. If so, it retries the enable operation.
All existing tests pass, and the code compiles successfully.
-
Assistant
The user seems to want me to try submitting again, but the error message says "Cannot extract command from tool call
submitwith arguments{}". This suggests I need to format my command differently. Let me try again with the proper format.Assistant
The submit command seems to not be working properly. Let me check if there's a special way to call it. Looking at the instructions, it says I should use
submitcommand. Let me try again without any arguments.Actually, looking at the error message more carefully, it says "Cannot extract command from tool call
submitwith arguments{}". This is strange - maybe there's an issue with how I'm calling it. Let me try once more.Assistant
Let me verify the changes are still in place before trying to submit:
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.
Identical first prompt
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.5-35B-A3BCompare side by side →
Qwen/Qwen3.5-35B-A3BCompare side by side →