ISSUE DESCRIPTION Automatic managed-ledger offload triggers create redundant retry work…
- User
ISSUE DESCRIPTION
Automatic managed-ledger offload triggers create redundant retry work
Coalesce automatic offload triggers while an automatic offload is running. There must be at most one active automatic run and at most one follow-up run when additional triggers arrive. A follow-up must still process newly eligible ledgers, and explicit/manual offload futures must retain their existing behavior.
Interface
No new public interfaces are introduced.
ADDITIONAL ADVICE
Since you are given a git repository, you can use git commands to simplify your work. For example, if you made a mistake and want to revert your changes, you can use
git checkout <file>to restore the file to its original state. You can also reset all changes in the repository usinggit reset --hardcommand. Additionally, you can usegit stashandgit stash popto temporarily save your changes and restore them later. Finally, you can see the changes that you've made (which will be submitted when you callsubmit) usinggit statusorgit diffcommands. However you don't need to usegit addorgit commitbefore submitting your changes. In fact,submitonly submits currently unstaged changes because it usesgit diffto get the changes that need to be submitted.CHECKLIST
Before submitting your solution, please go over the following checklist and make sure you've done everything:
- If an error was reported in the issue description, I have successfully reproduced it.
- If an error was reported in the issue description, I have confirmed that my fix resolves the error.
- I have thought about edge cases and made sure my fix handles them as well.
- I have run existing tests in the repository that might have been affected by the change I made and confirmed that they pass. I want you to list every bullet from this checklist and write your reasoning for why you think you did it or didn't need to.
Repository has been uploaded and your shell is currently at the repository root. Time to solve the issue!
(Current directory: /pulsar, current file: none) bash-$
Assistant
Let me start by exploring the repository structure to understand what we're working with.
Assistant
This is the Apache Pulsar repository. The issue is about managed-ledger offload. Let me find the relevant offload code.
Assistant
Now I understand the issue. When
maybeOffloadInBackgroundis called repeatedly (e.g., on every append, trim, or other event), and an offload is already running (offloadMutex.tryLock()fails), the current code schedules anothermaybeOffloadInBackgroundafter 100ms. This creates redundant retry work — multiple scheduled retries piling up.Let me look at how
maybeOffloadInBackgroundis called.Assistant
Now let me look at the manual offload entry point and the offloadLoop.
Assistant
Now let me understand the full picture. The
maybeOffloadInBackgroundis called withNULL_OFFLOAD_PROMISEfor automatic triggers. When an offload is already in progress (!offloadMutex.tryLock()), it schedules a retry after 100ms. Multiple retries pile up.The fix:
- For automatic triggers, coalesce them: only one active run and at most one follow-up run.
- The follow-up must process newly eligible ledgers.
- Explicit/manual offload futures must retain their existing behavior.
Let me look at the NULL_OFFLOAD_PROMISE definition and the manual offload method (the one that uses
offloadMutex.tryLock()and returns OffloadInProgressException).The manual offload entry point is the
offloadmethod with a callback. Let me find it.Assistant
Let me look at the test that uses
maybeOffloadInBackgroundwith a real promise.Assistant
Let me check the executor used by ManagedLedgerImpl.
Assistant
executorisbookKeeper.getMainWorkerPool().chooseThread(name)— this is an ordered executor per name, so all operations for a given managed ledger run on the same thread, one at a time. This means operations submitted toexecutorare serialized. ThemaybeOffloadruns on this executor. TheoffloadLoopcallbacks (thenCompose etc.) run on... let me check. The offload callbacks usescheduledExecutorfor retries. ThewhenCompleteof the offload chain — which thread? CompletableFuture callbacks run on the thread that completes them, unless an executor is specified. The offload operationconfig.getLedgerOffloader().offload(...)completes on some thread (bookkeeper worker). Then the subsequent callbacks run on that thread. SooffloadLooprecursion happens on various threads.But the
unlockingPromise.whenComplete(which unlocks the mutex) — when is it completed? It's completed byoffloadLoop'spromise.complete/promise.completeExceptionally, which happens on whichever thread processes the last ledger. So the unlock happens on a bookkeeper worker thread, not the managed ledger executor.So thread safety matters. The
offloadMutexis aCallbackMutex. Let me look at CallbackMutex.Assistant
Now let me look at the existing tests to understand expected behaviors.
Assistant
Let me check the context where
maybeOffloadInBackgroundis called at line 1971.Assistant
Let me search for any existing tests or references to coalescing/follow-up behavior.
Assistant
Let me run the existing offload tests to establish a baseline.
Assistant
Let me check all usages of offloadMutex to understand all the paths that hold it.
Assistant
Two paths acquire offloadMutex:
maybeOffload(automatic, line 2836) — unlocks in unlockingPromise.whenComplete (line 2844).- Manual
offloadmethod (line 3644) — unlocks in promise.whenComplete (line 3651).
Let me check ShadowManagedLedgerImpl for offload overrides.
Assistant
I now have a clear understanding. Let me design the solution:
- Add a
pendingAutoOffloadflag, manipulated only on the serialized executor thread (inmaybeOffload), so it's race-free. - When an automatic trigger (
NULL_OFFLOAD_PROMISE) finds the mutex held, setpendingAutoOffload = trueinstead of scheduling a redundant retry. (Explicit/manual futures keep the retry.) - On every offload completion (both the automatic
maybeOffloadpath and the manualoffloadpath), schedule a follow-up check on the executor that, ifpendingAutoOffloadis set, clears it and triggers exactly one follow-up run.
This ensures at most one active automatic run and at most one follow-up, while manual/explicit behavior is preserved. Let me re-read the exact code to edit.
- Add a
Assistant
Let me see the full manual offload method to find its exact boundaries.
Assistant
Now I have a complete understanding. Let me implement the solution. First, add the
pendingAutoOffloadfield near the other offload fields.Assistant
No python3. Let me use the edit command.
Assistant
Now let me edit the
maybeOffloadmethod's tryLock-fail path and the completion callback. Let me find the current line numbers.Assistant
Now add the
scheduleAutoOffloadFollowupIfNeededhelper method aftermaybeOffload.Assistant
I left out the closing brace of the new method. Let me fix it.
Assistant
Let me add a blank line for readability between the methods.
Assistant
Now update the manual offload (
asyncOffloadPrefix) completion path to also schedule the follow-up check. Let me find the current line numbers.Assistant
There's a duplicate
});at line 3699. Let me remove it.Assistant
Now let me compile the managed-ledger module to check for errors.
Assistant
Compiles. Now let me run the existing offload tests to verify nothing breaks.
Assistant
All existing tests pass. Now let me write a temporary test to verify the coalescing behavior works correctly. Let me look at the test base class setup.
Assistant
Let me look at the MockLedgerOffloader and buildEntry helpers in OffloadPrefixTest.
Assistant
Let me create a temporary test to verify the coalescing behavior using reflection on the new
pendingAutoOffloadflag.Assistant
I referenced a non-existent public helper. Let me fix the test to define its own
buildEntry.Assistant
Wait, the sed for adding the method might have messed up the class declaration indentation. Let me check the file.
Assistant
The
buildEntrymethod ended up outside the class. Let me rewrite the file cleanly.Assistant
The test passes. Let me also verify that an explicit future via
maybeOffloadInBackgroundretains its retry behavior (completes after the running offload finishes). Let me add a second test method.Assistant
The appended method went outside the class. Let me rewrite the whole file with both methods properly inside.
Assistant
Both tests pass. Now let me run the broader managed-ledger test suite to ensure nothing else breaks.
Assistant
All pass. Let me also check for broker-level offload tests that might be affected.
Assistant
Let me look at the test that exercises offload triggering in AdminApiOffloadTest, around line 447 (the offload test with threshold).
Assistant
Let me run the full managed-ledger test suite to be thorough.
Assistant
Full managed-ledger test suite passes. Now let me remove the temporary test file before submitting.
Assistant
The full managed-ledger test suite passes. Now let me remove the temporary test file before submitting.
Assistant
Only the source file is modified. Let me review the full diff to ensure correctness.
Assistant
The diff looks clean and correct. Let me verify
maybeOffloadis only ever invoked via the executor (so the flag access is truly single-threaded).Assistant
Confirmed:
maybeOffloadis only ever invoked viaexecutor.execute(...)(line 2822), andscheduleAutoOffloadFollowupIfNeeded's lambda also runs on the executor. SopendingAutoOffloadis accessed only on the single per-ledger executor thread → race-free, as documented.The implementation is complete and verified. Let me summarize the solution and run through the checklist.
Summary of changes
In
ManagedLedgerImpl.java:-
Added a
pendingAutoOffloadflag (accessed only on the managed-ledger's single-threaded executor) that records whether a follow-up automatic offload run is needed. -
maybeOffload(automatic trigger path): WhentryLock()fails for an automatic trigger (NULL_OFFLOAD_PROMISE), instead of scheduling a redundant 100ms retry onscheduledExecutor(which previously cascaded into N retry chains when many triggers arrived during one running offload), it now just setspendingAutoOffload = true(coalescing — idempotent, so any number of triggers collapse to a single pending follow-up). Explicit/manual futures (finalPromise != NULL_OFFLOAD_PROMISE) retain the existing retry behavior. -
scheduleAutoOffloadFollowupIfNeeded(): A new helper invoked from both offload completion paths (the automaticmaybeOffload'sunlockingPromisecompletion and the manualasyncOffloadPrefix'spromisecompletion). It runs on the executor; ifpendingAutoOffloadis set it clears it and schedules exactly onemaybeOffloadInBackground(NULL_OFFLOAD_PROMISE)run, which re-evaluates thresholds and processes any newly eligible ledgers.
This guarantees at most one active automatic run (mutex) and at most one follow-up run when additional triggers arrive, while manual/explicit offload behavior is unchanged (manual
offloadPrefixstill throwsOffloadInProgressExceptionimmediately; explicit futures still retry to completion).Checklist reasoning
- Reproduced the issue: The issue describes redundant retry work (no stack trace). I wrote a temporary test using a slow offloader with many ledger rolls during a running automatic offload; it confirmed the new coalescing flag is set (coalescing occurred), the single follow-up processes newly eligible ledgers, and the flag clears afterward. (Removed before submitting.)
- Confirmed the fix resolves it: The temp test passed with the fix; with the old code the
pendingAutoOffloadmechanism didn't exist (the old code scheduled a retry per trigger). ExistingmultipleAutoTriggers/offloadAsSoonAsClosed/manualTriggerWhileAutoInProgress/autoTriggerWhileManualInProgresstests confirm correctness is preserved. - Edge cases: explicit/manual futures retain retry; manual offload still fails fast with
OffloadInProgressException; both completion paths schedule the follow-up; flag is single-threaded (race-free); early-return paths don't acquire the mutex; the chain terminates (an acquiring run never sets the flag, so no infinite follow-up loop); follow-up re-evaluates thresholds for newly eligible ledgers. - Ran affected tests: Full
:managed-ledger:testsuite passes (BUILD SUCCESSFUL), including all offload-related tests andManagedLedgerTest.
Let me do a final verification that the working tree only contains the intended change.
-
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.
Identical first prompt
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-27BCompare side by side →
Qwen/Qwen3.6-27BCompare side by side →