ISSUE DESCRIPTION OPD losses error for Megatron when i use megatron for compute losses…
- User
ISSUE DESCRIPTION
OPD losses error for Megatron when i use megatron for compute losses the code position is: /verl/trainer/distillation/megatron/losses.py Line 248 teacher_topk_log_probs_cp_split, *_ = preprocess_bshd_engine(teacher_topk_log_probs, pre_process=True)
the teacher_topk_log_probs is [b, length, 64] but preprocess_bshd_engine this function need [n, length] dim, so error happens
ADDITIONAL ADVICE
Since you are given a git repository, you can use git commands to simplify your work. For example, if you made a mistake and want to revert your changes, you can use
git checkout <file>to restore the file to its original state. You can also reset all changes in the repository usinggit reset --hardcommand. Additionally, you can usegit stashandgit stash popto temporarily save your changes and restore them later. Finally, you can see the changes that you've made (which will be submitted when you callsubmit) usinggit statusorgit diffcommands. However you don't need to usegit addorgit commitbefore submitting your changes. In fact,submitonly submits currently unstaged changes because it usesgit diffto get the changes that need to be submitted.CHECKLIST
Before submitting your solution, please go over the following checklist and make sure you've done everything:
- If an error was reported in the issue description, I have successfully reproduced it.
- If an error was reported in the issue description, I have confirmed that my fix resolves the error.
- I have thought about edge cases and made sure my fix handles them as well.
- I have run existing tests in the repository that might have been affected by the change I made and confirmed that they pass. I want you to list every bullet from this checklist and write your reasoning for why you think you did it or didn't need to.
Repository has been uploaded and your shell is currently at the repository root. Time to solve the issue!
(Current directory: /verl, current file: none) bash-$
Assistant
Let me start by understanding the issue. The problem is in
/verl/trainer/distillation/megatron/losses.pyat line 248, whereteacher_topk_log_probshas shape[b, length, 64](3D), butpreprocess_bshd_engineexpects[n, length](2D). Let me explore the repository structure and understand the code.Assistant
Let me look at the specific file mentioned in the issue.
Assistant
Now let me look at the
preprocess_bshd_engineandpreprocess_thd_enginefunctions to understand their expected input shapes.Assistant
Let me look at how
compute_forward_kl_topkis called to understand the data flow betterAssistant
Let me look at the test file and the calling code to understand the expected data
Assistant
Let me look at how
compute_forward_kl_topkis called in production code:Assistant
Let me look at the FSDP version for comparison:
Assistant
Now let me look at `preprocess_thd_engine
Assistant
Let me look at the
preprocess_thd_enginefunction:Assistant
Let me trace where
teacher_logprobsandteacher_idsare created:Assistant
Let me look at how teacher outputs are padded:
Assistant
Let me find the correct path.
Assistant
Now let me look at how the outputs are batched in agent_loop.py
Assistant
Now I understand the issue. Let me create a reproduction script first to confirm the error.
Assistant
I've confirmed the error. Now let me fix
preprocess_bshd_engineto handle multi-dimensional nested tensors. Let me look at the full function one more timeAssistant
Let me open the file and see the function:
Assistant
Let me use the right command format.
Assistant
Let me see the remaining lines:
Assistant
Now I have a clear understanding of the issue. The
preprocess_bshd_enginefunction only handles 2D nested tensors[b, jagged](scalars), but it receives 3D nested tensors[b, jagged, topk]from the teacher log probs. I need to modify it to preserve trailing dimensions, similar to howpreprocess_thd_enginealready does.Let me make the fixes
Assistant
Let me use
replaceinstead since it's more reliable with the exact text matchingAssistant
Let me try with heredoc syntax:
Assistant
Let me use a Python script to make the changes instead:
Assistant
Let me verify the changes:
Assistant
Let me verify that
expand_aswould fail with mismatched dimensions:Assistant
Let me check all callers of
preprocess_bshd_engineto understand usage:Assistant
Let me look at how
preprocess_bshd_engineis used inmodel_forward.pyto understand position_ids usage:Assistant
I need to fix the
expand_ascalls for position_ids to work with multi-dimensionalinput_ids_bshd. Let me update both expand_as calls:Assistant
Let me verify the final state of the changes:
Assistant
Now let me test with the reproduction script to confirm the fix works:
Assistant
Good, the basic logic works. Now let me run the existing tests to make sure nothing is broken:
Assistant
Let me run the distillation-related tests:
Assistant
Let me also run any other related tests I can find:
Assistant
The megatron test can't run because it requires
megatron.corewhich is not installed in this environment. That's expected - it's a distributed test.Let me run any other unit tests that might be affected:
Assistant
The timeout option isn't there. Let me just run the tests without timeout.
Assistant
Let me verify the full diff of my changes:
Assistant
Let me also think about whether the
_VocabParallelKLDivergencecan handle the 3D output frompreprocess_bshd_engine. Let me trace through the forward functionAssistant
That's expected - the random indices go out of bounds. Let me fix the test:
Assistant
All changes are verified and working. Let me run the distillation tests one more time to be sure:
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.
Identical first prompt
Qwen/Qwen3.6-27BCompare side by side →
Qwen/Qwen3.6-27BCompare side by side →
Qwen/Qwen3.6-27BCompare side by side →
Qwen/Qwen3.6-27BCompare side by side →
Qwen/Qwen3.6-27BCompare side by side →
- xiaomi/mimo-v2.5-proCompare side by side →
- xiaomi/mimo-v2.5-proCompare side by side →
- xiaomi/mimo-v2.5-proCompare side by side →
- xiaomi/mimo-v2.5-proCompare side by side →
- gpt-5.6-solCompare side by side →
- gpt-5.6-solCompare side by side →
- gpt-5.6-solCompare side by side →