ISSUE DESCRIPTION OPD losses error for Megatron when i use megatron for compute losses…
- User
ISSUE DESCRIPTION
OPD losses error for Megatron when i use megatron for compute losses the code position is: /verl/trainer/distillation/megatron/losses.py Line 248 teacher_topk_log_probs_cp_split, *_ = preprocess_bshd_engine(teacher_topk_log_probs, pre_process=True)
the teacher_topk_log_probs is [b, length, 64] but preprocess_bshd_engine this function need [n, length] dim, so error happens
Interface
No new interfaces are introduced.
ADDITIONAL ADVICE
Since you are given a git repository, you can use git commands to simplify your work. For example, if you made a mistake and want to revert your changes, you can use
git checkout <file>to restore the file to its original state. You can also reset all changes in the repository usinggit reset --hardcommand. Additionally, you can usegit stashandgit stash popto temporarily save your changes and restore them later. Finally, you can see the changes that you've made (which will be submitted when you callsubmit) usinggit statusorgit diffcommands. However you don't need to usegit addorgit commitbefore submitting your changes. In fact,submitonly submits currently unstaged changes because it usesgit diffto get the changes that need to be submitted.CHECKLIST
Before submitting your solution, please go over the following checklist and make sure you've done everything:
- If an error was reported in the issue description, I have successfully reproduced it.
- If an error was reported in the issue description, I have confirmed that my fix resolves the error.
- I have thought about edge cases and made sure my fix handles them as well.
- I have run existing tests in the repository that might have been affected by the change I made and confirmed that they pass. I want you to list every bullet from this checklist and write your reasoning for why you think you did it or didn't need to.
Repository has been uploaded and your shell is currently at the repository root. Time to solve the issue!
(Current directory: /verl, current file: none) bash-$
Assistant
Let me start by exploring the repository structure to understand the codebase.
Assistant
Now let me look at the
preprocess_bshd_enginefunction to understand its input expectations.Assistant
Let me see how
preprocess_thd_enginehandles similar input:Assistant
Let me look at how
teacher_topk_log_probsis constructed in the codebase:Assistant
Let me look at the FSDP version to see how it handles CP:
Assistant
Now I see the FSDP version uses
values().unsqueeze(0)to convert the nested tensor to a dense tensor, then usesslice_input_tensorto split it along dimension 1 (sequence).Let me look at
slice_input_tensorto understand it:Assistant
Let me look at existing tests to understand the expected behavior better:
Assistant
Let me check the test_distillation_topk_symmetry_on_cpu.py:
Assistant
Now let me check the
_nested_from_rowsto understand the format:Assistant
Let me find where the teacher_topk_log_probs is constructed:
Assistant
Let me look at where teacher_logprobs are constructed:
Assistant
Let me look at how the teacher_logprobs becomes a nested tensor:
Assistant
Let me look at the megatron CP code:
Assistant
Let me look at preprocess_bshd_engine more carefully, specifically the CP splitting part:
Assistant
Let me look at preprocess_thd_engine more carefully:
Assistant
Let me check the nested tensor API:
Assistant
Let me check where these preprocess functions are used:
Assistant
The preprocess functions are used in:
model_forward.py- for general model forward, where input is 1D (token IDs).model_forward_fused.py- for fused model forward, where input is 1D (token IDs).trainer/distillation/megatron/losses.py- THIS IS WHERE THE BUG IS. The input is 2D (topk logprobs/ids).router_replay_utils.py- need to check.
Let me look at router_replay_utils.py:
Assistant
So
preprocess_thd_engineis used inrouter_replay_utils.pywherelayers_topk_idxis a nested tensor that includes a feature dim. Let me check this:Assistant
OK, in router_replay_utils.py,
layers_topk_idxis converted to a dense tensor first (line 248:gather_from_sequence_parallel_region(layers_topk_idx, ...).unsqueeze(0).contiguous()), so it's not a nested tensor by the time preprocess_thd_engine is called.But wait, line 290 checks
if layers_topk_idx.is_nested:. Let me look more carefully:Assistant
Let me look at the existing test for this code:
Assistant
Let me first write a reproducer to confirm the bug:
Assistant
The megatron module is not installed. Let me check what's available:
Assistant
Megatron isn't installed. Let me read the preprocess function and analyze it carefully to confirm the bug:
Assistant
Let me trace back to understand the data shape better:
Assistant
Let me look at the data structure more carefully. Let me create a comprehensive test script to understand:
Assistant
Let me also check the layers_topk_idx case:
Assistant
Let me check the
router_replay_utils.pyto see if it would also have this issue:Assistant
Looking at this, the comment says "shape [bs, max_seq_len, layer_num, topk]". But if
layers_topk_idx.is_nestedis True, it means the data is a nested tensor with a jagged dim at position 1 (the seq_len dim). The inner shape would be(seq_len_i, layer_num, topk).So this code path has the same bug. However, the actual data shape might differ - let me check more carefully:
Assistant
Let me look at the structure of the losses file to understand how to add the fix:
Assistant
Let me write a test to confirm the issue with bshd:
Assistant
Now let me also test the thd function:
Assistant
Now I have a clear picture. The issue is specifically with
preprocess_bshd_engine. The thd function already handles 2D nested tensors correctly. Let me implement the fix:Assistant
Now I have a clear understanding of the issue and the fix. Let me apply the fix to
preprocess_bshd_engineinutil.py:Assistant
Now let me apply the fix:
Assistant
Let me now also think about the thd function with cp_size > 1. Let me write a test for the cp_size > 1 case:
Assistant
The thd function works for 2D nested tensors. So I only need to fix the bshd function. Let me apply the fix:
Assistant
Now let me fix the
seq_paddedallocation:Assistant
I have a duplicate
seq = input_ids[i]line. Let me fix that:Assistant
Let me delete the duplicate line:
Assistant
The fix is applied. Now let me also test the fix by simulating the function logic:
Assistant
Now let me verify the fix is properly in the file:
Assistant
Let me also verify the thd function works for 2D:
Assistant
Now let me check the postprocess_bshd_engine function to make sure it works with 2D outputs:
Assistant
The postprocess function should also work for 2D. Let me now also think about whether the thd function needs a similar fix:
Assistant
Let me also think about whether there are any other places that might have issues. Let me look at the need_roll case for thd:
Assistant
Let me now also check the existing test setup and run a quick check that the code is syntactically valid:
Assistant
Let me also look at the existing test for the megatron KL loss to see if it tests the BSHD case:
Assistant
The existing test only tests the "thd" case. Let me look at the data flow more carefully:
Assistant
Let me also run the existing test to make sure I didn't break anything:
Assistant
Let me also update the docstring of the function to reflect the new behavior:
Assistant
Let me use a different approach with
replace:Assistant
Let me use a heredoc to handle the special characters:
Assistant
Now let me also verify the syntax is OK:
Assistant
Now let me also write a more comprehensive test to ensure the fix works in all scenarios:
Assistant
All tests pass! Let me also verify by looking at the diff:
Assistant
Let me also verify by looking at the issue once more and checking if my fix addresses the issue:
Assistant
Now let me also think about the thd function in the thd case. Let me check if the thd function's existing logic for 2D nested tensors is correct. Let me check what happens when the input is 2D in thd:
Assistant
Test 3 had a wrong expected shape. Let me fix the test:
Assistant
Hmm, the actual shape is [6, 64], not [8, 64]. Let me check the function logic again:
align_size = tp_size * cp_size * 2 if cp_size > 1 else tp_sizeWith tp_size=1, cp_size=2, align_size = 122 = 4.
seqlens_in_batch = input_ids.offsets().diff() pad_size = (align_size - seqlens_in_batch % align_size) % align_size seqlens_in_batch_padded = seqlens_in_batch + pad_sizeseqlens_in_batch = [4, 6] pad_size = (4 - 4%4) % 4, (4 - 6%4) % 4 = 0, 2 seqlens_in_batch_padded = [4, 8]
cu_seqlens_padded = [0, 4, 12] sum(seqlens_in_batch_padded_cpu) = 4 + 8 = 12 shape[0] = 12 / 2 = 6
So the actual shape is [6, 64]. OK my test had wrong calculation. Let me also try with tp_size=2:
With tp_size=2, cp_size=2, align_size = 222 = 8. pad_size = (8 - 4%8) % 8, (8 - 6%8) % 8 = 4, 2 seqlens_in_batch_padded = [8, 8] sum = 16 shape[0] = 16/2 = 8
So with tp_size=2, the shape would be [8, 64]. Let me update the test:
Assistant
Good, the thd function works correctly for 2D nested tensors. Now let me also run the existing test to make sure my fix doesn't break anything. Let me look at the test:
Assistant
The test requires distributed mode and Megatron. Let me also check if I can run any CPU tests:
Assistant
Let me try running this test to see if it works:
Assistant
The existing tests pass. Let me also run the other related tests:
Assistant
All relevant tests pass. Let me also run the other tests in the workers folder:
Assistant
All 30 tests pass. Let me also try the models tests:
Assistant
Some tests can't run because of missing dependencies. That's not related to my fix. Let me focus on tests that can run:
Assistant
These tests can't run because of missing dependencies. Let me run only the tests that I can:
Assistant
All 88 tests pass. My fix doesn't break any existing tests. Let me also clean up the temporary test files:
Assistant
Good, all temporary files are cleaned up. Let me verify my final fix:
Assistant
Let me also re-verify my fix by looking at the relevant code:
Assistant
The fix is complete. Let me do a final review:
Assistant
Let me do a final check on the diff and review:
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.
Identical first prompt
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.5-35B-A3BCompare side by side →
Qwen/Qwen3.5-35B-A3BCompare side by side →