Skip to content
TheChatArchiveThe Chat Archive
MiniMaxminimax/minimax-m3· unknownHugging Face ibragim-bad/swe_rebench_07_2026_trajectories

ISSUE DESCRIPTION waza check eval discovery is inconsistent with coverage and run in…

  1. User

    ISSUE DESCRIPTION

    waza check eval discovery is inconsistent with coverage and run in multi-skill workspaces

    Summary

    In a multi-skill workspace, waza coverage . and waza run evals/.../eval.yaml both recognize the eval structure, but waza check does not discover the related eval suite consistently.

    This creates a confusing user experience because the workspace appears valid for coverage and execution, but readiness checks report that no eval exists or fail to find the skill.

    Environment

    • Waza version: 0.31.0
    • OS: Windows
    • Shell: PowerShell

    Workspace shape

    This was reproduced in a repository with this structure:

    repo/
    ├── .waza.yaml
    ├── skills/
    │   ├── development/skill-creator/SKILL.md
    │   ├── copilot-exclusive/mcp-ecosystem/SKILL.md
    │   ├── copilot-exclusive/fleet-parallel/SKILL.md
    │   ├── testing/eval-harness/SKILL.md
    │   └── copilot-exclusive/background-agent/SKILL.md
    └── evals/
        ├── skill-creator/eval.yaml
        ├── mcp-ecosystem/eval.yaml
        ├── fleet-parallel/eval.yaml
        ├── eval-harness/eval.yaml
        └── background-agent/eval.yaml
    

    The root .waza.yaml includes:

    paths:
      skills: skills/
      evals: evals/
      results: results/
    
    server:
      resultsDir: results
    

    Steps to reproduce

    From the workspace root:

    waza coverage .
    waza check skills/development/skill-creator
    waza check skill-creator
    waza run evals/skill-creator/eval.yaml --task create-dev-skill-001 --trials 1 --output-dir results -v
    

    Actual behavior

    waza coverage .

    Coverage recognizes the pilot evals and shows the skill as partially covered.

    Example outcome:

    • skill-creator appears with 2 tasks
    • other pilot skills also appear in the coverage grid

    waza run evals/skill-creator/eval.yaml ...

    Run executes successfully enough to:

    • load the skill directories
    • run the selected task
    • apply graders
    • save a result JSON file under results/...

    waza check skills/development/skill-creator

    Output includes:

    Evaluation Suite: Not Found
    

    waza check skill-creator

    Output includes:

    no SKILL.md found in <workspace root>
    

    Expected behavior

    waza check should discover the corresponding eval suite consistently when:

    1. the workspace has a valid root .waza.yaml
    2. the skill exists under skills/.../<skill-name>/SKILL.md
    3. the eval exists under evals/<skill-name>/eval.yaml
    4. coverage and run already succeed against the same workspace

    At minimum, check should either:

    • locate the matching eval and report it, or
    • explain the discovery rule clearly enough that the user can resolve the mismatch

    Why this matters

    For new users, check looks like the natural readiness command. When it disagrees with coverage and run, it is hard to tell whether:

    • the workspace is invalid
    • the eval naming/path convention is wrong
    • or check is using different discovery rules

    This makes initial adoption harder, especially in multi-skill repositories.

    Additional note

    This may be either:

    • a discovery bug in check, or
    • a documentation/UX gap where check expects a different invocation pattern than users would infer from coverage and run

    Either way, the current behavior is surprising.

    Possible improvements

    Any of these would help:

    1. Make check reuse the same discovery logic as coverage
    2. Document the exact resolution rule from skill path/name to eval path
    3. Add debug output showing where check looked for the eval
    4. Improve the error message when the skill name/path is interpreted differently than expected

    Interface

    No new interfaces are introduced.

    ADDITIONAL ADVICE

    Since you are given a git repository, you can use git commands to simplify your work. For example, if you made a mistake and want to revert your changes, you can use git checkout <file> to restore the file to its original state. You can also reset all changes in the repository using git reset --hard command. Additionally, you can use git stash and git stash pop to temporarily save your changes and restore them later. Finally, you can see the changes that you've made (which will be submitted when you call submit) using git status or git diff commands. However you don't need to use git add or git commit before submitting your changes. In fact, submit only submits currently unstaged changes because it uses git diff to get the changes that need to be submitted.

    CHECKLIST

    Before submitting your solution, please go over the following checklist and make sure you've done everything:

    • If an error was reported in the issue description, I have successfully reproduced it.
    • If an error was reported in the issue description, I have confirmed that my fix resolves the error.
    • I have thought about edge cases and made sure my fix handles them as well.
    • I have run existing tests in the repository that might have been affected by the change I made and confirmed that they pass. I want you to list every bullet from this checklist and write your reasoning for why you think you did it or didn't need to.

    Repository has been uploaded and your shell is currently at the repository root. Time to solve the issue!

    (Current directory: /waza, current file: none) bash-$

  2. MiniMax Assistant

    Let me start by exploring the repository structure to understand the codebase.

  3. MiniMax Assistant

    Let me explore the main source files and understand the structure better.

  4. MiniMax Assistant

    Let me look at the key files: cmd_check.go, cmd_coverage.go, cmd_run.go, and workspace_resolve.go to understand how they discover evals.

  5. MiniMax Assistant

    Let me see the workspace_resolve.go file and find the checkReadiness function.

  6. MiniMax Assistant

    Now let me look at the workspace package to understand the FindEval function.

  7. MiniMax Assistant

    Now let me look at how coverage and run discover evals to understand the discrepancy.

  8. MiniMax Assistant

    Let me see the run command to understand how it discovers evals.

  9. MiniMax Assistant

    Let me look at how coverage and run find the skills to understand why they work.

  10. MiniMax Assistant

    Let me look at the existing tests to understand the expected behavior.

  11. MiniMax Assistant

    Let me first reproduce the issue with a test workspace before making any changes.

  12. MiniMax Assistant

    Coverage finds all 5 skills. Now let me test waza check.

  13. MiniMax Assistant

    Confirmed issue 1. Now let me test the other case.

  14. MiniMax Assistant

    Both issues are confirmed. Let me also examine the run command flow more carefully.

  15. MiniMax Assistant

    Let me look at more of checkReadiness and the tests for it.

  16. MiniMax Assistant

    Let me look at more of the scanForSkills and how it's called.

  17. MiniMax Assistant

    Let me look at how the run command deals with the same issue.

  18. MiniMax Assistant

    Let me look at the projectconfig to understand the structure.

  19. MiniMax Assistant

    Let me check the existing cmd_run_workspace_test.go to ensure I don't break it.

  20. MiniMax Assistant

    Now I have a good understanding. Let me design the fix.

    Plan

    1. Make scanForSkills in /waza/internal/workspace/workspace.go recursive. This fixes the discovery of nested skills (e.g., skills/development/skill-creator/SKILL.md), making waza check consistent with waza coverage which uses filepath.WalkDir.

    2. Pass configDetectOptions() to the fallback DetectContext call in checkReadiness. This ensures the user's project config (.waza.yaml) is respected when looking for the eval from a path-based invocation.

    3. Improve error messages when a skill is not found or eval is not found. Provide actionable information so users can diagnose the issue.

    Let me check the current state of the test for nested skills to make sure the test would still pass.

  21. MiniMax Assistant

    Now let me look at the current scanForSkills to understand its structure better.

  22. MiniMax Assistant

    Now let me make the changes. First, I'll modify scanForSkills to be recursive.

  23. MiniMax Assistant

    Let me run the existing tests to make sure I didn't break anything.

  24. MiniMax Assistant

    Let me look at the failing test.

  25. MiniMax Assistant

    I need to be more careful. The recursive scan is needed for configured skills dirs (step 3), but step 4 should still scan only immediate children (fallback to root). Let me refactor properly.

  26. MiniMax Assistant

    All existing tests pass. Now let me verify the changes work for the user's case.

  27. MiniMax Assistant

    waza check skill-creator now finds the eval. Let me also test the path-based invocation.

  28. MiniMax Assistant

    Both cases work. Now let me test the case with custom paths to verify the second issue.

  29. MiniMax Assistant

    The test used the wrong path. Let me use the correct path.

  30. MiniMax Assistant

    The path-based invocation still fails for custom paths. Let me fix the fallback in checkReadiness to pass config options.

  31. MiniMax Assistant

    All three test cases work. Let me run all the existing tests to make sure I didn't break anything.

  32. MiniMax Assistant

    All tests pass. Now let me also look at the related test cases in the broader test suite.

  33. MiniMax Assistant

    All tests pass. Now let me also consider improving the error messages when the skill is not found.

  34. MiniMax Assistant

    Let me also improve the error handling when a skill name is provided but not found.

  35. MiniMax Assistant

    Now let me also test the case where a non-existent skill is requested:

  36. MiniMax Assistant

    A better error message. Now let me run all the tests.

  37. MiniMax Assistant

    All tests pass. Now let me also verify the original test cases in the issue work as expected.

  38. MiniMax Assistant

    Let me also test waza check for all 5 skills in the workspace.

  39. MiniMax Assistant

    All 5 skills are found and their evals are detected. Now let me also verify the JSON output works correctly.

  40. MiniMax Assistant

    The eval is correctly found. Now let me also test the case where the user provides just a skill name with the path-based argument.

  41. MiniMax Assistant

    Let me also test the case where the user provides a relative path to a skill:

  42. MiniMax Assistant

    Let me also check what happens with a custom paths config.

  43. MiniMax Assistant

    Both the skill-name and path-based invocations correctly find the eval in the custom-paths workspace.

    Now let me look at the edge cases I should think about.

  44. MiniMax Assistant

    Let me check if there are other places that use scanForSkills that might be affected.

  45. MiniMax Assistant

    Let me also verify the existing check tests still work.

  46. MiniMax Assistant

    Let me now create a temporary test script to verify all the issue scenarios in detail.

  47. MiniMax Assistant

    The shell script is being aborted by set -e when waza exits with non-zero (e.g., for the nonexistent skill). Let me remove set -e and test more carefully.

  48. MiniMax Assistant

    All scenarios work correctly. Let me also look at the final diff to make sure everything is in order.

  49. MiniMax Assistant

    Let me run all tests one more time to make sure everything passes.

  50. MiniMax Assistant

    All tests pass. Let me also run a comprehensive verification of the fix.

  51. MiniMax Assistant

    All 10 verification tests pass. Let me clean up the temporary files before submitting.

  52. MiniMax Assistant

    Now let me also re-check the diff and verify the changes are complete.

  53. MiniMax Assistant

    Let me run the full test suite one more time to confirm everything is good.

  54. MiniMax Assistant

    All tests pass. Let me also verify the fix one more time by rebuilding the binary and testing the exact scenarios from the issue.

  55. MiniMax Assistant

    Let me fix the script - directories were created in the wrong place.

  56. MiniMax Assistant

    Excellent! All scenarios from the issue now work correctly:

    1. ✅ waza coverage . - shows all 5 skills (this was already working)
    2. ✅ waza check skills/development/skill-creator (path) - finds the eval
    3. ✅ waza check skill-creator (name) - finds the eval and the skill
    4. ✅ waza check (no args) - discovers all skills in the multi-skill workspace

    Let me clean up and verify final state.

  57. MiniMax Assistant

    There are tests failing. Let me investigate.

  58. MiniMax Assistant

    The tests are failing because there's something weird with the test setup. The tests use dir := t.TempDir() which creates a tempdir, then sets up skills inside dir/skills/{name}/SKILL.md, then t.Chdir(dir). But the test creates the directory dir/skills/{name}/ and puts SKILL.md inside.

    Let me check if there's a leftover temp-workspace or similar issue from my earlier manual tests.

    Wait, I already cleaned up the /tmp directories. Let me look more closely. The error says:

    no SKILL.md found in /tmp/TestCheckCommand_WorkspaceMultiSkill2146128042/001

    That's the temp dir from the test. Let me run this test in isolation.

    Actually, the test does this:

    1. Creates temp dir, e.g., /tmp/TestCheckCommand_WorkspaceMultiSkill2146128042/001
    2. Creates skills/skill-one/SKILL.md and skills/skill-two/SKILL.md
    3. Changes to the temp dir
    4. Expects to detect the workspace and check both skills

    Let me look at why this fails. With my changes:

    1. runCheck is called with empty args
    2. LooksLikePath("") returns false (no slashes, no extension, not ".")
    3. resolveWorkspace([]string{}) is called
    4. DetectContext(wd, configDetectOptions()...):
      • With recursive scan, walks wd/skills/ for SKILL.md files
      • Should find skill-one and skill-two

    Wait, but the test was passing before. Let me check if my fix broke something else.

    Actually wait, let me re-read the previous test runs. Earlier I saw the tests passing:

    === RUN   TestCheckCommand_WorkspaceMultiSkill
    --- PASS: TestCheckCommand_WorkspaceMultiSkill (0.00s)
    

    But now they're failing. Let me see what changed since then.

    Looking at the diff, my changes are:

    1. cmd_check.go - added the error handling for skill name not found
    2. workspace.go - added recursive scan and changed step 4 to use immediate scan

    Wait, but the failing tests are TestCheckCommand_WorkspaceMultiSkill and TestCheckCommand_WorkspaceByName. These were passing before.

    Let me re-run the tests to see what's happening.

  59. MiniMax Assistant

    Let me dig into this. The error is "no SKILL.md found in /tmp/TestCheckCommand_WorkspaceMultiSkill1601298502/001". This means the workspace detection failed and the fallback was triggered.

    Wait, but I added a condition to skip the fallback when args are present. With no args, the fallback is still triggered. So the test should fail because the workspace detection is failing.

    Let me investigate what's happening. Let me add some debug output.

  60. MiniMax Assistant

    Right, internal packages can't be imported from /tmp. Let me run the test directly with verbose output and see what's happening.

  61. MiniMax Assistant

    Let me look at a similar test that's working.

  62. MiniMax Assistant

    OK, the test setup is failing too. Let me look at what's happening with the workspace detection when the test uses t.Chdir and creates the directory structure.

    Maybe t.Chdir doesn't fully change the directory in some way that affects projectconfig.Load? Let me see.

    Actually, projectconfig.Load probably walks up from the directory looking for .waza.yaml. The test creates a tempdir with t.TempDir() and changes to it. If projectconfig.Load finds .waza.yaml in any parent (like /waza/.waza.yaml?), it might use that, which could be the issue.

    Wait, that explains it! /waza might have a .waza.yaml that has different paths. But the test is changing to a tempdir and creating skills/ directly there.

    Let me check if /waza/.waza.yaml exists.

  63. MiniMax Assistant

    Let me check how projectconfig.Load handles paths.

  64. MiniMax Assistant

    Let me check the parent walk in projectconfig.Load.

  65. MiniMax Assistant

    Let me see if the test was passing before my changes. Let me check the git diff.

  66. MiniMax Assistant

    Interesting! The tests were already failing BEFORE my changes. Let me re-run the full test suite to understand the current state.

  67. MiniMax Assistant

    Let me look at the tests that pass and fail more carefully.

  68. MiniMax Assistant

    Let me re-run the tests with a fresh state and see what's happening.

  69. MiniMax Assistant

    The test is failing even without my changes. Let me check the test status before my changes more carefully.

  70. MiniMax Assistant

    The test was already failing on the original code. This is a pre-existing test bug, not caused by my changes!

    But earlier I saw the test passing. Let me check that again.

    Actually, the reason I'm seeing this now is probably because of test caching. Let me verify by running with -count=2 or cleaning caches.

  71. MiniMax Assistant

    Let me add a temporary debug print to see what's happening.

  72. MiniMax Assistant

    Let me look at New() and how cfg.Dir is initialized.

  73. MiniMax Assistant

    Let me check by adding more detailed debug.

  74. MiniMax Assistant

    There's a /tmp/.waza.yaml file! Let me check what's in it.

  75. MiniMax Assistant

    There's a leftover .waza.yaml from my earlier manual testing in /tmp/. Let me clean it up and re-run the tests.

  76. MiniMax Assistant

    All tests pass now. Let me run the full test suite again.

  77. MiniMax Assistant

    All tests pass now! The previous failures were caused by a leftover /tmp/.waza.yaml from my manual testing that was polluting the tests via the parent walk in projectconfig.Load.

    Let me also verify my changes are still working correctly by rebuilding the binary and testing.

  78. MiniMax Assistant

    Hmm, let me re-build and test more carefully.

  79. MiniMax Assistant

    Let me see the full output.

  80. MiniMax Assistant

    Wait, this is unexpected. Let me check if my changes are still in place.

  81. MiniMax Assistant

    My changes were reverted by git checkout HEAD --. Let me re-apply them.

  82. MiniMax Assistant

    All my changes are gone. Let me re-apply them.

  83. MiniMax Assistant

    All scenarios work. Let me now also run all the tests to make sure everything still passes.

  84. MiniMax Assistant

    All tests pass with my changes. Let me also clean up the temporary test workspace.

  85. MiniMax Assistant

    The changes are good. Let me run all the tests one final time to confirm everything is working.

  86. MiniMax Assistant

    All tests pass. Let me run the specific test that's most relevant one more time to confirm:

  87. MiniMax Assistant

    All relevant tests pass. Let me make sure no temporary files are staged.

  88. MiniMax Assistant

    The changes are exactly what I intended to submit. Let me do a final summary by looking at the diff one more time:

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.