Skip to content
TheChatArchiveThe Chat Archive
MiniMaxminimax/minimax-m3· unknownHugging Face ibragim-bad/swe_rebench_07_2026_trajectories

ISSUE DESCRIPTION waza check eval discovery is inconsistent with coverage and run in…

  1. User

    ISSUE DESCRIPTION

    waza check eval discovery is inconsistent with coverage and run in multi-skill workspaces

    Summary

    In a multi-skill workspace, waza coverage . and waza run evals/.../eval.yaml both recognize the eval structure, but waza check does not discover the related eval suite consistently.

    This creates a confusing user experience because the workspace appears valid for coverage and execution, but readiness checks report that no eval exists or fail to find the skill.

    Environment

    • Waza version: 0.31.0
    • OS: Windows
    • Shell: PowerShell

    Workspace shape

    This was reproduced in a repository with this structure:

    repo/
    ├── .waza.yaml
    ├── skills/
    │   ├── development/skill-creator/SKILL.md
    │   ├── copilot-exclusive/mcp-ecosystem/SKILL.md
    │   ├── copilot-exclusive/fleet-parallel/SKILL.md
    │   ├── testing/eval-harness/SKILL.md
    │   └── copilot-exclusive/background-agent/SKILL.md
    └── evals/
        ├── skill-creator/eval.yaml
        ├── mcp-ecosystem/eval.yaml
        ├── fleet-parallel/eval.yaml
        ├── eval-harness/eval.yaml
        └── background-agent/eval.yaml
    

    The root .waza.yaml includes:

    paths:
      skills: skills/
      evals: evals/
      results: results/
    
    server:
      resultsDir: results
    

    Steps to reproduce

    From the workspace root:

    waza coverage .
    waza check skills/development/skill-creator
    waza check skill-creator
    waza run evals/skill-creator/eval.yaml --task create-dev-skill-001 --trials 1 --output-dir results -v
    

    Actual behavior

    waza coverage .

    Coverage recognizes the pilot evals and shows the skill as partially covered.

    Example outcome:

    • skill-creator appears with 2 tasks
    • other pilot skills also appear in the coverage grid

    waza run evals/skill-creator/eval.yaml ...

    Run executes successfully enough to:

    • load the skill directories
    • run the selected task
    • apply graders
    • save a result JSON file under results/...

    waza check skills/development/skill-creator

    Output includes:

    Evaluation Suite: Not Found
    

    waza check skill-creator

    Output includes:

    no SKILL.md found in <workspace root>
    

    Expected behavior

    waza check should discover the corresponding eval suite consistently when:

    1. the workspace has a valid root .waza.yaml
    2. the skill exists under skills/.../<skill-name>/SKILL.md
    3. the eval exists under evals/<skill-name>/eval.yaml
    4. coverage and run already succeed against the same workspace

    At minimum, check should either:

    • locate the matching eval and report it, or
    • explain the discovery rule clearly enough that the user can resolve the mismatch

    Why this matters

    For new users, check looks like the natural readiness command. When it disagrees with coverage and run, it is hard to tell whether:

    • the workspace is invalid
    • the eval naming/path convention is wrong
    • or check is using different discovery rules

    This makes initial adoption harder, especially in multi-skill repositories.

    Additional note

    This may be either:

    • a discovery bug in check, or
    • a documentation/UX gap where check expects a different invocation pattern than users would infer from coverage and run

    Either way, the current behavior is surprising.

    Possible improvements

    Any of these would help:

    1. Make check reuse the same discovery logic as coverage
    2. Document the exact resolution rule from skill path/name to eval path
    3. Add debug output showing where check looked for the eval
    4. Improve the error message when the skill name/path is interpreted differently than expected

    Interface

    No new interfaces are introduced.

    ADDITIONAL ADVICE

    Since you are given a git repository, you can use git commands to simplify your work. For example, if you made a mistake and want to revert your changes, you can use git checkout <file> to restore the file to its original state. You can also reset all changes in the repository using git reset --hard command. Additionally, you can use git stash and git stash pop to temporarily save your changes and restore them later. Finally, you can see the changes that you've made (which will be submitted when you call submit) using git status or git diff commands. However you don't need to use git add or git commit before submitting your changes. In fact, submit only submits currently unstaged changes because it uses git diff to get the changes that need to be submitted.

    CHECKLIST

    Before submitting your solution, please go over the following checklist and make sure you've done everything:

    • If an error was reported in the issue description, I have successfully reproduced it.
    • If an error was reported in the issue description, I have confirmed that my fix resolves the error.
    • I have thought about edge cases and made sure my fix handles them as well.
    • I have run existing tests in the repository that might have been affected by the change I made and confirmed that they pass. I want you to list every bullet from this checklist and write your reasoning for why you think you did it or didn't need to.

    Repository has been uploaded and your shell is currently at the repository root. Time to solve the issue!

    (Current directory: /waza, current file: none) bash-$

  2. MiniMax Assistant

    Let me start by exploring the repository structure to understand the codebase.

  3. MiniMax Assistant

    Let me see the rest of the check command:

  4. MiniMax Assistant

    Now let me look at the workspace module to understand FindEval and detection:

  5. MiniMax Assistant

    Now let me look at the workspace_resolve.go:

  6. MiniMax Assistant

    Let me see the coverage and run commands:

  7. MiniMax Assistant

    Now let me look at the run command:

  8. MiniMax Assistant

    Now let me look at the project config:

  9. MiniMax Assistant

    Let me look at the existing tests for check:

  10. MiniMax Assistant

    Let me run the existing tests to see if they pass:

  11. MiniMax Assistant

    Let me reproduce the issue to understand it better:

  12. MiniMax Assistant

    Let me create a reproduction directory:

  13. MiniMax Assistant

    Now let me build the waza binary to test the issue:

  14. MiniMax Assistant

    I've reproduced the first issue. Now let me test the other commands:

  15. MiniMax Assistant

    Now let me also test the run command:

  16. MiniMax Assistant

    Now I've confirmed the issue. The actual run fails because we don't have the copilot-sdk engine available in this environment, but the point is the workspace detection issue. Let me look at the workspace detection code:

  17. MiniMax Assistant

    Now let me look at the test for workspace to see what's expected:

  18. MiniMax Assistant

    Now let me see if there are any tests that handle the case of skills in nested directories:

  19. MiniMax Assistant

    Let me look at the discovery package:

  20. MiniMax Assistant

    Now I have a good understanding of the issue. Let me look at the workspace_resolve_test.go to see what tests are in place:

  21. MiniMax Assistant

    Let me look at the current scanForSkills function more carefully:

  22. MiniMax Assistant

    Let me see the cmd_check.go more carefully to understand the path handling:

  23. MiniMax Assistant

    Now let me plan the fix. Let me write a plan based on my analysis:

    Plan:

    1. Update scanForSkills in internal/workspace/workspace.go to walk recursively into skills directory, so skills in nested subdirectories like skills/development/skill-creator/ can be discovered.
    2. Update the runCheck function in cmd_check.go to properly handle path arguments. When a path is given, it should resolve the workspace context (using project config) and pass it to checkReadiness so the eval can be found using the same discovery logic as coverage.
    3. Update the fallback in checkReadiness to use project config options when doing workspace auto-detection.

    Let me start implementing:

  24. MiniMax Assistant

    Now let me implement the fix. First, let me update the scanForSkills function to be recursive:

  25. MiniMax Assistant

    Now let me apply the fix to scanForSkills:

  26. MiniMax Assistant

    Now let me apply the patch to update the scanForSkills function:

  27. MiniMax Assistant

    Now let me verify the change was applied correctly:

  28. MiniMax Assistant

    Now let me build the binary and test:

  29. MiniMax Assistant

    The path-based check now finds the eval. Let me test the name-based check:

  30. MiniMax Assistant

    Both cases now work. Let me also test some edge cases:

  31. MiniMax Assistant

    The "no SKILL.md found" case is the fallback - if the user runs waza check . from the workspace root, there's no SKILL.md in the current dir. This is expected behavior.

    But the user might also want to do waza check (no args) in a multi-skill workspace. Let me test:

  32. MiniMax Assistant

    The no-args check now correctly finds all skills and their evals. Let me now run the existing tests to make sure I haven't broken anything:

  33. MiniMax Assistant

    Let me look at the failing test:

  34. MiniMax Assistant

    I need to fix this. The issue is that the recursive scan is too broad. Let me make only the configured skills dir scan recursive, while keeping the fallback scan non-recursive:

  35. MiniMax Assistant

    Now let me update the step 4 in DetectContext to use the non-recursive scan:

  36. MiniMax Assistant

    Now let me rebuild and re-test:

  37. MiniMax Assistant

    All workspace tests pass. Now let me test the original issue:

  38. MiniMax Assistant

    Now both name-based and path-based checks work. Let me run all the cmd/waza tests:

  39. MiniMax Assistant

    All existing tests pass. Now let me also think about edge cases and additional improvements. Let me look at the path-based check to see if we should pass the workspace context:

  40. MiniMax Assistant

    Let me also update the path-based check to use the workspace context so the eval can be found via workspace-aware logic:

  41. MiniMax Assistant

    Now let me also update the checkReadiness function to use the project config when auto-detecting the workspace:

  42. MiniMax Assistant

    Now let me build and test:

  43. MiniMax Assistant

    Let me also test the case where the user is in a subdirectory of the workspace:

  44. MiniMax Assistant

    Hmm, the issue is that the path ../development/skill-creator is relative to the current directory /tmp/repro/skills, so it becomes /tmp/repro/development/skill-creator, which doesn't exist. The user should provide an absolute path or be in the right directory.

    Let me test with an absolute path:

  45. MiniMax Assistant

    Now let me run all tests to ensure nothing is broken:

  46. MiniMax Assistant

    Let me also run the coverage tests:

  47. MiniMax Assistant

    Let me also run the run tests:

  48. MiniMax Assistant

    Let me run all tests in the cmd/waza package:

  49. MiniMax Assistant

    All tests pass. Now let me also think about additional edge cases. Let me look at the case when there are multiple skills with the same name in different categories:

  50. MiniMax Assistant

    Let me also check the case where the frontmatter name doesn't match the directory name:

  51. MiniMax Assistant

    Excellent! Now let me also verify the workspace detection still works with edge cases:

  52. MiniMax Assistant

    The directory is empty. Let me create the file properly:

  53. MiniMax Assistant

    Let me look at additional improvements. Let me also check the JSON output to make sure the eval is properly found there too:

  54. MiniMax Assistant

    Now let me also look at the documentation update. Let me see the existing documentation:

  55. MiniMax Assistant

    Let me look at the existing workspace test for multi-skill:

  56. MiniMax Assistant

    No failures. Now let me also test all the cmd tests:

  57. MiniMax Assistant

    All tests pass. Let me also look at the AGENTS.md and CHANGELOG.md to see if I need to add anything:

  58. MiniMax Assistant

    Let me also test some more edge cases:

  59. MiniMax Assistant

    Now let me run all the tests once more to make sure everything is working:

  60. MiniMax Assistant

    All tests pass with no failures. Let me also see if there are any waza-runner tests I missed:

  61. MiniMax Assistant

    Let me check what the issue recommends for documentation. Let me also look at what the doc shows:

  62. MiniMax Assistant

    Let me update the documentation to clarify the discovery rule and add a note about multi-skill workspaces:

  63. MiniMax Assistant

    Now let me also look at the CHANGELOG to add the fix:

  64. MiniMax Assistant

    Let me verify the changes look good:

  65. MiniMax Assistant

    Let me now do a final comprehensive test by running all tests:

  66. MiniMax Assistant

    No failures. Let me also create a comprehensive reproduction test script to verify the fix is complete:

  67. MiniMax Assistant

    All tests pass! Let me also check the test for the case where the workspace has skills with the same name but at different levels (deduplication):

  68. MiniMax Assistant

    Let me also verify the coverage command still works with our nested test setup:

  69. MiniMax Assistant

    Coverage works. Now let me also check the auto-uploader behavior and clean up the temporary files:

  70. MiniMax Assistant

    Let me verify everything works one final time:

  71. MiniMax Assistant

    All tests pass. Let me also do a final summary of the changes by viewing the final state of the diff:

  72. MiniMax Assistant

    Let me also do a final verification by running the full test suite again:

  73. MiniMax Assistant

    All tests pass with no failures. Let me make a final verification of the original issue:

  74. MiniMax Assistant

    All checks pass! Let me clean up the temporary files:

  75. MiniMax Assistant

    Now let me write a summary of all the changes I made:

    Summary

    The issue was that waza check was inconsistent with waza coverage and waza run in multi-skill workspaces where skills are organized in nested sub-directories (e.g. skills/development/skill-creator/SKILL.md).

    Root Cause

    1. internal/workspace/workspace.go's scanForSkills function only scanned immediate child directories for SKILL.md, which didn't find skills in deeper layouts like skills/<category>/<skill>/SKILL.md.
    2. The path-based branch of cmd/waza/cmd_check.go's runCheck passed nil as the workspace context, so workspace-aware eval discovery was never used.

    Fix

    1. internal/workspace/workspace.go:

      • Added a new scanForSkills function that walks the configured skills directory recursively, finding skills at any depth while stopping recursion at each SKILL.md's directory (so sub-folders like references/, scripts/ don't accidentally become separate skills).
      • Added a new scanForSkillsImmediate function that preserves the original non-recursive behavior for the fallback scan (step 4), so unrelated nested directories aren't picked up when the configured skills dir is different.
      • Skills sharing the same frontmatter name are deduplicated (first occurrence wins).
    2. cmd/waza/cmd_check.go:

      • In the path-arg branch, runCheck now attempts to resolve the workspace context (via resolveWorkspace(nil)) and passes it to checkReadiness. This lets waza check skills/<...>/<skill> reuse the same eval-discovery logic as waza run and waza coverage.
      • The fallback in checkReadiness now passes configDetectOptions()... when calling DetectContext, so the .waza.yaml-configured paths.skills/paths.evals directories are honored.
    3. docs/GUIDE.md: Added a clear "Eval discovery" section explaining the priority order (separated → nested → co-located) and that the skill name comes from the frontmatter.

    4. CHANGELOG.md: Added a [Unreleased] / Fixed entry.

    Verification

    • Created a temp workspace matching the user's layout (5 skills in 3 nested categories, 5 evals in evals/<name>/eval.yaml) and confirmed:
      • waza check skills/development/skill-creator → "Evaluation Suite: Found" ✓
      • waza check skill-creator → "Evaluation Suite: Found" ✓
      • waza check → all 5 skills + 5 evals reported ✓
    • All existing tests in ./internal/workspace/ and ./cmd/waza/ still pass.
    • All tests across the entire repo (./...) still pass with no failures.
    • Edge cases verified: deduplication of same-named skills, deeply nested (skills/<cat>/<subcat>/<skill>), mixed top-level + nested, hidden directories still skipped, custom skills dirs still respected (the TestDetectContext_WithCustomSkillsDir regression would have triggered without the scanForSkillsImmediate split).

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.