Skip to content
TheChatArchiveThe Chat Archive
GLMz-ai/glm-5.2· unknownHugging Face ibragim-bad/swe_rebench_07_2026_trajectories

ISSUE DESCRIPTION waza check eval discovery is inconsistent with coverage and run in…

  1. User

    ISSUE DESCRIPTION

    waza check eval discovery is inconsistent with coverage and run in multi-skill workspaces

    Summary

    In a multi-skill workspace, waza coverage . and waza run evals/.../eval.yaml both recognize the eval structure, but waza check does not discover the related eval suite consistently.

    This creates a confusing user experience because the workspace appears valid for coverage and execution, but readiness checks report that no eval exists or fail to find the skill.

    Environment

    • Waza version: 0.31.0
    • OS: Windows
    • Shell: PowerShell

    Workspace shape

    This was reproduced in a repository with this structure:

    repo/
    ├── .waza.yaml
    ├── skills/
    │   ├── development/skill-creator/SKILL.md
    │   ├── copilot-exclusive/mcp-ecosystem/SKILL.md
    │   ├── copilot-exclusive/fleet-parallel/SKILL.md
    │   ├── testing/eval-harness/SKILL.md
    │   └── copilot-exclusive/background-agent/SKILL.md
    └── evals/
        ├── skill-creator/eval.yaml
        ├── mcp-ecosystem/eval.yaml
        ├── fleet-parallel/eval.yaml
        ├── eval-harness/eval.yaml
        └── background-agent/eval.yaml
    

    The root .waza.yaml includes:

    paths:
      skills: skills/
      evals: evals/
      results: results/
    
    server:
      resultsDir: results
    

    Steps to reproduce

    From the workspace root:

    waza coverage .
    waza check skills/development/skill-creator
    waza check skill-creator
    waza run evals/skill-creator/eval.yaml --task create-dev-skill-001 --trials 1 --output-dir results -v
    

    Actual behavior

    waza coverage .

    Coverage recognizes the pilot evals and shows the skill as partially covered.

    Example outcome:

    • skill-creator appears with 2 tasks
    • other pilot skills also appear in the coverage grid

    waza run evals/skill-creator/eval.yaml ...

    Run executes successfully enough to:

    • load the skill directories
    • run the selected task
    • apply graders
    • save a result JSON file under results/...

    waza check skills/development/skill-creator

    Output includes:

    Evaluation Suite: Not Found
    

    waza check skill-creator

    Output includes:

    no SKILL.md found in <workspace root>
    

    Expected behavior

    waza check should discover the corresponding eval suite consistently when:

    1. the workspace has a valid root .waza.yaml
    2. the skill exists under skills/.../<skill-name>/SKILL.md
    3. the eval exists under evals/<skill-name>/eval.yaml
    4. coverage and run already succeed against the same workspace

    At minimum, check should either:

    • locate the matching eval and report it, or
    • explain the discovery rule clearly enough that the user can resolve the mismatch

    Why this matters

    For new users, check looks like the natural readiness command. When it disagrees with coverage and run, it is hard to tell whether:

    • the workspace is invalid
    • the eval naming/path convention is wrong
    • or check is using different discovery rules

    This makes initial adoption harder, especially in multi-skill repositories.

    Additional note

    This may be either:

    • a discovery bug in check, or
    • a documentation/UX gap where check expects a different invocation pattern than users would infer from coverage and run

    Either way, the current behavior is surprising.

    Possible improvements

    Any of these would help:

    1. Make check reuse the same discovery logic as coverage
    2. Document the exact resolution rule from skill path/name to eval path
    3. Add debug output showing where check looked for the eval
    4. Improve the error message when the skill name/path is interpreted differently than expected

    Interface

    No new interfaces are introduced.

    ADDITIONAL ADVICE

    Since you are given a git repository, you can use git commands to simplify your work. For example, if you made a mistake and want to revert your changes, you can use git checkout <file> to restore the file to its original state. You can also reset all changes in the repository using git reset --hard command. Additionally, you can use git stash and git stash pop to temporarily save your changes and restore them later. Finally, you can see the changes that you've made (which will be submitted when you call submit) using git status or git diff commands. However you don't need to use git add or git commit before submitting your changes. In fact, submit only submits currently unstaged changes because it uses git diff to get the changes that need to be submitted.

    CHECKLIST

    Before submitting your solution, please go over the following checklist and make sure you've done everything:

    • If an error was reported in the issue description, I have successfully reproduced it.
    • If an error was reported in the issue description, I have confirmed that my fix resolves the error.
    • I have thought about edge cases and made sure my fix handles them as well.
    • I have run existing tests in the repository that might have been affected by the change I made and confirmed that they pass. I want you to list every bullet from this checklist and write your reasoning for why you think you did it or didn't need to.

    Repository has been uploaded and your shell is currently at the repository root. Time to solve the issue!

    (Current directory: /waza, current file: none) bash-$

  2. GLM Assistant

    I'll start by exploring the repository structure to understand the codebase and find the relevant files.

  3. GLM Assistant

    This is a Go project. Let me explore the structure further.

  4. GLM Assistant

    Let me look at the check command implementation and the coverage command to compare their discovery logic.

  5. GLM Assistant

    Let me find the checkReadiness function and the eval discovery logic.

  6. GLM Assistant

    Now let me look at the workspace.FindEval and workspace.DetectContext functions, as well as resolveWorkspace.

  7. GLM Assistant

    Now let me look at how coverage and run handle discovery, and the resolveWorkspace function.

  8. GLM Assistant

    Now I see the key issue. Let me look at how coverage discovers skills, since it works but check doesn't.

  9. GLM Assistant

    Let me examine the actual skills directory in the repo and check the workspace detection tests.

  10. GLM Assistant

    Now I understand the structure. Let me look at the key issue - the scanForSkills function only scans immediate children, but the issue's workspace has nested skills like skills/development/skill-creator/SKILL.md. Let me verify the tests and the workspace tests for check.

  11. GLM Assistant

    Let me create a reproduction workspace to confirm the issue.

  12. GLM Assistant

    I've reproduced the issue. The root cause is that scanForSkills in internal/workspace only scans immediate children of the skills directory, but the workspace has skills nested under category directories (skills/development/skill-creator/SKILL.md). The coverage command uses filepath.WalkDir (recursive), so it works. Let me fix scanForSkills to be recursive and consistent with coverage's discovery.

    Let me first look at the full workspace source file to make the edit:

  13. GLM Assistant

    Now let me check the imports in workspace.go:

  14. GLM Assistant

    Now I understand the root cause. The scanForSkills function only scans immediate children, missing nested skills like skills/development/skill-creator/SKILL.md. Let me also check the check workspace tests to understand expectations before making changes.

  15. GLM Assistant

    Now let me look at the cmd_check_test.go to understand more test expectations.

  16. GLM Assistant

    Now let me make the fix to scanForSkills to be recursive. Let me open the file:

  17. GLM Assistant

    Now let me make scanForSkills recursive so it can find skills nested under category directories, consistent with coverage's recursive WalkDir:

  18. GLM Assistant

    Now let me fix the second issue in checkReadiness - the auto-detection doesn't use the .waza.yaml config options. Let me open that section:

  19. GLM Assistant

    Now let me update the auto-detection in checkReadiness to use configDetectOptions() so it respects the .waza.yaml paths:

  20. GLM Assistant

    Now let me rebuild and test the reproduction:

  21. GLM Assistant

    Both cases now find the eval suite. The schema errors are just because my test eval files don't conform to the schema (expected for a repro). Let me now test a fuller multi-skill workspace matching the issue's exact structure:

  22. GLM Assistant

    The coverage finds all 5 skills (good). The eval parse warning is just my test grader format. Let me check the correct grader format from an existing example:

  23. GLM Assistant

    Let me look at the grader config structure to create valid eval files:

  24. GLM Assistant

    Let me regenerate the repro with valid eval files (using type instead of kind):

  25. GLM Assistant

    Coverage now shows 100% fully covered. The check . exited with code 1 - let me see the full output:

  26. GLM Assistant

    The check . on a multi-skill root still fails because . is treated as a single-skill path. Let me check the TestCheckCommandNoSkillMd test to understand the expected behavior when no SKILL.md is found:

  27. GLM Assistant

    The TestCheckCommandNoSkillMd test uses an empty dir with no skills, so it would still pass. The reported issue cases (check skills/development/skill-creator and check skill-creator) are now fixed. Let me run the existing tests to make sure nothing breaks:

  28. GLM Assistant

    There's a test failure. Let me look at that test:

  29. GLM Assistant

    I see - the test TestDetectContext_WithCustomSkillsDir expects that default detection does NOT recurse into arbitrary subdirectories of the CWD (like my-skills/). The recursive scan should only apply to the configured skills/ directory, not the CWD fallback (step 4). Let me split the function:

  30. GLM Assistant

    The file shown is cmd_check.go but I need to edit workspace.go. Let me open workspace.go first.

  31. GLM Assistant

    Now let me replace the recursive scanForSkills with a split version (immediate for CWD fallback, recursive for skills dirs):

  32. GLM Assistant

    Now let me update the DetectContext function to use scanForSkillsRecursive for the skills directories (steps 3 and 3b) while keeping scanForSkills (immediate) for the CWD fallback (step 4):

  33. GLM Assistant

    Let me use the edit command with line numbers:

  34. GLM Assistant

    Now let me run the workspace tests again:

  35. GLM Assistant

    Workspace tests pass. Now let me rebuild and re-test the reproduction:

  36. GLM Assistant

    All cases now work consistently. Let me also verify the no-args case (check all skills in workspace) and the summary table:

  37. GLM Assistant

    All 5 skills are discovered and their eval suites found. Now let me run the full check command test suite and related tests to ensure nothing is broken:

  38. GLM Assistant

    All tests pass. Let me run the broader test suite to check for any regressions, especially in commands that use workspace detection:

  39. GLM Assistant

    Let me check if other packages use the workspace package and run a broader test to catch any regressions:

  40. GLM Assistant

    Let me verify how projectconfig.Load resolves the config path (to ensure it works from subdirectories):

  41. GLM Assistant

    All tests pass. Let me review my complete changes with git diff:

  42. GLM Assistant

    My changes look clean. Let me do a final verification with the original simple repro and also test the edge case where skills are nested even deeper (3 levels), and where there's a co-located eval:

  43. GLM Assistant

    3-level deep nesting works. Let me verify the original simple repro one final time and confirm consistency across all three commands:

  44. GLM Assistant

    All three commands (coverage, check by name, check by path, and check with no args) now consistently discover the eval suites. Let me clean up the temporary repro directories and run the full affected test suite one final time:

  45. GLM Assistant

    Let me run a final build verification and go vet to ensure no compilation issues:

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.