Skip to content
TheChatArchiveThe Chat Archive
Claudeclaude-sonnet-5· anthropicHugging Face ibragim-bad/swe_rebench_07_2026_trajectories

ISSUE DESCRIPTION waza check eval discovery is inconsistent with coverage and run in…

  1. User

    ISSUE DESCRIPTION

    waza check eval discovery is inconsistent with coverage and run in multi-skill workspaces

    Summary

    In a multi-skill workspace, waza coverage . and waza run evals/.../eval.yaml both recognize the eval structure, but waza check does not discover the related eval suite consistently.

    This creates a confusing user experience because the workspace appears valid for coverage and execution, but readiness checks report that no eval exists or fail to find the skill.

    Environment

    • Waza version: 0.31.0
    • OS: Windows
    • Shell: PowerShell

    Workspace shape

    This was reproduced in a repository with this structure:

    repo/
    ├── .waza.yaml
    ├── skills/
    │   ├── development/skill-creator/SKILL.md
    │   ├── copilot-exclusive/mcp-ecosystem/SKILL.md
    │   ├── copilot-exclusive/fleet-parallel/SKILL.md
    │   ├── testing/eval-harness/SKILL.md
    │   └── copilot-exclusive/background-agent/SKILL.md
    └── evals/
        ├── skill-creator/eval.yaml
        ├── mcp-ecosystem/eval.yaml
        ├── fleet-parallel/eval.yaml
        ├── eval-harness/eval.yaml
        └── background-agent/eval.yaml
    

    The root .waza.yaml includes:

    paths:
      skills: skills/
      evals: evals/
      results: results/
    
    server:
      resultsDir: results
    

    Steps to reproduce

    From the workspace root:

    waza coverage .
    waza check skills/development/skill-creator
    waza check skill-creator
    waza run evals/skill-creator/eval.yaml --task create-dev-skill-001 --trials 1 --output-dir results -v
    

    Actual behavior

    waza coverage .

    Coverage recognizes the pilot evals and shows the skill as partially covered.

    Example outcome:

    • skill-creator appears with 2 tasks
    • other pilot skills also appear in the coverage grid

    waza run evals/skill-creator/eval.yaml ...

    Run executes successfully enough to:

    • load the skill directories
    • run the selected task
    • apply graders
    • save a result JSON file under results/...

    waza check skills/development/skill-creator

    Output includes:

    Evaluation Suite: Not Found
    

    waza check skill-creator

    Output includes:

    no SKILL.md found in <workspace root>
    

    Expected behavior

    waza check should discover the corresponding eval suite consistently when:

    1. the workspace has a valid root .waza.yaml
    2. the skill exists under skills/.../<skill-name>/SKILL.md
    3. the eval exists under evals/<skill-name>/eval.yaml
    4. coverage and run already succeed against the same workspace

    At minimum, check should either:

    • locate the matching eval and report it, or
    • explain the discovery rule clearly enough that the user can resolve the mismatch

    Why this matters

    For new users, check looks like the natural readiness command. When it disagrees with coverage and run, it is hard to tell whether:

    • the workspace is invalid
    • the eval naming/path convention is wrong
    • or check is using different discovery rules

    This makes initial adoption harder, especially in multi-skill repositories.

    Additional note

    This may be either:

    • a discovery bug in check, or
    • a documentation/UX gap where check expects a different invocation pattern than users would infer from coverage and run

    Either way, the current behavior is surprising.

    Possible improvements

    Any of these would help:

    1. Make check reuse the same discovery logic as coverage
    2. Document the exact resolution rule from skill path/name to eval path
    3. Add debug output showing where check looked for the eval
    4. Improve the error message when the skill name/path is interpreted differently than expected

    Interface

    No new interfaces are introduced.

    ADDITIONAL ADVICE

    Since you are given a git repository, you can use git commands to simplify your work. For example, if you made a mistake and want to revert your changes, you can use git checkout <file> to restore the file to its original state. You can also reset all changes in the repository using git reset --hard command. Additionally, you can use git stash and git stash pop to temporarily save your changes and restore them later. Finally, you can see the changes that you've made (which will be submitted when you call submit) using git status or git diff commands. However you don't need to use git add or git commit before submitting your changes. In fact, submit only submits currently unstaged changes because it uses git diff to get the changes that need to be submitted.

    CHECKLIST

    Before submitting your solution, please go over the following checklist and make sure you've done everything:

    • If an error was reported in the issue description, I have successfully reproduced it.
    • If an error was reported in the issue description, I have confirmed that my fix resolves the error.
    • I have thought about edge cases and made sure my fix handles them as well.
    • I have run existing tests in the repository that might have been affected by the change I made and confirmed that they pass. I want you to list every bullet from this checklist and write your reasoning for why you think you did it or didn't need to.

    Repository has been uploaded and your shell is currently at the repository root. Time to solve the issue!

    (Current directory: /waza, current file: none) bash-$

  2. Claude Assistant

    Build works now. Let's set up the reproduction workspace.

  3. Claude Assistant

    Now let's build and test.

  4. Claude Assistant

    Both check invocations now correctly find the eval. Schema errors are just because my quick test eval.yaml doesn't match the real schema — that's expected and unrelated to this issue. Now let's run the existing test suite for the workspace package and cmd_check tests.

  5. Claude Assistant

    Let's rename and add a shallow variant for step 4, keeping recursive scan only for the explicit skills directories.

  6. Claude Assistant

    All tests pass now. Let's rebuild the binary and re-verify the reproduction scenario still works.

  7. Claude Assistant

    All tests pass across the entire repository. Let's view the final diff.

  8. Claude Assistant

    Need to fix the comment placement — the recursive scanForSkills doc comment is misplaced above scanForSkillsShallow. Let me clean this up properly.

  9. Claude Assistant

    Now clean and correctly ordered. Let's rebuild, retest, and re-verify the reproduction.

  10. Claude Assistant

    All tests pass, gofmt is clean. Now let's re-run the full reproduction one more time to confirm consistent behavior end-to-end, including waza run.

  11. Claude Assistant

    Both check invocations now consistently discover the eval, matching coverage's discovery of the same nested skill structure. Let's clean up the temp reproduction directory and view the final diff before submitting.

  12. Claude Assistant

    This is a clean, focused fix. Let's do a final full test run to be safe.

  13. Claude Assistant

    All tests pass (grep exit 1 just means no failing lines matched, i.e., no output besides "ok"/"no test files"). Now let me review the checklist.

  14. Claude Assistant

    Checklist review:

    • Reproduced the reported error: Yes. I built a temp workspace matching the issue's exact multi-skill structure (skills/development/skill-creator, skills/copilot-exclusive/mcp-ecosystem, evals/skill-creator, etc., with root .waza.yaml). I confirmed waza check skills/development/skill-creator reported "Evaluation Suite: Not Found" and waza check skill-creator reported "no SKILL.md found in ", exactly as described in the issue.

    • Confirmed the fix resolves the error: Yes. After the fix, both waza check skills/development/skill-creator and waza check skill-creator report "Evaluation Suite: Found" with the correct eval path, matching waza coverage .'s discovery of the same skills. The root cause was that workspace.DetectContext's skill-scanning (scanForSkills) only looked one directory level deep under the configured skills/ directory, while waza coverage used a full recursive filepath.WalkDir. This meant multi-skill workspaces with an extra category-level nesting (e.g. skills/<category>/<skill-name>/SKILL.md) were invisible to check's workspace detection (used by both the name-based and path-based flows), even though coverage found them fine. I made the skills-directory scan (scanForSkills) recursive, mirroring coverage's discovery logic (skip hidden/node_modules/vendor dirs, stop descending once a skill directory is found so nested references//evals/ dirs aren't misidentified as skills).

    • Edge cases considered:

      • Kept a separate shallow-only scanForSkillsShallow for step 4 of DetectContext (the fallback that scans the given directory itself when no configured skills/ subdirectory exists), so we don't start recursively treating arbitrary deeply-nested directories in an unrelated root as "skills" — this preserves the existing, more conservative fallback behavior and is why TestDetectContext_WithCustomSkillsDir still passes (it explicitly expects ContextNone when scanning the default skills/ dir that doesn't exist, without falling back to deep recursion).
      • Preserved exclusion of hidden directories, node_modules, and vendor during recursive scanning, consistent with coverage's discovery.
      • Verified .github/skills/ GitHub Copilot convention path also benefits from the same recursive scan since it reuses scanForSkills.
      • Verified single-skill workspace detection, sibling-skill-dirs workspace, and custom skills/evals dir configuration all still work via existing tests.
    • Ran existing tests: Yes. go build ./... and go test ./... for the entire repository pass, including internal/workspace and cmd/waza packages that exercise this logic directly.

    I'm confident this resolves the issue consistently across check, coverage, and run. Submitting now.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.