Skip to content
TheChatArchiveThe Chat Archive
fable-5· unknownHugging Face ibragim-bad/swe_rebench_07_2026_trajectories

waza check eval discovery is inconsistent with coverage and run in multi-skill…

  1. User

    waza check eval discovery is inconsistent with coverage and run in multi-skill workspaces

    Summary

    In a multi-skill workspace, waza coverage . and waza run evals/.../eval.yaml both recognize the eval structure, but waza check does not discover the related eval suite consistently.

    This creates a confusing user experience because the workspace appears valid for coverage and execution, but readiness checks report that no eval exists or fail to find the skill.

    Environment

    • Waza version: 0.31.0
    • OS: Windows
    • Shell: PowerShell

    Workspace shape

    This was reproduced in a repository with this structure:

    repo/
    ├── .waza.yaml
    ├── skills/
    │   ├── development/skill-creator/SKILL.md
    │   ├── copilot-exclusive/mcp-ecosystem/SKILL.md
    │   ├── copilot-exclusive/fleet-parallel/SKILL.md
    │   ├── testing/eval-harness/SKILL.md
    │   └── copilot-exclusive/background-agent/SKILL.md
    └── evals/
        ├── skill-creator/eval.yaml
        ├── mcp-ecosystem/eval.yaml
        ├── fleet-parallel/eval.yaml
        ├── eval-harness/eval.yaml
        └── background-agent/eval.yaml
    

    The root .waza.yaml includes:

    paths:
      skills: skills/
      evals: evals/
      results: results/
    
    server:
      resultsDir: results
    

    Steps to reproduce

    From the workspace root:

    waza coverage .
    waza check skills/development/skill-creator
    waza check skill-creator
    waza run evals/skill-creator/eval.yaml --task create-dev-skill-001 --trials 1 --output-dir results -v
    

    Actual behavior

    waza coverage .

    Coverage recognizes the pilot evals and shows the skill as partially covered.

    Example outcome:

    • skill-creator appears with 2 tasks
    • other pilot skills also appear in the coverage grid

    waza run evals/skill-creator/eval.yaml ...

    Run executes successfully enough to:

    • load the skill directories
    • run the selected task
    • apply graders
    • save a result JSON file under results/...

    waza check skills/development/skill-creator

    Output includes:

    Evaluation Suite: Not Found
    

    waza check skill-creator

    Output includes:

    no SKILL.md found in <workspace root>
    

    Expected behavior

    waza check should discover the corresponding eval suite consistently when:

    1. the workspace has a valid root .waza.yaml
    2. the skill exists under skills/.../<skill-name>/SKILL.md
    3. the eval exists under evals/<skill-name>/eval.yaml
    4. coverage and run already succeed against the same workspace

    At minimum, check should either:

    • locate the matching eval and report it, or
    • explain the discovery rule clearly enough that the user can resolve the mismatch

    Why this matters

    For new users, check looks like the natural readiness command. When it disagrees with coverage and run, it is hard to tell whether:

    • the workspace is invalid
    • the eval naming/path convention is wrong
    • or check is using different discovery rules

    This makes initial adoption harder, especially in multi-skill repositories.

    Additional note

    This may be either:

    • a discovery bug in check, or
    • a documentation/UX gap where check expects a different invocation pattern than users would infer from coverage and run

    Either way, the current behavior is surprising.

    Possible improvements

    Any of these would help:

    1. Make check reuse the same discovery logic as coverage
    2. Document the exact resolution rule from skill path/name to eval path
    3. Add debug output showing where check looked for the eval
    4. Improve the error message when the skill name/path is interpreted differently than expected

    Interface

    No new interfaces are introduced.

  2. Assistant

    Now let me look at the existing tests to understand current expectations.

  3. Assistant

    Let me check where resolveWorkspace and DetectContext are used across commands:

  4. Assistant

    Now add the recursive scanner and improve the FindSkill error message:

  5. Assistant

    Now fix cmd_check.go — pass config options in the fallback detection, and stop silently falling back to cwd when a skill name arg fails to resolve:

  6. Assistant

    Also document the eval resolution rule in the check help text (improvement #2 from the issue):

  7. Assistant

    Clean. Now let me reproduce the exact workspace shape from the issue and verify end-to-end:

  8. Assistant

    All fixed. Verify coverage still agrees:

  9. Assistant

    Consistent. Now add regression tests. First to the workspace package:

  10. Assistant

    Now regression tests for the check command matching the issue's scenario:

  11. Assistant

    Full suite passes. Run vet and check the Makefile's lint targets:

  12. Assistant

    Full suite again to be safe, then re-verify the end-to-end repro:

  13. Assistant

    All scenarios work. Final formatting check and diff review:

  14. Assistant

    The fix is complete. Here's a summary of what was wrong and what changed:

    Root cause

    waza coverage discovers skills by recursively walking skills/ (filepath.WalkDir), but waza check relied on workspace.DetectContext, whose scanForSkills only looked at immediate children of skills/. In a workspace with category folders (skills/development/skill-creator/), the two commands disagreed:

    • waza check skill-creator → workspace detection found zero skills → silent fallback to cwd → no SKILL.md found in <workspace root>
    • waza check skills/development/skill-creator → the skill loaded, but the eval-discovery fallback called DetectContext without the .waza.yaml path options and couldn't see the nested skill, so FindEval never resolved evals/skill-creator/eval.yaml → Evaluation Suite: Not Found

    Changes

    internal/workspace/workspace.go

    • Added scanForSkillsRecursive (mirrors coverage's walk: skips hidden/node_modules/vendor dirs, stops descending once a skill dir is found) and use it for the configured skills dir and .github/skills/. Nested skills are now discovered by check, run, dev, new-task, etc. — the same discovery rule as coverage.
    • FindSkill error now lists available skills: skill "x" not found in workspace (available: a, b, c) (issue improvement #4).

    cmd/waza/cmd_check.go

    • Eval-discovery fallback in checkReadiness now passes configDetectOptions() so .waza.yaml paths.skills/paths.evals are honored.
    • A skill-name argument that can't be resolved now surfaces the discovery error instead of silently falling back to cwd (no more misleading no SKILL.md found in <workspace root>).
    • A directory path that isn't itself a skill falls back to workspace detection rooted at that path, so waza check . at a multi-skill root behaves like waza coverage ..
    • Help text now documents the eval resolution rule (separated → nested → co-located) (issue improvement #2).

    cmd/waza/workspace_resolve.go — extracted configDetectOptionsFor(dir) so config-derived options can be scoped to an explicit target path rather than always the cwd.

    Verification

    • 8 new regression tests covering the exact issue layout (skills/<category>/<name>/SKILL.md + evals/<name>/eval.yaml + root .waza.yaml): check by name, by path, by ., unknown-name error, nested detection, fixture/hidden-dir skipping, nested FindEval.
    • Full go test ./... passes, go vet clean, gofmt clean.
    • End-to-end repro of the issue's workspace confirms check, coverage, and eval resolution now agree, while an empty directory still errors as before.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.