waza check eval discovery is inconsistent with coverage and run in multi-skill…
- User
waza checkeval discovery is inconsistent withcoverageandrunin multi-skill workspacesSummary
In a multi-skill workspace,
waza coverage .andwaza run evals/.../eval.yamlboth recognize the eval structure, butwaza checkdoes not discover the related eval suite consistently.This creates a confusing user experience because the workspace appears valid for coverage and execution, but readiness checks report that no eval exists or fail to find the skill.
Environment
- Waza version:
0.31.0 - OS: Windows
- Shell: PowerShell
Workspace shape
This was reproduced in a repository with this structure:
repo/ ├── .waza.yaml ├── skills/ │ ├── development/skill-creator/SKILL.md │ ├── copilot-exclusive/mcp-ecosystem/SKILL.md │ ├── copilot-exclusive/fleet-parallel/SKILL.md │ ├── testing/eval-harness/SKILL.md │ └── copilot-exclusive/background-agent/SKILL.md └── evals/ ├── skill-creator/eval.yaml ├── mcp-ecosystem/eval.yaml ├── fleet-parallel/eval.yaml ├── eval-harness/eval.yaml └── background-agent/eval.yamlThe root
.waza.yamlincludes:paths: skills: skills/ evals: evals/ results: results/ server: resultsDir: resultsSteps to reproduce
From the workspace root:
waza coverage . waza check skills/development/skill-creator waza check skill-creator waza run evals/skill-creator/eval.yaml --task create-dev-skill-001 --trials 1 --output-dir results -vActual behavior
waza coverage .Coverage recognizes the pilot evals and shows the skill as partially covered.
Example outcome:
skill-creatorappears with2tasks- other pilot skills also appear in the coverage grid
waza run evals/skill-creator/eval.yaml ...Run executes successfully enough to:
- load the skill directories
- run the selected task
- apply graders
- save a result JSON file under
results/...
waza check skills/development/skill-creatorOutput includes:
Evaluation Suite: Not Foundwaza check skill-creatorOutput includes:
no SKILL.md found in <workspace root>Expected behavior
waza checkshould discover the corresponding eval suite consistently when:- the workspace has a valid root
.waza.yaml - the skill exists under
skills/.../<skill-name>/SKILL.md - the eval exists under
evals/<skill-name>/eval.yaml coverageandrunalready succeed against the same workspace
At minimum,
checkshould either:- locate the matching eval and report it, or
- explain the discovery rule clearly enough that the user can resolve the mismatch
Why this matters
For new users,
checklooks like the natural readiness command. When it disagrees withcoverageandrun, it is hard to tell whether:- the workspace is invalid
- the eval naming/path convention is wrong
- or
checkis using different discovery rules
This makes initial adoption harder, especially in multi-skill repositories.
Additional note
This may be either:
- a discovery bug in
check, or - a documentation/UX gap where
checkexpects a different invocation pattern than users would infer fromcoverageandrun
Either way, the current behavior is surprising.
Possible improvements
Any of these would help:
- Make
checkreuse the same discovery logic ascoverage - Document the exact resolution rule from skill path/name to eval path
- Add debug output showing where
checklooked for the eval - Improve the error message when the skill name/path is interpreted differently than expected
Interface
No new interfaces are introduced.
- Waza version:
- Assistant
Now let me look at the existing tests to understand current expectations.
- Assistant
Let me check where
resolveWorkspaceandDetectContextare used across commands: - Assistant
Now add the recursive scanner and improve the
FindSkillerror message: - Assistant
Now fix
cmd_check.go— pass config options in the fallback detection, and stop silently falling back to cwd when a skill name arg fails to resolve: - Assistant
Also document the eval resolution rule in the
checkhelp text (improvement #2 from the issue): - Assistant
Clean. Now let me reproduce the exact workspace shape from the issue and verify end-to-end:
- Assistant
All fixed. Verify
coveragestill agrees: - Assistant
Consistent. Now add regression tests. First to the workspace package:
- Assistant
Now regression tests for the
checkcommand matching the issue's scenario: - Assistant
Full suite passes. Run vet and check the Makefile's lint targets:
- Assistant
Full suite again to be safe, then re-verify the end-to-end repro:
- Assistant
All scenarios work. Final formatting check and diff review:
- Assistant
The fix is complete. Here's a summary of what was wrong and what changed:
Root cause
waza coveragediscovers skills by recursively walkingskills/(filepath.WalkDir), butwaza checkrelied onworkspace.DetectContext, whosescanForSkillsonly looked at immediate children ofskills/. In a workspace with category folders (skills/development/skill-creator/), the two commands disagreed:waza check skill-creator→ workspace detection found zero skills → silent fallback to cwd →no SKILL.md found in <workspace root>waza check skills/development/skill-creator→ the skill loaded, but the eval-discovery fallback calledDetectContextwithout the.waza.yamlpath options and couldn't see the nested skill, soFindEvalnever resolvedevals/skill-creator/eval.yaml→Evaluation Suite: Not Found
Changes
internal/workspace/workspace.go- Added
scanForSkillsRecursive(mirrors coverage's walk: skips hidden/node_modules/vendordirs, stops descending once a skill dir is found) and use it for the configured skills dir and.github/skills/. Nested skills are now discovered bycheck,run,dev,new-task, etc. — the same discovery rule ascoverage. FindSkillerror now lists available skills:skill "x" not found in workspace (available: a, b, c)(issue improvement #4).
cmd/waza/cmd_check.go- Eval-discovery fallback in
checkReadinessnow passesconfigDetectOptions()so.waza.yamlpaths.skills/paths.evalsare honored. - A skill-name argument that can't be resolved now surfaces the discovery error instead of silently falling back to cwd (no more misleading
no SKILL.md found in <workspace root>). - A directory path that isn't itself a skill falls back to workspace detection rooted at that path, so
waza check .at a multi-skill root behaves likewaza coverage .. - Help text now documents the eval resolution rule (separated → nested → co-located) (issue improvement #2).
cmd/waza/workspace_resolve.go— extractedconfigDetectOptionsFor(dir)so config-derived options can be scoped to an explicit target path rather than always the cwd.Verification
- 8 new regression tests covering the exact issue layout (
skills/<category>/<name>/SKILL.md+evals/<name>/eval.yaml+ root.waza.yaml): check by name, by path, by., unknown-name error, nested detection, fixture/hidden-dir skipping, nestedFindEval. - Full
go test ./...passes,go vetclean,gofmtclean. - End-to-end repro of the issue's workspace confirms
check,coverage, and eval resolution now agree, while an empty directory still errors as before.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.
Identical first prompt
- composer-2.5Compare side by side →
- composer-2.5Compare side by side →
- composer-2.5Compare side by side →
- composer-2.5Compare side by side →
- composer-2.5Compare side by side →
- gpt-5.6-solCompare side by side →
- gpt-5.6-solCompare side by side →
- gpt-5.6-solCompare side by side →
- gpt-5.6-solCompare side by side →
- gpt-5.6-solCompare side by side →
- fable-5Compare side by side →
- fable-5Compare side by side →