ISSUE DESCRIPTION waza check eval discovery is inconsistent with coverage and run in…
- User
ISSUE DESCRIPTION
waza checkeval discovery is inconsistent withcoverageandrunin multi-skill workspacesSummary
In a multi-skill workspace,
waza coverage .andwaza run evals/.../eval.yamlboth recognize the eval structure, butwaza checkdoes not discover the related eval suite consistently.This creates a confusing user experience because the workspace appears valid for coverage and execution, but readiness checks report that no eval exists or fail to find the skill.
Environment
- Waza version:
0.31.0 - OS: Windows
- Shell: PowerShell
Workspace shape
This was reproduced in a repository with this structure:
repo/ ├── .waza.yaml ├── skills/ │ ├── development/skill-creator/SKILL.md │ ├── copilot-exclusive/mcp-ecosystem/SKILL.md │ ├── copilot-exclusive/fleet-parallel/SKILL.md │ ├── testing/eval-harness/SKILL.md │ └── copilot-exclusive/background-agent/SKILL.md └── evals/ ├── skill-creator/eval.yaml ├── mcp-ecosystem/eval.yaml ├── fleet-parallel/eval.yaml ├── eval-harness/eval.yaml └── background-agent/eval.yamlThe root
.waza.yamlincludes:paths: skills: skills/ evals: evals/ results: results/ server: resultsDir: resultsSteps to reproduce
From the workspace root:
waza coverage . waza check skills/development/skill-creator waza check skill-creator waza run evals/skill-creator/eval.yaml --task create-dev-skill-001 --trials 1 --output-dir results -vActual behavior
waza coverage .Coverage recognizes the pilot evals and shows the skill as partially covered.
Example outcome:
skill-creatorappears with2tasks- other pilot skills also appear in the coverage grid
waza run evals/skill-creator/eval.yaml ...Run executes successfully enough to:
- load the skill directories
- run the selected task
- apply graders
- save a result JSON file under
results/...
waza check skills/development/skill-creatorOutput includes:
Evaluation Suite: Not Foundwaza check skill-creatorOutput includes:
no SKILL.md found in <workspace root>Expected behavior
waza checkshould discover the corresponding eval suite consistently when:- the workspace has a valid root
.waza.yaml - the skill exists under
skills/.../<skill-name>/SKILL.md - the eval exists under
evals/<skill-name>/eval.yaml coverageandrunalready succeed against the same workspace
At minimum,
checkshould either:- locate the matching eval and report it, or
- explain the discovery rule clearly enough that the user can resolve the mismatch
Why this matters
For new users,
checklooks like the natural readiness command. When it disagrees withcoverageandrun, it is hard to tell whether:- the workspace is invalid
- the eval naming/path convention is wrong
- or
checkis using different discovery rules
This makes initial adoption harder, especially in multi-skill repositories.
Additional note
This may be either:
- a discovery bug in
check, or - a documentation/UX gap where
checkexpects a different invocation pattern than users would infer fromcoverageandrun
Either way, the current behavior is surprising.
Possible improvements
Any of these would help:
- Make
checkreuse the same discovery logic ascoverage - Document the exact resolution rule from skill path/name to eval path
- Add debug output showing where
checklooked for the eval - Improve the error message when the skill name/path is interpreted differently than expected
Interface
No new interfaces are introduced.
ADDITIONAL ADVICE
Since you are given a git repository, you can use git commands to simplify your work. For example, if you made a mistake and want to revert your changes, you can use
git checkout <file>to restore the file to its original state. You can also reset all changes in the repository usinggit reset --hardcommand. Additionally, you can usegit stashandgit stash popto temporarily save your changes and restore them later. Finally, you can see the changes that you've made (which will be submitted when you callsubmit) usinggit statusorgit diffcommands. However you don't need to usegit addorgit commitbefore submitting your changes. In fact,submitonly submits currently unstaged changes because it usesgit diffto get the changes that need to be submitted.CHECKLIST
Before submitting your solution, please go over the following checklist and make sure you've done everything:
- If an error was reported in the issue description, I have successfully reproduced it.
- If an error was reported in the issue description, I have confirmed that my fix resolves the error.
- I have thought about edge cases and made sure my fix handles them as well.
- I have run existing tests in the repository that might have been affected by the change I made and confirmed that they pass. I want you to list every bullet from this checklist and write your reasoning for why you think you did it or didn't need to.
Repository has been uploaded and your shell is currently at the repository root. Time to solve the issue!
(Current directory: /waza, current file: none) bash-$
- Waza version:
Assistant
Now let's check
skill.IsAgentFilesignature and existing tests forscanForSkillsto understand expected behavior/edge cases before modifying.Assistant
Let's check test file for scanForSkills related tests and existing multi-skill structure expectations (e.g. does it test that nested categorized dirs are NOT included, meaning we must not break that assumption).
Assistant
No test seems to assert that nested category directories are excluded. Let's search for anything mentioning "category" or deeper nesting expectations.
Assistant
No matches (grep exit 1 with no output means nothing found). Now let's implement the fix.
Assistant
Build succeeded. Now let's run the existing workspace tests.
Assistant
Now let's rename this recursive function and restore a shallow one for step 4, plus fix step 3/3b calls and clean up doc comment duplication.
Assistant
All workspace tests pass. Now let's run the full test suite, especially cmd/waza tests related to check/coverage.
Assistant
All tests pass. Now let's reproduce the exact issue scenario with a test workspace to verify the fix works end-to-end.
Assistant
Coverage recognizes both. Now let's test
checkbefore confirming with the fix (this is already the patched binary), by path and by name.Assistant
All tests pass. Let's also test
waza runwith a nested skill to make sure nothing broke there, and check thevet/gofmtstate.Assistant
Clean. Let's view the final diff.
Assistant
This is a clean, targeted fix. Let's clean up the temp repro and test files, then run the full test suite once more for final confirmation.
Assistant
All tests pass. Let's review the checklist:
Assistant
Let me explicitly confirm the bug existed before the fix, to satisfy the reproduction requirement in the checklist.
Assistant
Confirmed: the bug reproduces exactly as described before the fix. Now let's rebuild with the fix and confirm resolution once more.
Assistant
The fix resolves both reported symptoms. Let's do a final full test run and diff review before submitting.
Assistant
All tests pass with no failures. Now let me summarize the checklist verification:
Checklist review:
-
Reproduced the error: Yes — I built a test workspace matching the issue's exact multi-skill nested structure (
skills/development/skill-creator/SKILL.md+evals/skill-creator/eval.yamlwith root.waza.yaml), and confirmed with the pre-fix binary thatwaza check skills/development/skill-creatorreported "Evaluation Suite: Not Found" andwaza check skill-creatorreported "no SKILL.md found in<root>" — exactly matching the issue. -
Confirmed the fix resolves it: Yes — after applying the fix and rebuilding, both invocations now correctly report "Evaluation Suite: Found" and successfully resolve the skill by name, matching
coverage's discovery behavior. -
Thought about edge cases:
- Kept the shallow (non-recursive)
scanForSkillsfor the fallback "flat siblings" case (step 4 inDetectContext, when no dedicatedskills/directory exists), so existing behavior for flat/root-level skill layouts is unchanged and doesn't over-eagerly scan unrelated nested directories. - Added a new
scanForSkillsRecursiveused specifically for the configuredskills/directory and.github/skills/, mirroringwaza coverage'sfilepath.WalkDir-based discovery, including skipping hidden/node_modules/vendordirectories. - Deduplicated by directory path to avoid double-counting a skill directory that has multiple matching files (e.g., stray
.agent.mdalongsideSKILL.md). - Verified the existing test
TestDetectContext_WithCustomSkillsDir(which asserts that skills under a differently-named directory should NOT be picked up by default detection) still passes, confirming the recursive scan is properly scoped only to the configured skills directory and doesn't leak into unrelated fallback scanning.
- Kept the shallow (non-recursive)
-
Ran existing tests: Yes — ran
go build ./...andgo test ./...across the entire repository; all packages pass, includinginternal/workspace(which houses the core discovery logic) andcmd/waza(which housescheck/coverage/runcommands).
I'm satisfied with the fix. Let's submit.
-
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.
Identical first prompt
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.5-35B-A3BCompare side by side →
Qwen/Qwen3.5-35B-A3BCompare side by side →