ISSUE DESCRIPTION waza check eval discovery is inconsistent with coverage and run in…
- User
ISSUE DESCRIPTION
waza checkeval discovery is inconsistent withcoverageandrunin multi-skill workspacesSummary
In a multi-skill workspace,
waza coverage .andwaza run evals/.../eval.yamlboth recognize the eval structure, butwaza checkdoes not discover the related eval suite consistently.This creates a confusing user experience because the workspace appears valid for coverage and execution, but readiness checks report that no eval exists or fail to find the skill.
Environment
- Waza version:
0.31.0 - OS: Windows
- Shell: PowerShell
Workspace shape
This was reproduced in a repository with this structure:
repo/ ├── .waza.yaml ├── skills/ │ ├── development/skill-creator/SKILL.md │ ├── copilot-exclusive/mcp-ecosystem/SKILL.md │ ├── copilot-exclusive/fleet-parallel/SKILL.md │ ├── testing/eval-harness/SKILL.md │ └── copilot-exclusive/background-agent/SKILL.md └── evals/ ├── skill-creator/eval.yaml ├── mcp-ecosystem/eval.yaml ├── fleet-parallel/eval.yaml ├── eval-harness/eval.yaml └── background-agent/eval.yamlThe root
.waza.yamlincludes:paths: skills: skills/ evals: evals/ results: results/ server: resultsDir: resultsSteps to reproduce
From the workspace root:
waza coverage . waza check skills/development/skill-creator waza check skill-creator waza run evals/skill-creator/eval.yaml --task create-dev-skill-001 --trials 1 --output-dir results -vActual behavior
waza coverage .Coverage recognizes the pilot evals and shows the skill as partially covered.
Example outcome:
skill-creatorappears with2tasks- other pilot skills also appear in the coverage grid
waza run evals/skill-creator/eval.yaml ...Run executes successfully enough to:
- load the skill directories
- run the selected task
- apply graders
- save a result JSON file under
results/...
waza check skills/development/skill-creatorOutput includes:
Evaluation Suite: Not Foundwaza check skill-creatorOutput includes:
no SKILL.md found in <workspace root>Expected behavior
waza checkshould discover the corresponding eval suite consistently when:- the workspace has a valid root
.waza.yaml - the skill exists under
skills/.../<skill-name>/SKILL.md - the eval exists under
evals/<skill-name>/eval.yaml coverageandrunalready succeed against the same workspace
At minimum,
checkshould either:- locate the matching eval and report it, or
- explain the discovery rule clearly enough that the user can resolve the mismatch
Why this matters
For new users,
checklooks like the natural readiness command. When it disagrees withcoverageandrun, it is hard to tell whether:- the workspace is invalid
- the eval naming/path convention is wrong
- or
checkis using different discovery rules
This makes initial adoption harder, especially in multi-skill repositories.
Additional note
This may be either:
- a discovery bug in
check, or - a documentation/UX gap where
checkexpects a different invocation pattern than users would infer fromcoverageandrun
Either way, the current behavior is surprising.
Possible improvements
Any of these would help:
- Make
checkreuse the same discovery logic ascoverage - Document the exact resolution rule from skill path/name to eval path
- Add debug output showing where
checklooked for the eval - Improve the error message when the skill name/path is interpreted differently than expected
Interface
No new interfaces are introduced.
ADDITIONAL ADVICE
Since you are given a git repository, you can use git commands to simplify your work. For example, if you made a mistake and want to revert your changes, you can use
git checkout <file>to restore the file to its original state. You can also reset all changes in the repository usinggit reset --hardcommand. Additionally, you can usegit stashandgit stash popto temporarily save your changes and restore them later. Finally, you can see the changes that you've made (which will be submitted when you callsubmit) usinggit statusorgit diffcommands. However you don't need to usegit addorgit commitbefore submitting your changes. In fact,submitonly submits currently unstaged changes because it usesgit diffto get the changes that need to be submitted.CHECKLIST
Before submitting your solution, please go over the following checklist and make sure you've done everything:
- If an error was reported in the issue description, I have successfully reproduced it.
- If an error was reported in the issue description, I have confirmed that my fix resolves the error.
- I have thought about edge cases and made sure my fix handles them as well.
- I have run existing tests in the repository that might have been affected by the change I made and confirmed that they pass. I want you to list every bullet from this checklist and write your reasoning for why you think you did it or didn't need to.
Repository has been uploaded and your shell is currently at the repository root. Time to solve the issue!
(Current directory: /waza, current file: none) bash-$
- Waza version:
Assistant
Build works now. Let's set up the reproduction workspace.
Assistant
Now let's build and test.
Assistant
Both
checkinvocations now correctly find the eval. Schema errors are just because my quick test eval.yaml doesn't match the real schema — that's expected and unrelated to this issue. Now let's run the existing test suite for the workspace package and cmd_check tests.Assistant
Let's rename and add a shallow variant for step 4, keeping recursive scan only for the explicit skills directories.
Assistant
All tests pass now. Let's rebuild the binary and re-verify the reproduction scenario still works.
Assistant
All tests pass across the entire repository. Let's view the final diff.
Assistant
Need to fix the comment placement — the recursive
scanForSkillsdoc comment is misplaced abovescanForSkillsShallow. Let me clean this up properly.Assistant
Now clean and correctly ordered. Let's rebuild, retest, and re-verify the reproduction.
Assistant
All tests pass, gofmt is clean. Now let's re-run the full reproduction one more time to confirm consistent behavior end-to-end, including
waza run.Assistant
Both
checkinvocations now consistently discover the eval, matchingcoverage's discovery of the same nested skill structure. Let's clean up the temp reproduction directory and view the final diff before submitting.Assistant
This is a clean, focused fix. Let's do a final full test run to be safe.
Assistant
All tests pass (grep exit 1 just means no failing lines matched, i.e., no output besides "ok"/"no test files"). Now let me review the checklist.
Assistant
Checklist review:
-
Reproduced the reported error: Yes. I built a temp workspace matching the issue's exact multi-skill structure (
skills/development/skill-creator,skills/copilot-exclusive/mcp-ecosystem,evals/skill-creator, etc., with root.waza.yaml). I confirmedwaza check skills/development/skill-creatorreported "Evaluation Suite: Not Found" andwaza check skill-creatorreported "no SKILL.md found in ", exactly as described in the issue. -
Confirmed the fix resolves the error: Yes. After the fix, both
waza check skills/development/skill-creatorandwaza check skill-creatorreport "Evaluation Suite: Found" with the correct eval path, matchingwaza coverage .'s discovery of the same skills. The root cause was thatworkspace.DetectContext's skill-scanning (scanForSkills) only looked one directory level deep under the configuredskills/directory, whilewaza coverageused a full recursivefilepath.WalkDir. This meant multi-skill workspaces with an extra category-level nesting (e.g.skills/<category>/<skill-name>/SKILL.md) were invisible tocheck's workspace detection (used by both the name-based and path-based flows), even thoughcoveragefound them fine. I made the skills-directory scan (scanForSkills) recursive, mirroring coverage's discovery logic (skip hidden/node_modules/vendordirs, stop descending once a skill directory is found so nestedreferences//evals/dirs aren't misidentified as skills). -
Edge cases considered:
- Kept a separate shallow-only
scanForSkillsShallowfor step 4 ofDetectContext(the fallback that scans the given directory itself when no configuredskills/subdirectory exists), so we don't start recursively treating arbitrary deeply-nested directories in an unrelated root as "skills" — this preserves the existing, more conservative fallback behavior and is whyTestDetectContext_WithCustomSkillsDirstill passes (it explicitly expectsContextNonewhen scanning the defaultskills/dir that doesn't exist, without falling back to deep recursion). - Preserved exclusion of hidden directories,
node_modules, andvendorduring recursive scanning, consistent withcoverage's discovery. - Verified
.github/skills/GitHub Copilot convention path also benefits from the same recursive scan since it reusesscanForSkills. - Verified single-skill workspace detection, sibling-skill-dirs workspace, and custom skills/evals dir configuration all still work via existing tests.
- Kept a separate shallow-only
-
Ran existing tests: Yes.
go build ./...andgo test ./...for the entire repository pass, includinginternal/workspaceandcmd/wazapackages that exercise this logic directly.
I'm confident this resolves the issue consistently across
check,coverage, andrun. Submitting now.-
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.
Identical first prompt
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.5-35B-A3BCompare side by side →
Qwen/Qwen3.5-35B-A3BCompare side by side →