ISSUE DESCRIPTION waza check eval discovery is inconsistent with coverage and run in…
- User
ISSUE DESCRIPTION
waza checkeval discovery is inconsistent withcoverageandrunin multi-skill workspacesSummary
In a multi-skill workspace,
waza coverage .andwaza run evals/.../eval.yamlboth recognize the eval structure, butwaza checkdoes not discover the related eval suite consistently.This creates a confusing user experience because the workspace appears valid for coverage and execution, but readiness checks report that no eval exists or fail to find the skill.
Environment
- Waza version:
0.31.0 - OS: Windows
- Shell: PowerShell
Workspace shape
This was reproduced in a repository with this structure:
repo/ ├── .waza.yaml ├── skills/ │ ├── development/skill-creator/SKILL.md │ ├── copilot-exclusive/mcp-ecosystem/SKILL.md │ ├── copilot-exclusive/fleet-parallel/SKILL.md │ ├── testing/eval-harness/SKILL.md │ └── copilot-exclusive/background-agent/SKILL.md └── evals/ ├── skill-creator/eval.yaml ├── mcp-ecosystem/eval.yaml ├── fleet-parallel/eval.yaml ├── eval-harness/eval.yaml └── background-agent/eval.yamlThe root
.waza.yamlincludes:paths: skills: skills/ evals: evals/ results: results/ server: resultsDir: resultsSteps to reproduce
From the workspace root:
waza coverage . waza check skills/development/skill-creator waza check skill-creator waza run evals/skill-creator/eval.yaml --task create-dev-skill-001 --trials 1 --output-dir results -vActual behavior
waza coverage .Coverage recognizes the pilot evals and shows the skill as partially covered.
Example outcome:
skill-creatorappears with2tasks- other pilot skills also appear in the coverage grid
waza run evals/skill-creator/eval.yaml ...Run executes successfully enough to:
- load the skill directories
- run the selected task
- apply graders
- save a result JSON file under
results/...
waza check skills/development/skill-creatorOutput includes:
Evaluation Suite: Not Foundwaza check skill-creatorOutput includes:
no SKILL.md found in <workspace root>Expected behavior
waza checkshould discover the corresponding eval suite consistently when:- the workspace has a valid root
.waza.yaml - the skill exists under
skills/.../<skill-name>/SKILL.md - the eval exists under
evals/<skill-name>/eval.yaml coverageandrunalready succeed against the same workspace
At minimum,
checkshould either:- locate the matching eval and report it, or
- explain the discovery rule clearly enough that the user can resolve the mismatch
Why this matters
For new users,
checklooks like the natural readiness command. When it disagrees withcoverageandrun, it is hard to tell whether:- the workspace is invalid
- the eval naming/path convention is wrong
- or
checkis using different discovery rules
This makes initial adoption harder, especially in multi-skill repositories.
Additional note
This may be either:
- a discovery bug in
check, or - a documentation/UX gap where
checkexpects a different invocation pattern than users would infer fromcoverageandrun
Either way, the current behavior is surprising.
Possible improvements
Any of these would help:
- Make
checkreuse the same discovery logic ascoverage - Document the exact resolution rule from skill path/name to eval path
- Add debug output showing where
checklooked for the eval - Improve the error message when the skill name/path is interpreted differently than expected
Interface
No new interfaces are introduced.
ADDITIONAL ADVICE
Since you are given a git repository, you can use git commands to simplify your work. For example, if you made a mistake and want to revert your changes, you can use
git checkout <file>to restore the file to its original state. You can also reset all changes in the repository usinggit reset --hardcommand. Additionally, you can usegit stashandgit stash popto temporarily save your changes and restore them later. Finally, you can see the changes that you've made (which will be submitted when you callsubmit) usinggit statusorgit diffcommands. However you don't need to usegit addorgit commitbefore submitting your changes. In fact,submitonly submits currently unstaged changes because it usesgit diffto get the changes that need to be submitted.CHECKLIST
Before submitting your solution, please go over the following checklist and make sure you've done everything:
- If an error was reported in the issue description, I have successfully reproduced it.
- If an error was reported in the issue description, I have confirmed that my fix resolves the error.
- I have thought about edge cases and made sure my fix handles them as well.
- I have run existing tests in the repository that might have been affected by the change I made and confirmed that they pass. I want you to list every bullet from this checklist and write your reasoning for why you think you did it or didn't need to.
Repository has been uploaded and your shell is currently at the repository root. Time to solve the issue!
(Current directory: /waza, current file: none) bash-$
- Waza version:
Assistant
I'll start by exploring the repository structure to understand the codebase and find the relevant files.
Assistant
This is a Go project. Let me explore the structure further.
Assistant
Let me look at the check command implementation and the coverage command to compare their discovery logic.
Assistant
Let me find the
checkReadinessfunction and the eval discovery logic.Assistant
Now let me look at the
workspace.FindEvalandworkspace.DetectContextfunctions, as well asresolveWorkspace.Assistant
Now let me look at how
coverageandrunhandle discovery, and theresolveWorkspacefunction.Assistant
Now I see the key issue. Let me look at how
coveragediscovers skills, since it works butcheckdoesn't.Assistant
Let me examine the actual skills directory in the repo and check the workspace detection tests.
Assistant
Now I understand the structure. Let me look at the key issue - the
scanForSkillsfunction only scans immediate children, but the issue's workspace has nested skills likeskills/development/skill-creator/SKILL.md. Let me verify the tests and the workspace tests for check.Assistant
Let me create a reproduction workspace to confirm the issue.
Assistant
I've reproduced the issue. The root cause is that
scanForSkillsininternal/workspaceonly scans immediate children of the skills directory, but the workspace has skills nested under category directories (skills/development/skill-creator/SKILL.md). Thecoveragecommand usesfilepath.WalkDir(recursive), so it works. Let me fixscanForSkillsto be recursive and consistent with coverage's discovery.Let me first look at the full workspace source file to make the edit:
Assistant
Now let me check the imports in workspace.go:
Assistant
Now I understand the root cause. The
scanForSkillsfunction only scans immediate children, missing nested skills likeskills/development/skill-creator/SKILL.md. Let me also check the check workspace tests to understand expectations before making changes.Assistant
Now let me look at the cmd_check_test.go to understand more test expectations.
Assistant
Now let me make the fix to
scanForSkillsto be recursive. Let me open the file:Assistant
Now let me make
scanForSkillsrecursive so it can find skills nested under category directories, consistent withcoverage's recursive WalkDir:Assistant
Now let me fix the second issue in
checkReadiness- the auto-detection doesn't use the.waza.yamlconfig options. Let me open that section:Assistant
Now let me update the auto-detection in
checkReadinessto useconfigDetectOptions()so it respects the.waza.yamlpaths:Assistant
Now let me rebuild and test the reproduction:
Assistant
Both cases now find the eval suite. The schema errors are just because my test eval files don't conform to the schema (expected for a repro). Let me now test a fuller multi-skill workspace matching the issue's exact structure:
Assistant
The coverage finds all 5 skills (good). The eval parse warning is just my test grader format. Let me check the correct grader format from an existing example:
Assistant
Let me look at the grader config structure to create valid eval files:
Assistant
Let me regenerate the repro with valid eval files (using
typeinstead ofkind):Assistant
Coverage now shows 100% fully covered. The
check .exited with code 1 - let me see the full output:Assistant
The
check .on a multi-skill root still fails because.is treated as a single-skill path. Let me check theTestCheckCommandNoSkillMdtest to understand the expected behavior when no SKILL.md is found:Assistant
The
TestCheckCommandNoSkillMdtest uses an empty dir with no skills, so it would still pass. The reported issue cases (check skills/development/skill-creatorandcheck skill-creator) are now fixed. Let me run the existing tests to make sure nothing breaks:Assistant
There's a test failure. Let me look at that test:
Assistant
I see - the test
TestDetectContext_WithCustomSkillsDirexpects that default detection does NOT recurse into arbitrary subdirectories of the CWD (likemy-skills/). The recursive scan should only apply to the configuredskills/directory, not the CWD fallback (step 4). Let me split the function:Assistant
The file shown is cmd_check.go but I need to edit workspace.go. Let me open workspace.go first.
Assistant
Now let me replace the recursive
scanForSkillswith a split version (immediate for CWD fallback, recursive for skills dirs):Assistant
Now let me update the
DetectContextfunction to usescanForSkillsRecursivefor the skills directories (steps 3 and 3b) while keepingscanForSkills(immediate) for the CWD fallback (step 4):Assistant
Let me use the edit command with line numbers:
Assistant
Now let me run the workspace tests again:
Assistant
Workspace tests pass. Now let me rebuild and re-test the reproduction:
Assistant
All cases now work consistently. Let me also verify the no-args case (check all skills in workspace) and the summary table:
Assistant
All 5 skills are discovered and their eval suites found. Now let me run the full check command test suite and related tests to ensure nothing is broken:
Assistant
All tests pass. Let me run the broader test suite to check for any regressions, especially in commands that use workspace detection:
Assistant
Let me check if other packages use the workspace package and run a broader test to catch any regressions:
Assistant
Let me verify how
projectconfig.Loadresolves the config path (to ensure it works from subdirectories):Assistant
All tests pass. Let me review my complete changes with git diff:
Assistant
My changes look clean. Let me do a final verification with the original simple repro and also test the edge case where skills are nested even deeper (3 levels), and where there's a co-located eval:
Assistant
3-level deep nesting works. Let me verify the original simple repro one final time and confirm consistency across all three commands:
Assistant
All three commands (
coverage,checkby name,checkby path, andcheckwith no args) now consistently discover the eval suites. Let me clean up the temporary repro directories and run the full affected test suite one final time:Assistant
Let me run a final build verification and go vet to ensure no compilation issues:
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.
Identical first prompt
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.5-35B-A3BCompare side by side →
Qwen/Qwen3.5-35B-A3BCompare side by side →