ISSUE DESCRIPTION waza check eval discovery is inconsistent with coverage and run in…
- User
ISSUE DESCRIPTION
waza checkeval discovery is inconsistent withcoverageandrunin multi-skill workspacesSummary
In a multi-skill workspace,
waza coverage .andwaza run evals/.../eval.yamlboth recognize the eval structure, butwaza checkdoes not discover the related eval suite consistently.This creates a confusing user experience because the workspace appears valid for coverage and execution, but readiness checks report that no eval exists or fail to find the skill.
Environment
- Waza version:
0.31.0 - OS: Windows
- Shell: PowerShell
Workspace shape
This was reproduced in a repository with this structure:
repo/ ├── .waza.yaml ├── skills/ │ ├── development/skill-creator/SKILL.md │ ├── copilot-exclusive/mcp-ecosystem/SKILL.md │ ├── copilot-exclusive/fleet-parallel/SKILL.md │ ├── testing/eval-harness/SKILL.md │ └── copilot-exclusive/background-agent/SKILL.md └── evals/ ├── skill-creator/eval.yaml ├── mcp-ecosystem/eval.yaml ├── fleet-parallel/eval.yaml ├── eval-harness/eval.yaml └── background-agent/eval.yamlThe root
.waza.yamlincludes:paths: skills: skills/ evals: evals/ results: results/ server: resultsDir: resultsSteps to reproduce
From the workspace root:
waza coverage . waza check skills/development/skill-creator waza check skill-creator waza run evals/skill-creator/eval.yaml --task create-dev-skill-001 --trials 1 --output-dir results -vActual behavior
waza coverage .Coverage recognizes the pilot evals and shows the skill as partially covered.
Example outcome:
skill-creatorappears with2tasks- other pilot skills also appear in the coverage grid
waza run evals/skill-creator/eval.yaml ...Run executes successfully enough to:
- load the skill directories
- run the selected task
- apply graders
- save a result JSON file under
results/...
waza check skills/development/skill-creatorOutput includes:
Evaluation Suite: Not Foundwaza check skill-creatorOutput includes:
no SKILL.md found in <workspace root>Expected behavior
waza checkshould discover the corresponding eval suite consistently when:- the workspace has a valid root
.waza.yaml - the skill exists under
skills/.../<skill-name>/SKILL.md - the eval exists under
evals/<skill-name>/eval.yaml coverageandrunalready succeed against the same workspace
At minimum,
checkshould either:- locate the matching eval and report it, or
- explain the discovery rule clearly enough that the user can resolve the mismatch
Why this matters
For new users,
checklooks like the natural readiness command. When it disagrees withcoverageandrun, it is hard to tell whether:- the workspace is invalid
- the eval naming/path convention is wrong
- or
checkis using different discovery rules
This makes initial adoption harder, especially in multi-skill repositories.
Additional note
This may be either:
- a discovery bug in
check, or - a documentation/UX gap where
checkexpects a different invocation pattern than users would infer fromcoverageandrun
Either way, the current behavior is surprising.
Possible improvements
Any of these would help:
- Make
checkreuse the same discovery logic ascoverage - Document the exact resolution rule from skill path/name to eval path
- Add debug output showing where
checklooked for the eval - Improve the error message when the skill name/path is interpreted differently than expected
Interface
No new interfaces are introduced.
ADDITIONAL ADVICE
Since you are given a git repository, you can use git commands to simplify your work. For example, if you made a mistake and want to revert your changes, you can use
git checkout <file>to restore the file to its original state. You can also reset all changes in the repository usinggit reset --hardcommand. Additionally, you can usegit stashandgit stash popto temporarily save your changes and restore them later. Finally, you can see the changes that you've made (which will be submitted when you callsubmit) usinggit statusorgit diffcommands. However you don't need to usegit addorgit commitbefore submitting your changes. In fact,submitonly submits currently unstaged changes because it usesgit diffto get the changes that need to be submitted.CHECKLIST
Before submitting your solution, please go over the following checklist and make sure you've done everything:
- If an error was reported in the issue description, I have successfully reproduced it.
- If an error was reported in the issue description, I have confirmed that my fix resolves the error.
- I have thought about edge cases and made sure my fix handles them as well.
- I have run existing tests in the repository that might have been affected by the change I made and confirmed that they pass. I want you to list every bullet from this checklist and write your reasoning for why you think you did it or didn't need to.
Repository has been uploaded and your shell is currently at the repository root. Time to solve the issue!
(Current directory: /waza, current file: none) bash-$
- Waza version:
Assistant
Let me start by exploring the repository structure to understand the codebase.
Assistant
Let me explore the main source files and understand the structure better.
Assistant
Let me look at the key files: cmd_check.go, cmd_coverage.go, cmd_run.go, and workspace_resolve.go to understand how they discover evals.
Assistant
Let me see the workspace_resolve.go file and find the checkReadiness function.
Assistant
Now let me look at the workspace package to understand the
FindEvalfunction.Assistant
Now let me look at how coverage and run discover evals to understand the discrepancy.
Assistant
Let me see the run command to understand how it discovers evals.
Assistant
Let me look at how coverage and run find the skills to understand why they work.
Assistant
Let me look at the existing tests to understand the expected behavior.
Assistant
Let me first reproduce the issue with a test workspace before making any changes.
Assistant
Coverage finds all 5 skills. Now let me test
waza check.Assistant
Confirmed issue 1. Now let me test the other case.
Assistant
Both issues are confirmed. Let me also examine the run command flow more carefully.
Assistant
Let me look at more of
checkReadinessand the tests for it.Assistant
Let me look at more of the
scanForSkillsand how it's called.Assistant
Let me look at how the run command deals with the same issue.
Assistant
Let me look at the projectconfig to understand the structure.
Assistant
Let me check the existing cmd_run_workspace_test.go to ensure I don't break it.
Assistant
Now I have a good understanding. Let me design the fix.
Plan
-
Make
scanForSkillsin/waza/internal/workspace/workspace.gorecursive. This fixes the discovery of nested skills (e.g.,skills/development/skill-creator/SKILL.md), makingwaza checkconsistent withwaza coveragewhich usesfilepath.WalkDir. -
Pass
configDetectOptions()to the fallbackDetectContextcall incheckReadiness. This ensures the user's project config (.waza.yaml) is respected when looking for the eval from a path-based invocation. -
Improve error messages when a skill is not found or eval is not found. Provide actionable information so users can diagnose the issue.
Let me check the current state of the test for nested skills to make sure the test would still pass.
-
Assistant
Now let me look at the current
scanForSkillsto understand its structure better.Assistant
Now let me make the changes. First, I'll modify
scanForSkillsto be recursive.Assistant
Let me run the existing tests to make sure I didn't break anything.
Assistant
Let me look at the failing test.
Assistant
I need to be more careful. The recursive scan is needed for configured skills dirs (step 3), but step 4 should still scan only immediate children (fallback to root). Let me refactor properly.
Assistant
All existing tests pass. Now let me verify the changes work for the user's case.
Assistant
waza check skill-creatornow finds the eval. Let me also test the path-based invocation.Assistant
Both cases work. Now let me test the case with custom paths to verify the second issue.
Assistant
The test used the wrong path. Let me use the correct path.
Assistant
The path-based invocation still fails for custom paths. Let me fix the fallback in
checkReadinessto pass config options.Assistant
All three test cases work. Let me run all the existing tests to make sure I didn't break anything.
Assistant
All tests pass. Now let me also look at the related test cases in the broader test suite.
Assistant
All tests pass. Now let me also consider improving the error messages when the skill is not found.
Assistant
Let me also improve the error handling when a skill name is provided but not found.
Assistant
Now let me also test the case where a non-existent skill is requested:
Assistant
A better error message. Now let me run all the tests.
Assistant
All tests pass. Now let me also verify the original test cases in the issue work as expected.
Assistant
Let me also test
waza checkfor all 5 skills in the workspace.Assistant
All 5 skills are found and their evals are detected. Now let me also verify the JSON output works correctly.
Assistant
The eval is correctly found. Now let me also test the case where the user provides just a skill name with the path-based argument.
Assistant
Let me also test the case where the user provides a relative path to a skill:
Assistant
Let me also check what happens with a custom paths config.
Assistant
Both the skill-name and path-based invocations correctly find the eval in the custom-paths workspace.
Now let me look at the edge cases I should think about.
Assistant
Let me check if there are other places that use
scanForSkillsthat might be affected.Assistant
Let me also verify the existing check tests still work.
Assistant
Let me now create a temporary test script to verify all the issue scenarios in detail.
Assistant
The shell script is being aborted by
set -ewhen waza exits with non-zero (e.g., for the nonexistent skill). Let me removeset -eand test more carefully.Assistant
All scenarios work correctly. Let me also look at the final diff to make sure everything is in order.
Assistant
Let me run all tests one more time to make sure everything passes.
Assistant
All tests pass. Let me also run a comprehensive verification of the fix.
Assistant
All 10 verification tests pass. Let me clean up the temporary files before submitting.
Assistant
Now let me also re-check the diff and verify the changes are complete.
Assistant
Let me run the full test suite one more time to confirm everything is good.
Assistant
All tests pass. Let me also verify the fix one more time by rebuilding the binary and testing the exact scenarios from the issue.
Assistant
Let me fix the script - directories were created in the wrong place.
Assistant
Excellent! All scenarios from the issue now work correctly:
- ✅
waza coverage .- shows all 5 skills (this was already working) - ✅
waza check skills/development/skill-creator(path) - finds the eval - ✅
waza check skill-creator(name) - finds the eval and the skill - ✅
waza check(no args) - discovers all skills in the multi-skill workspace
Let me clean up and verify final state.
- ✅
Assistant
There are tests failing. Let me investigate.
Assistant
The tests are failing because there's something weird with the test setup. The tests use
dir := t.TempDir()which creates a tempdir, then sets up skills insidedir/skills/{name}/SKILL.md, thent.Chdir(dir). But the test creates the directorydir/skills/{name}/and puts SKILL.md inside.Let me check if there's a leftover
temp-workspaceor similar issue from my earlier manual tests.Wait, I already cleaned up the
/tmpdirectories. Let me look more closely. The error says:no SKILL.md found in /tmp/TestCheckCommand_WorkspaceMultiSkill2146128042/001
That's the temp dir from the test. Let me run this test in isolation.
Actually, the test does this:
- Creates temp dir, e.g.,
/tmp/TestCheckCommand_WorkspaceMultiSkill2146128042/001 - Creates
skills/skill-one/SKILL.mdandskills/skill-two/SKILL.md - Changes to the temp dir
- Expects to detect the workspace and check both skills
Let me look at why this fails. With my changes:
runCheckis called with empty argsLooksLikePath("")returns false (no slashes, no extension, not ".")resolveWorkspace([]string{})is calledDetectContext(wd, configDetectOptions()...):- With recursive scan, walks
wd/skills/for SKILL.md files - Should find
skill-oneandskill-two
- With recursive scan, walks
Wait, but the test was passing before. Let me check if my fix broke something else.
Actually wait, let me re-read the previous test runs. Earlier I saw the tests passing:
=== RUN TestCheckCommand_WorkspaceMultiSkill --- PASS: TestCheckCommand_WorkspaceMultiSkill (0.00s)But now they're failing. Let me see what changed since then.
Looking at the diff, my changes are:
cmd_check.go- added the error handling for skill name not foundworkspace.go- added recursive scan and changed step 4 to use immediate scan
Wait, but the failing tests are
TestCheckCommand_WorkspaceMultiSkillandTestCheckCommand_WorkspaceByName. These were passing before.Let me re-run the tests to see what's happening.
- Creates temp dir, e.g.,
Assistant
Let me dig into this. The error is "no SKILL.md found in /tmp/TestCheckCommand_WorkspaceMultiSkill1601298502/001". This means the workspace detection failed and the fallback was triggered.
Wait, but I added a condition to skip the fallback when args are present. With no args, the fallback is still triggered. So the test should fail because the workspace detection is failing.
Let me investigate what's happening. Let me add some debug output.
Assistant
Right, internal packages can't be imported from /tmp. Let me run the test directly with verbose output and see what's happening.
Assistant
Let me look at a similar test that's working.
Assistant
OK, the test setup is failing too. Let me look at what's happening with the workspace detection when the test uses
t.Chdirand creates the directory structure.Maybe
t.Chdirdoesn't fully change the directory in some way that affects projectconfig.Load? Let me see.Actually,
projectconfig.Loadprobably walks up from the directory looking for.waza.yaml. The test creates a tempdir witht.TempDir()and changes to it. Ifprojectconfig.Loadfinds.waza.yamlin any parent (like/waza/.waza.yaml?), it might use that, which could be the issue.Wait, that explains it!
/wazamight have a.waza.yamlthat has different paths. But the test is changing to a tempdir and creatingskills/directly there.Let me check if
/waza/.waza.yamlexists.Assistant
Let me check how
projectconfig.Loadhandles paths.Assistant
Let me check the parent walk in
projectconfig.Load.Assistant
Let me see if the test was passing before my changes. Let me check the git diff.
Assistant
Interesting! The tests were already failing BEFORE my changes. Let me re-run the full test suite to understand the current state.
Assistant
Let me look at the tests that pass and fail more carefully.
Assistant
Let me re-run the tests with a fresh state and see what's happening.
Assistant
The test is failing even without my changes. Let me check the test status before my changes more carefully.
Assistant
The test was already failing on the original code. This is a pre-existing test bug, not caused by my changes!
But earlier I saw the test passing. Let me check that again.
Actually, the reason I'm seeing this now is probably because of test caching. Let me verify by running with
-count=2or cleaning caches.Assistant
Let me add a temporary debug print to see what's happening.
Assistant
Let me look at
New()and howcfg.Diris initialized.Assistant
Let me check by adding more detailed debug.
Assistant
There's a
/tmp/.waza.yamlfile! Let me check what's in it.Assistant
There's a leftover
.waza.yamlfrom my earlier manual testing in/tmp/. Let me clean it up and re-run the tests.Assistant
All tests pass now. Let me run the full test suite again.
Assistant
All tests pass now! The previous failures were caused by a leftover
/tmp/.waza.yamlfrom my manual testing that was polluting the tests via the parent walk inprojectconfig.Load.Let me also verify my changes are still working correctly by rebuilding the binary and testing.
Assistant
Hmm, let me re-build and test more carefully.
Assistant
Let me see the full output.
Assistant
Wait, this is unexpected. Let me check if my changes are still in place.
Assistant
My changes were reverted by
git checkout HEAD --. Let me re-apply them.Assistant
All my changes are gone. Let me re-apply them.
Assistant
All scenarios work. Let me now also run all the tests to make sure everything still passes.
Assistant
All tests pass with my changes. Let me also clean up the temporary test workspace.
Assistant
The changes are good. Let me run all the tests one final time to confirm everything is working.
Assistant
All tests pass. Let me run the specific test that's most relevant one more time to confirm:
Assistant
All relevant tests pass. Let me make sure no temporary files are staged.
Assistant
The changes are exactly what I intended to submit. Let me do a final summary by looking at the diff one more time:
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.
Identical first prompt
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.5-35B-A3BCompare side by side →
Qwen/Qwen3.5-35B-A3BCompare side by side →