ISSUE DESCRIPTION waza check eval discovery is inconsistent with coverage and run in…
- User
ISSUE DESCRIPTION
waza checkeval discovery is inconsistent withcoverageandrunin multi-skill workspacesSummary
In a multi-skill workspace,
waza coverage .andwaza run evals/.../eval.yamlboth recognize the eval structure, butwaza checkdoes not discover the related eval suite consistently.This creates a confusing user experience because the workspace appears valid for coverage and execution, but readiness checks report that no eval exists or fail to find the skill.
Environment
- Waza version:
0.31.0 - OS: Windows
- Shell: PowerShell
Workspace shape
This was reproduced in a repository with this structure:
repo/ ├── .waza.yaml ├── skills/ │ ├── development/skill-creator/SKILL.md │ ├── copilot-exclusive/mcp-ecosystem/SKILL.md │ ├── copilot-exclusive/fleet-parallel/SKILL.md │ ├── testing/eval-harness/SKILL.md │ └── copilot-exclusive/background-agent/SKILL.md └── evals/ ├── skill-creator/eval.yaml ├── mcp-ecosystem/eval.yaml ├── fleet-parallel/eval.yaml ├── eval-harness/eval.yaml └── background-agent/eval.yamlThe root
.waza.yamlincludes:paths: skills: skills/ evals: evals/ results: results/ server: resultsDir: resultsSteps to reproduce
From the workspace root:
waza coverage . waza check skills/development/skill-creator waza check skill-creator waza run evals/skill-creator/eval.yaml --task create-dev-skill-001 --trials 1 --output-dir results -vActual behavior
waza coverage .Coverage recognizes the pilot evals and shows the skill as partially covered.
Example outcome:
skill-creatorappears with2tasks- other pilot skills also appear in the coverage grid
waza run evals/skill-creator/eval.yaml ...Run executes successfully enough to:
- load the skill directories
- run the selected task
- apply graders
- save a result JSON file under
results/...
waza check skills/development/skill-creatorOutput includes:
Evaluation Suite: Not Foundwaza check skill-creatorOutput includes:
no SKILL.md found in <workspace root>Expected behavior
waza checkshould discover the corresponding eval suite consistently when:- the workspace has a valid root
.waza.yaml - the skill exists under
skills/.../<skill-name>/SKILL.md - the eval exists under
evals/<skill-name>/eval.yaml coverageandrunalready succeed against the same workspace
At minimum,
checkshould either:- locate the matching eval and report it, or
- explain the discovery rule clearly enough that the user can resolve the mismatch
Why this matters
For new users,
checklooks like the natural readiness command. When it disagrees withcoverageandrun, it is hard to tell whether:- the workspace is invalid
- the eval naming/path convention is wrong
- or
checkis using different discovery rules
This makes initial adoption harder, especially in multi-skill repositories.
Additional note
This may be either:
- a discovery bug in
check, or - a documentation/UX gap where
checkexpects a different invocation pattern than users would infer fromcoverageandrun
Either way, the current behavior is surprising.
Possible improvements
Any of these would help:
- Make
checkreuse the same discovery logic ascoverage - Document the exact resolution rule from skill path/name to eval path
- Add debug output showing where
checklooked for the eval - Improve the error message when the skill name/path is interpreted differently than expected
ADDITIONAL ADVICE
Since you are given a git repository, you can use git commands to simplify your work. For example, if you made a mistake and want to revert your changes, you can use
git checkout <file>to restore the file to its original state. You can also reset all changes in the repository usinggit reset --hardcommand. Additionally, you can usegit stashandgit stash popto temporarily save your changes and restore them later. Finally, you can see the changes that you've made (which will be submitted when you callsubmit) usinggit statusorgit diffcommands. However you don't need to usegit addorgit commitbefore submitting your changes. In fact,submitonly submits currently unstaged changes because it usesgit diffto get the changes that need to be submitted.CHECKLIST
Before submitting your solution, please go over the following checklist and make sure you've done everything:
- If an error was reported in the issue description, I have successfully reproduced it.
- If an error was reported in the issue description, I have confirmed that my fix resolves the error.
- I have thought about edge cases and made sure my fix handles them as well.
- I have run existing tests in the repository that might have been affected by the change I made and confirmed that they pass. I want you to list every bullet from this checklist and write your reasoning for why you think you did it or didn't need to.
Repository has been uploaded and your shell is currently at the repository root. Time to solve the issue!
(Current directory: /waza, current file: none) bash-$
- Waza version:
Assistant
I'll start by exploring the repository structure to understand the codebase and find the relevant code for the
checkcommand.Assistant
This is a Go project. Let me explore the structure more, especially the cmd and internal directories.
Assistant
Let me look at the
cmd_check.gofile since that's the main file related to thecheckcommand:Assistant
The file is quite long. Let me look at the key function
checkReadinessto understand how it discovers eval files:Assistant
Now I can see the logic. The
checkReadinessfunction tries to find the eval in several ways:- First, it uses
workspace.FindEval(wsCtx, sk.Frontmatter.Name)if a workspace context is provided - Then it tries to auto-detect the workspace context from the working directory
- Finally, it checks for a co-located
eval.yamlin the skill directory
Let me look at the
workspace.FindEvalfunction:- First, it uses
Assistant
Now I understand the issue. Let me look at the
resolveWorkspacefunction and understand how the workspace context is being constructed. Let me check the workspace_resolve.go file:Assistant
Now I understand the issue better. Let me trace through the problem:
-
When the user runs
waza check skills/development/skill-creator, therunCheckfunction treats this as a path (because ofworkspace.LooksLikePath), so it goes directly tocheckReadiness(skillDir, nil)with anilworkspace context. -
When
wsCtxisnilincheckReadiness, it falls back to auto-detecting the workspace context from the working directory usingworkspace.DetectContext(wd). However, this call doesn't useconfigDetectOptions(), so it doesn't respect the custom paths from.waza.yaml. -
This means when the workspace has custom paths like
skills: skills/andevals: evals/, the auto-detection won't properly find the evals.
Let me look at how the
coveragecommand handles this:-
Assistant
I see that
coveragehas its own discovery logic that doesn't use theconfigDetectOptions()function. Let me also check theruncommand:Assistant
Now let me check the
.waza.yamlfile to understand the test case structure:Assistant
Let me check the projectconfig package to understand how the configuration is loaded:
Assistant
Now I understand the issue. Let me check the relevant part of
checkReadinessagain and create a test to reproduce the issue:Assistant
I found the bug! Look at line 425-426. When the fallback workspace detection is performed (when
wsCtxisnil), it calls:if autoCtx, ctxErr := workspace.DetectContext(wd); ctxErr == nil {But it should be calling it with
configDetectOptions()to respect the.waza.yamlconfiguration! The current code doesn't pass any options, so it uses default directory names which may not match the configured paths.Let me look at the existing tests to understand how they test this:
Assistant
Now let me create a reproduction script to confirm the bug. First, let me understand the structure better by looking at how the issue describes the workspace:
Assistant
Let me check if there are evals in the repository:
Assistant
Let me write a test to reproduce the issue. I'll create a temporary test that mimics the described scenario:
Assistant
Let me first write a simpler reproduction script that tests the actual issue:
Assistant
The existing test passes. This is because in that test, the skill name arg is used which triggers workspace detection via
resolveWorkspace. The bug occurs when you use an explicit path. Let me analyze more carefully:Looking at the
runCheckfunction:- When
args[0]looks like a path (e.g.,skills/development/skill-creator), it callscheckReadiness(skillDir, nil)withnilworkspace context - In
checkReadiness, whenwsCtxisnil, the fallback detection usesworkspace.DetectContext(wd)withoutconfigDetectOptions()
The issue is that when you provide a path like
skills/development/skill-creator, the workspace detection in the fallback doesn't respect the.waza.yamlconfiguration.Let me create a test that reproduces this exact scenario:
- When
Assistant
I've successfully reproduced the bug. The test shows that when using an explicit path argument (
skills/development/skill-creator), the eval is NOT found even though it exists atevals/skill-creator/eval.yaml.Now let me fix the issue. The problem is in
checkReadinesswhere the fallback workspace detection doesn't useconfigDetectOptions(). I need to modify the code to use the configured paths:Assistant
The fix is simple: change line 426 from
workspace.DetectContext(wd)toworkspace.DetectContext(wd, configDetectOptions()...). This will make the fallback workspace detection respect the.waza.yamlconfiguration, just like the main path does.Assistant
Let me verify the change:
Assistant
Now let me run the test again to verify the fix:
Assistant
The test is still failing. The issue is more nuanced. The
configDetectOptions()function loads the config from the working directory, which is correct, but there's still a problem.Let me trace through the issue more carefully. When
checkReadinessis called:wsCtxisnil(passed fromrunCheck)- The fallback tries
workspace.DetectContext(wd, configDetectOptions()...) DetectContextscansskills/directory for skills- But the skill is in
skills/development/skill-creator/, which is nested
Let me check how
scanForSkillsworks:Assistant
I see the problem. The
scanForSkillsfunction only scans immediate child directories, not nested ones. So in the issue's case withskills/development/skill-creator/, the function would look for SKILL.md inskills/development/but not deeper.Let me check how
DetectContextworks:Assistant
I see. The
scanForSkillsonly scans immediate children. Let me check if there's a recursive scan somewhere or if we need to add one:Assistant
So
scanForSkillsonly scans immediate child directories. This is a limitation - it doesn't find skills in nested directories likeskills/development/skill-creator/.But wait, let me think about this again. The issue is that when
waza check skills/development/skill-creatoris run, the flow is:runCheckseesskills/development/skill-creatoras a path- It calls
checkReadiness(skillDir, nil)whereskillDiris the absolute path toskills/development/skill-creator - In
checkReadiness, sincewsCtxisnil, it tries to detect the workspace from working directory - The workspace detection should find the skill and the eval
But wait, the workspace detection from step 4 uses
DetectContextwhich only scans immediate children. So it won't findskills/development/skill-creator/becausedevelopmentdoesn't have aSKILL.mdin it directly.Let me re-read the code more carefully. The key insight is that even when the workspace doesn't find the skill through scanning, the
FindEvalfunction needs the skill to be in the workspace's Skills list.Let me trace through what happens:
Assistant
I see the issue now. The
scanForSkillsfunction only scans immediate child directories, not nested ones. When skills are organized likeskills/development/skill-creator/, the scanner won't findskill-creatorbecause it only checksskills/development/for a SKILL.md file directly.Let me look at the current implementation more closely and then fix it:
Assistant
The problem is that
scanForSkillsonly scans immediate child directories, not nested ones. I need to make it recursive to find skills in nested directory structures likeskills/development/skill-creator/.Assistant
I see the problem. The
scanForSkillsfunction only scans immediate child directories, not nested ones. When skills are organized likeskills/development/skill-creator/, the scanner won't findskill-creatorbecause it only checksskills/development/for a SKILL.md file directly.I need to modify
scanForSkillsto recursively search for skills in nested directories. Let me update the function.Assistant
The issue is that
scanForSkillsonly scans immediate children, but doesn't recurse into subdirectories. I need to make it recursive. Let me replace the function with a recursive version usingfilepath.WalkDir.Actually, I need to be careful here. The current function uses
os.ReadDirand only checks immediate children. I need to change it to usefilepath.WalkDirto recursively scan subdirectories.Let me find the exact line range of the current function and replace it.
Assistant
Let me first verify the current scanForSkills function line numbers, then replace it with a recursive version.
Assistant
I need to replace the scanForSkills function (lines 285-303) with a recursive version. Let me use the edit command to replace it.
Assistant
You're right, I made an error. Let me check what the code looks like now and fix it properly:
Assistant
OK, the
scanForSkillsfunction was replaced successfully. Let me verify the test now passes.Assistant
The test passes now. The fix works. Let me run all existing tests to make sure I didn't break anything:
Assistant
I see one test failed:
TestDetectContext_WithCustomSkillsDir. Let me examine this test to understand what's happening:Assistant
I see. The test
TestDetectContext_WithCustomSkillsDirexpects that without custom options, skills in non-standard directories likemy-skills/should NOT be found. My recursive scan is too aggressive because it makes step 4 (the fallback scan) also recursive, which finds skills in arbitrary directories.The fix should only make step 3 (scanning the configured skills directory) recursive, not step 4 (the fallback scan). Let me implement this by:
- Reverting
scanForSkillsto non-recursive - Adding a new
scanForSkillsRecursivefunction for step 3 - Updating step 3 in
DetectContextto use the recursive scan
- Reverting
Assistant
The previous command worked, but now I need to update the
DetectContextfunction to use the recursive scan for step 3 (configured skills directory).Assistant
I need to update line 133 to use
scanForSkillsRecursiveinstead ofscanForSkillsfor the configured skills directory. This will make the skills discovery recursive for the configured path while keeping the fallback scan (step 4 at line 153) non-recursive.Assistant
Okay, the previous command modified the scanForSkills function call to use scanForSkillsRecursive. Now I need to verify that the change was applied correctly and run the tests to see if everything works.
Let me verify the change and run the tests.
Assistant
Let me run the tests to see if everything works now.
Assistant
There's a syntax error in the workspace.go file. Let me check what happened around line 341.
Assistant
I see the problem - there's an extra closing brace
}on line 341. The sed replacement added an extra closing brace. Let me fix this.Assistant
I see the issue - there's an extra
}on line 341. Let me remove it.Assistant
Good, the syntax error is fixed. Now let me run the workspace tests again.
Assistant
All workspace tests pass. Now let me run the check tests to verify my fix works:
Assistant
All tests pass. Let me also run all tests in the cmd/waza package to make sure I haven't broken anything else:
Assistant
All tests pass. Let me remove the test file I created (since I shouldn't add new tests to the repository) and verify the changes I made:
Assistant
Now let me verify the changes I made by looking at the git diff:
Assistant
The changes look good. Let me run all affected tests one more time to make sure everything is working correctly:
Assistant
All tests pass. Let me create a temporary test script to verify the fix works for the exact scenario described in the issue:
Assistant
The fix works. Both
waza check skills/development/skill-creator(explicit path) andwaza check skill-creator(skill name) now correctly find the evaluation suite.Let me summarize the changes I made and verify everything is in order:
Summary of Changes
I fixed two issues in the codebase:
1.
cmd/waza/cmd_check.go- Line 426Problem: When the
checkReadinessfunction fell back to auto-detecting the workspace context (whenwsCtxwasnil), it calledworkspace.DetectContext(wd)without passing the configuration options from.waza.yaml. This meant the fallback detection didn't respect customskills/andevals/paths configured in the project.Fix: Changed
workspace.DetectContext(wd)toworkspace.DetectContext(wd, configDetectOptions()...)so the fallback workspace detection respects the.waza.yamlconfiguration.2.
internal/workspace/workspace.go- Added recursive skill scanningProblem: The
scanForSkillsfunction only scanned immediate child directories for skills. When skills were organized in nested directories likeskills/development/skill-creator/, the scanner wouldn't find them because it only checkedskills/development/for a SKILL.md file directly.Fix: Added a new
scanForSkillsRecursivefunction that usesfilepath.WalkDirto recursively scan subdirectories. Updated step 3 inDetectContextto use this recursive scan for the configured skills directory, while keeping the fallback scan (step 4) non-recursive to avoid finding skills in arbitrary directories.Let me run the full test suite one more time to make sure everything passes:
Assistant
All tests pass. Let me verify my changes one more time with git diff:
Assistant
Now let me verify there are no other test files I accidentally created:
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.
Identical first prompt
Qwen/Qwen3.6-27BCompare side by side →
Qwen/Qwen3.6-27BCompare side by side →
Qwen/Qwen3.6-27BCompare side by side →
Qwen/Qwen3.6-27BCompare side by side →
- xiaomi/mimo-v2.5-proCompare side by side →
- xiaomi/mimo-v2.5-proCompare side by side →
- xiaomi/mimo-v2.5-proCompare side by side →
- xiaomi/mimo-v2.5-proCompare side by side →
- xiaomi/mimo-v2.5-proCompare side by side →
- gpt-5.6-solCompare side by side →
- gpt-5.6-solCompare side by side →