Skip to content
TheChatArchiveThe Chat Archive
QwenQwen/Qwen3.6-27B· alibabaHugging Face ibragim-bad/swe_rebench_07_2026_trajectories

ISSUE DESCRIPTION waza check eval discovery is inconsistent with coverage and run in…

  1. User

    ISSUE DESCRIPTION

    waza check eval discovery is inconsistent with coverage and run in multi-skill workspaces

    Summary

    In a multi-skill workspace, waza coverage . and waza run evals/.../eval.yaml both recognize the eval structure, but waza check does not discover the related eval suite consistently.

    This creates a confusing user experience because the workspace appears valid for coverage and execution, but readiness checks report that no eval exists or fail to find the skill.

    Environment

    • Waza version: 0.31.0
    • OS: Windows
    • Shell: PowerShell

    Workspace shape

    This was reproduced in a repository with this structure:

    repo/
    ├── .waza.yaml
    ├── skills/
    │   ├── development/skill-creator/SKILL.md
    │   ├── copilot-exclusive/mcp-ecosystem/SKILL.md
    │   ├── copilot-exclusive/fleet-parallel/SKILL.md
    │   ├── testing/eval-harness/SKILL.md
    │   └── copilot-exclusive/background-agent/SKILL.md
    └── evals/
        ├── skill-creator/eval.yaml
        ├── mcp-ecosystem/eval.yaml
        ├── fleet-parallel/eval.yaml
        ├── eval-harness/eval.yaml
        └── background-agent/eval.yaml
    

    The root .waza.yaml includes:

    paths:
      skills: skills/
      evals: evals/
      results: results/
    
    server:
      resultsDir: results
    

    Steps to reproduce

    From the workspace root:

    waza coverage .
    waza check skills/development/skill-creator
    waza check skill-creator
    waza run evals/skill-creator/eval.yaml --task create-dev-skill-001 --trials 1 --output-dir results -v
    

    Actual behavior

    waza coverage .

    Coverage recognizes the pilot evals and shows the skill as partially covered.

    Example outcome:

    • skill-creator appears with 2 tasks
    • other pilot skills also appear in the coverage grid

    waza run evals/skill-creator/eval.yaml ...

    Run executes successfully enough to:

    • load the skill directories
    • run the selected task
    • apply graders
    • save a result JSON file under results/...

    waza check skills/development/skill-creator

    Output includes:

    Evaluation Suite: Not Found
    

    waza check skill-creator

    Output includes:

    no SKILL.md found in <workspace root>
    

    Expected behavior

    waza check should discover the corresponding eval suite consistently when:

    1. the workspace has a valid root .waza.yaml
    2. the skill exists under skills/.../<skill-name>/SKILL.md
    3. the eval exists under evals/<skill-name>/eval.yaml
    4. coverage and run already succeed against the same workspace

    At minimum, check should either:

    • locate the matching eval and report it, or
    • explain the discovery rule clearly enough that the user can resolve the mismatch

    Why this matters

    For new users, check looks like the natural readiness command. When it disagrees with coverage and run, it is hard to tell whether:

    • the workspace is invalid
    • the eval naming/path convention is wrong
    • or check is using different discovery rules

    This makes initial adoption harder, especially in multi-skill repositories.

    Additional note

    This may be either:

    • a discovery bug in check, or
    • a documentation/UX gap where check expects a different invocation pattern than users would infer from coverage and run

    Either way, the current behavior is surprising.

    Possible improvements

    Any of these would help:

    1. Make check reuse the same discovery logic as coverage
    2. Document the exact resolution rule from skill path/name to eval path
    3. Add debug output showing where check looked for the eval
    4. Improve the error message when the skill name/path is interpreted differently than expected

    ADDITIONAL ADVICE

    Since you are given a git repository, you can use git commands to simplify your work. For example, if you made a mistake and want to revert your changes, you can use git checkout <file> to restore the file to its original state. You can also reset all changes in the repository using git reset --hard command. Additionally, you can use git stash and git stash pop to temporarily save your changes and restore them later. Finally, you can see the changes that you've made (which will be submitted when you call submit) using git status or git diff commands. However you don't need to use git add or git commit before submitting your changes. In fact, submit only submits currently unstaged changes because it uses git diff to get the changes that need to be submitted.

    CHECKLIST

    Before submitting your solution, please go over the following checklist and make sure you've done everything:

    • If an error was reported in the issue description, I have successfully reproduced it.
    • If an error was reported in the issue description, I have confirmed that my fix resolves the error.
    • I have thought about edge cases and made sure my fix handles them as well.
    • I have run existing tests in the repository that might have been affected by the change I made and confirmed that they pass. I want you to list every bullet from this checklist and write your reasoning for why you think you did it or didn't need to.

    Repository has been uploaded and your shell is currently at the repository root. Time to solve the issue!

    (Current directory: /waza, current file: none) bash-$

  2. Qwen Assistant

    I'll start by exploring the repository structure to understand the codebase and find the relevant code for the check command.

  3. Qwen Assistant

    This is a Go project. Let me explore the structure more, especially the cmd and internal directories.

  4. Qwen Assistant

    Let me look at the cmd_check.go file since that's the main file related to the check command:

  5. Qwen Assistant

    The file is quite long. Let me look at the key function checkReadiness to understand how it discovers eval files:

  6. Qwen Assistant

    Now I can see the logic. The checkReadiness function tries to find the eval in several ways:

    1. First, it uses workspace.FindEval(wsCtx, sk.Frontmatter.Name) if a workspace context is provided
    2. Then it tries to auto-detect the workspace context from the working directory
    3. Finally, it checks for a co-located eval.yaml in the skill directory

    Let me look at the workspace.FindEval function:

  7. Qwen Assistant

    Now I understand the issue. Let me look at the resolveWorkspace function and understand how the workspace context is being constructed. Let me check the workspace_resolve.go file:

  8. Qwen Assistant

    Now I understand the issue better. Let me trace through the problem:

    1. When the user runs waza check skills/development/skill-creator, the runCheck function treats this as a path (because of workspace.LooksLikePath), so it goes directly to checkReadiness(skillDir, nil) with a nil workspace context.

    2. When wsCtx is nil in checkReadiness, it falls back to auto-detecting the workspace context from the working directory using workspace.DetectContext(wd). However, this call doesn't use configDetectOptions(), so it doesn't respect the custom paths from .waza.yaml.

    3. This means when the workspace has custom paths like skills: skills/ and evals: evals/, the auto-detection won't properly find the evals.

    Let me look at how the coverage command handles this:

  9. Qwen Assistant

    I see that coverage has its own discovery logic that doesn't use the configDetectOptions() function. Let me also check the run command:

  10. Qwen Assistant

    Now let me check the .waza.yaml file to understand the test case structure:

  11. Qwen Assistant

    Let me check the projectconfig package to understand how the configuration is loaded:

  12. Qwen Assistant

    Now I understand the issue. Let me check the relevant part of checkReadiness again and create a test to reproduce the issue:

  13. Qwen Assistant

    I found the bug! Look at line 425-426. When the fallback workspace detection is performed (when wsCtx is nil), it calls:

    if autoCtx, ctxErr := workspace.DetectContext(wd); ctxErr == nil {
    

    But it should be calling it with configDetectOptions() to respect the .waza.yaml configuration! The current code doesn't pass any options, so it uses default directory names which may not match the configured paths.

    Let me look at the existing tests to understand how they test this:

  14. Qwen Assistant

    Now let me create a reproduction script to confirm the bug. First, let me understand the structure better by looking at how the issue describes the workspace:

  15. Qwen Assistant

    Let me check if there are evals in the repository:

  16. Qwen Assistant

    Let me write a test to reproduce the issue. I'll create a temporary test that mimics the described scenario:

  17. Qwen Assistant

    Let me first write a simpler reproduction script that tests the actual issue:

  18. Qwen Assistant

    The existing test passes. This is because in that test, the skill name arg is used which triggers workspace detection via resolveWorkspace. The bug occurs when you use an explicit path. Let me analyze more carefully:

    Looking at the runCheck function:

    1. When args[0] looks like a path (e.g., skills/development/skill-creator), it calls checkReadiness(skillDir, nil) with nil workspace context
    2. In checkReadiness, when wsCtx is nil, the fallback detection uses workspace.DetectContext(wd) without configDetectOptions()

    The issue is that when you provide a path like skills/development/skill-creator, the workspace detection in the fallback doesn't respect the .waza.yaml configuration.

    Let me create a test that reproduces this exact scenario:

  19. Qwen Assistant

    I've successfully reproduced the bug. The test shows that when using an explicit path argument (skills/development/skill-creator), the eval is NOT found even though it exists at evals/skill-creator/eval.yaml.

    Now let me fix the issue. The problem is in checkReadiness where the fallback workspace detection doesn't use configDetectOptions(). I need to modify the code to use the configured paths:

  20. Qwen Assistant

    The fix is simple: change line 426 from workspace.DetectContext(wd) to workspace.DetectContext(wd, configDetectOptions()...). This will make the fallback workspace detection respect the .waza.yaml configuration, just like the main path does.

  21. Qwen Assistant

    Let me verify the change:

  22. Qwen Assistant

    Now let me run the test again to verify the fix:

  23. Qwen Assistant

    The test is still failing. The issue is more nuanced. The configDetectOptions() function loads the config from the working directory, which is correct, but there's still a problem.

    Let me trace through the issue more carefully. When checkReadiness is called:

    1. wsCtx is nil (passed from runCheck)
    2. The fallback tries workspace.DetectContext(wd, configDetectOptions()...)
    3. DetectContext scans skills/ directory for skills
    4. But the skill is in skills/development/skill-creator/, which is nested

    Let me check how scanForSkills works:

  24. Qwen Assistant

    I see the problem. The scanForSkills function only scans immediate child directories, not nested ones. So in the issue's case with skills/development/skill-creator/, the function would look for SKILL.md in skills/development/ but not deeper.

    Let me check how DetectContext works:

  25. Qwen Assistant

    I see. The scanForSkills only scans immediate children. Let me check if there's a recursive scan somewhere or if we need to add one:

  26. Qwen Assistant

    So scanForSkills only scans immediate child directories. This is a limitation - it doesn't find skills in nested directories like skills/development/skill-creator/.

    But wait, let me think about this again. The issue is that when waza check skills/development/skill-creator is run, the flow is:

    1. runCheck sees skills/development/skill-creator as a path
    2. It calls checkReadiness(skillDir, nil) where skillDir is the absolute path to skills/development/skill-creator
    3. In checkReadiness, since wsCtx is nil, it tries to detect the workspace from working directory
    4. The workspace detection should find the skill and the eval

    But wait, the workspace detection from step 4 uses DetectContext which only scans immediate children. So it won't find skills/development/skill-creator/ because development doesn't have a SKILL.md in it directly.

    Let me re-read the code more carefully. The key insight is that even when the workspace doesn't find the skill through scanning, the FindEval function needs the skill to be in the workspace's Skills list.

    Let me trace through what happens:

  27. Qwen Assistant

    I see the issue now. The scanForSkills function only scans immediate child directories, not nested ones. When skills are organized like skills/development/skill-creator/, the scanner won't find skill-creator because it only checks skills/development/ for a SKILL.md file directly.

    Let me look at the current implementation more closely and then fix it:

  28. Qwen Assistant

    The problem is that scanForSkills only scans immediate child directories, not nested ones. I need to make it recursive to find skills in nested directory structures like skills/development/skill-creator/.

  29. Qwen Assistant

    I see the problem. The scanForSkills function only scans immediate child directories, not nested ones. When skills are organized like skills/development/skill-creator/, the scanner won't find skill-creator because it only checks skills/development/ for a SKILL.md file directly.

    I need to modify scanForSkills to recursively search for skills in nested directories. Let me update the function.

  30. Qwen Assistant

    The issue is that scanForSkills only scans immediate children, but doesn't recurse into subdirectories. I need to make it recursive. Let me replace the function with a recursive version using filepath.WalkDir.

    Actually, I need to be careful here. The current function uses os.ReadDir and only checks immediate children. I need to change it to use filepath.WalkDir to recursively scan subdirectories.

    Let me find the exact line range of the current function and replace it.

  31. Qwen Assistant

    Let me first verify the current scanForSkills function line numbers, then replace it with a recursive version.

  32. Qwen Assistant

    I need to replace the scanForSkills function (lines 285-303) with a recursive version. Let me use the edit command to replace it.

  33. Qwen Assistant

    You're right, I made an error. Let me check what the code looks like now and fix it properly:

  34. Qwen Assistant

    OK, the scanForSkills function was replaced successfully. Let me verify the test now passes.

  35. Qwen Assistant

    The test passes now. The fix works. Let me run all existing tests to make sure I didn't break anything:

  36. Qwen Assistant

    I see one test failed: TestDetectContext_WithCustomSkillsDir. Let me examine this test to understand what's happening:

  37. Qwen Assistant

    I see. The test TestDetectContext_WithCustomSkillsDir expects that without custom options, skills in non-standard directories like my-skills/ should NOT be found. My recursive scan is too aggressive because it makes step 4 (the fallback scan) also recursive, which finds skills in arbitrary directories.

    The fix should only make step 3 (scanning the configured skills directory) recursive, not step 4 (the fallback scan). Let me implement this by:

    1. Reverting scanForSkills to non-recursive
    2. Adding a new scanForSkillsRecursive function for step 3
    3. Updating step 3 in DetectContext to use the recursive scan
  38. Qwen Assistant

    The previous command worked, but now I need to update the DetectContext function to use the recursive scan for step 3 (configured skills directory).

  39. Qwen Assistant

    I need to update line 133 to use scanForSkillsRecursive instead of scanForSkills for the configured skills directory. This will make the skills discovery recursive for the configured path while keeping the fallback scan (step 4 at line 153) non-recursive.

  40. Qwen Assistant

    Okay, the previous command modified the scanForSkills function call to use scanForSkillsRecursive. Now I need to verify that the change was applied correctly and run the tests to see if everything works.

    Let me verify the change and run the tests.

  41. Qwen Assistant

    Let me run the tests to see if everything works now.

  42. Qwen Assistant

    There's a syntax error in the workspace.go file. Let me check what happened around line 341.

  43. Qwen Assistant

    I see the problem - there's an extra closing brace } on line 341. The sed replacement added an extra closing brace. Let me fix this.

  44. Qwen Assistant

    I see the issue - there's an extra } on line 341. Let me remove it.

  45. Qwen Assistant

    Good, the syntax error is fixed. Now let me run the workspace tests again.

  46. Qwen Assistant

    All workspace tests pass. Now let me run the check tests to verify my fix works:

  47. Qwen Assistant

    All tests pass. Let me also run all tests in the cmd/waza package to make sure I haven't broken anything else:

  48. Qwen Assistant

    All tests pass. Let me remove the test file I created (since I shouldn't add new tests to the repository) and verify the changes I made:

  49. Qwen Assistant

    Now let me verify the changes I made by looking at the git diff:

  50. Qwen Assistant

    The changes look good. Let me run all affected tests one more time to make sure everything is working correctly:

  51. Qwen Assistant

    All tests pass. Let me create a temporary test script to verify the fix works for the exact scenario described in the issue:

  52. Qwen Assistant

    The fix works. Both waza check skills/development/skill-creator (explicit path) and waza check skill-creator (skill name) now correctly find the evaluation suite.

    Let me summarize the changes I made and verify everything is in order:

    Summary of Changes

    I fixed two issues in the codebase:

    1. cmd/waza/cmd_check.go - Line 426

    Problem: When the checkReadiness function fell back to auto-detecting the workspace context (when wsCtx was nil), it called workspace.DetectContext(wd) without passing the configuration options from .waza.yaml. This meant the fallback detection didn't respect custom skills/ and evals/ paths configured in the project.

    Fix: Changed workspace.DetectContext(wd) to workspace.DetectContext(wd, configDetectOptions()...) so the fallback workspace detection respects the .waza.yaml configuration.

    2. internal/workspace/workspace.go - Added recursive skill scanning

    Problem: The scanForSkills function only scanned immediate child directories for skills. When skills were organized in nested directories like skills/development/skill-creator/, the scanner wouldn't find them because it only checked skills/development/ for a SKILL.md file directly.

    Fix: Added a new scanForSkillsRecursive function that uses filepath.WalkDir to recursively scan subdirectories. Updated step 3 in DetectContext to use this recursive scan for the configured skills directory, while keeping the fallback scan (step 4) non-recursive to avoid finding skills in arbitrary directories.

    Let me run the full test suite one more time to make sure everything passes:

  53. Qwen Assistant

    All tests pass. Let me verify my changes one more time with git diff:

  54. Qwen Assistant

    Now let me verify there are no other test files I accidentally created:

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.