Skip to content
TheChatArchiveThe Chat Archive
Claudeclaude-sonnet-5· anthropicHugging Face ibragim-bad/swe_rebench_07_2026_trajectories

ISSUE DESCRIPTION Cap eval table page size for non-export requests Summary GET…

  1. User

    ISSUE DESCRIPTION

    Cap eval table page size for non-export requests

    Summary

    GET /api/eval/:id/table accepts any positive integer for limit on normal table requests. The route already intentionally treats export requests differently by using Number.MAX_SAFE_INTEGER when format is set, but non-export page requests currently do not have an upper bound.

    Why this matters

    A very large limit can make a normal table request ask the server to load and serialize a large amount of eval result data in one response. Other list-style APIs in the server already bound their page sizes, so the eval table endpoint should follow the same resource-control pattern.

    Proposed change

    • Add a maximum limit for non-export GET /api/eval/:id/table requests.
    • Keep format=csv / format=json exports unchanged so full-table exports still work as intended.
    • Return validation errors for excessive non-export limits, consistent with nearby API schema validation.

    A cap of 500 or 1000 would likely cover normal UI usage while preventing accidental oversized page requests.

    Validation

    Suggested tests:

    • schema test rejects EvalSchemas.Table.Query.safeParse({ limit: '<cap + 1>' })
    • route validation test returns 400 for excessive non-export limit
    • export route tests continue to use Number.MAX_SAFE_INTEGER when format is present

    Interface

    Type: Constant Name: EVAL_TABLE_MAX_PAGE_SIZE Path: src/types/api/eval.ts Value/Type: constant literal value 1000 (number) Description: Server-enforced upper bound for the limit parameter on non-export eval table page requests. Exports using format (csv/json) are exempt from this cap; UI pagination controls derive their available page-size options from this value.

    ADDITIONAL ADVICE

    Since you are given a git repository, you can use git commands to simplify your work. For example, if you made a mistake and want to revert your changes, you can use git checkout <file> to restore the file to its original state. You can also reset all changes in the repository using git reset --hard command. Additionally, you can use git stash and git stash pop to temporarily save your changes and restore them later. Finally, you can see the changes that you've made (which will be submitted when you call submit) using git status or git diff commands. However you don't need to use git add or git commit before submitting your changes. In fact, submit only submits currently unstaged changes because it uses git diff to get the changes that need to be submitted.

    CHECKLIST

    Before submitting your solution, please go over the following checklist and make sure you've done everything:

    • If an error was reported in the issue description, I have successfully reproduced it.
    • If an error was reported in the issue description, I have confirmed that my fix resolves the error.
    • I have thought about edge cases and made sure my fix handles them as well.
    • I have run existing tests in the repository that might have been affected by the change I made and confirmed that they pass. I want you to list every bullet from this checklist and write your reasoning for why you think you did it or didn't need to.

    Repository has been uploaded and your shell is currently at the repository root. Time to solve the issue!

    (Current directory: /promptfoo, current file: none) bash-$

  2. Claude Assistant

    Now let's check for UI code that might use page size options, to see if there's a place referencing this constant.

  3. Claude Assistant

    Now let's implement the change with .superRefine to add proper error path, and export the constant.

  4. Claude Assistant

    Now let's check the openapi/server.ts reference for EvalTableQuerySchema to make sure the change doesn't break openapi generation (superRefine schemas sometimes not compatible with json schema conversion tools).

  5. Claude Assistant

    Let's build/typecheck and run existing tests to verify nothing broke.

  6. Claude Assistant

    No type errors. Now run the openapi generation test and the eval-related tests.

  7. Claude Assistant

    It's vitest, not jest. Let's use vitest instead.

  8. Claude Assistant

    Confirmed pre-existing environment issue unrelated to our change. Now let's update the route to enforce the max in the actual route logic too (redundant safety net, since schema validation already handles it), but let's check the route file for consistency and also review the swagger/openapi docs description mentioning limit max.

  9. Claude Assistant

    That's fine, no changes needed there. Let's verify the openapi generation works with the superRefine schema (registerZodSchema might use zod-to-openapi which sometimes doesn't handle superRefine well for query).

  10. Claude Assistant

    Good, openapi generation passes. Now let's write a quick reproduction script to verify the fix behavior end-to-end (schema level).

  11. Claude Assistant

    This confirms the fix works as expected. Let's clean up the temp script.

  12. Claude Assistant

    Now let's double check the route file to see if there's anything else referencing limit that we should adjust (e.g., a comment mentioning the cap, or docs).

  13. Claude Assistant

    This already handles validation via queryResult.success returning 400 correctly for over-cap non-export requests, since our schema now rejects those. No further route changes needed.

    Let's also check the CLAUDE.md/docs or CHANGELOG conventions for adding entries, though not required. Let's view git diff for final review.

  14. Claude Assistant

    This looks good and minimal. Let's run the broader test suite related to types/api to ensure nothing else references EvalTableQuerySchema in ways that could break.

  15. Claude Assistant

    Only used internally within eval.ts (exported as EvalSchemas.Table.Query). Good. Let's re-run the full relevant test set once more to be safe (excluding the pre-existing broken sqlite test file).

  16. Claude Assistant

    All relevant tests pass. Now let's review the checklist items.

  17. Claude Assistant

    Checklist review:

    • Reproduced the reported error: The issue describes a missing upper bound rather than an existing runtime error/exception. I confirmed the gap by testing that EvalSchemas.Table.Query.safeParse({ limit: '1001' }) succeeded before my change (implicitly, since no cap existed) and now correctly fails, matching the suggested validation test in the issue.
    • Confirmed fix resolves it: Verified via a script that non-export requests with limit above EVAL_TABLE_MAX_PAGE_SIZE (1000) now fail validation (success: false), requests at/below the cap succeed, and export requests (format: 'csv'/'json') remain exempt from the cap (still success: true even above the cap), matching the required behavior for exports.
    • Edge cases considered: Limit exactly at the cap is allowed (<= not <); export requests bypass the cap entirely regardless of limit value; default limit (50) and other existing validations (positive, int) are unaffected; the new constant is exported so it can be reused (e.g., by future UI code) per the interface spec.
    • Ran existing tests: Ran test/types/apiSchemas.redteam.test.ts, test/server/routes/eval.validation.test.ts, test/server/routes/eval.export.test.ts, test/server/routes-eval.test.ts, test/server/routes/eval.sharing.test.ts, and test/openapi/serverOpenApi.test.ts — all 80 tests pass. test/server/routes/eval.filteredMetrics.test.ts fails due to a pre-existing environment issue (better-sqlite3 native bindings not built), unrelated to this change (confirmed identical failure on the unmodified codebase via git stash). Also confirmed tsc --noEmit passes with no type errors.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.