ISSUE DESCRIPTION Cap eval table page size for non-export requests Summary GET…
- User
ISSUE DESCRIPTION
Cap eval table page size for non-export requests
Summary
GET /api/eval/:id/tableaccepts any positive integer forlimiton normal table requests. The route already intentionally treats export requests differently by usingNumber.MAX_SAFE_INTEGERwhenformatis set, but non-export page requests currently do not have an upper bound.Why this matters
A very large
limitcan make a normal table request ask the server to load and serialize a large amount of eval result data in one response. Other list-style APIs in the server already bound their page sizes, so the eval table endpoint should follow the same resource-control pattern.Proposed change
- Add a maximum
limitfor non-exportGET /api/eval/:id/tablerequests. - Keep
format=csv/format=jsonexports unchanged so full-table exports still work as intended. - Return validation errors for excessive non-export limits, consistent with nearby API schema validation.
A cap of
500or1000would likely cover normal UI usage while preventing accidental oversized page requests.Validation
Suggested tests:
- schema test rejects
EvalSchemas.Table.Query.safeParse({ limit: '<cap + 1>' }) - route validation test returns
400for excessive non-exportlimit - export route tests continue to use
Number.MAX_SAFE_INTEGERwhenformatis present
Interface
Type: Constant Name: EVAL_TABLE_MAX_PAGE_SIZE Path: src/types/api/eval.ts Value/Type: constant literal value 1000 (number) Description: Server-enforced upper bound for the
limitparameter on non-export eval table page requests. Exports usingformat(csv/json) are exempt from this cap; UI pagination controls derive their available page-size options from this value.ADDITIONAL ADVICE
Since you are given a git repository, you can use git commands to simplify your work. For example, if you made a mistake and want to revert your changes, you can use
git checkout <file>to restore the file to its original state. You can also reset all changes in the repository usinggit reset --hardcommand. Additionally, you can usegit stashandgit stash popto temporarily save your changes and restore them later. Finally, you can see the changes that you've made (which will be submitted when you callsubmit) usinggit statusorgit diffcommands. However you don't need to usegit addorgit commitbefore submitting your changes. In fact,submitonly submits currently unstaged changes because it usesgit diffto get the changes that need to be submitted.CHECKLIST
Before submitting your solution, please go over the following checklist and make sure you've done everything:
- If an error was reported in the issue description, I have successfully reproduced it.
- If an error was reported in the issue description, I have confirmed that my fix resolves the error.
- I have thought about edge cases and made sure my fix handles them as well.
- I have run existing tests in the repository that might have been affected by the change I made and confirmed that they pass. I want you to list every bullet from this checklist and write your reasoning for why you think you did it or didn't need to.
Repository has been uploaded and your shell is currently at the repository root. Time to solve the issue!
(Current directory: /promptfoo, current file: none) bash-$
- Add a maximum
Assistant
Now let's check for UI code that might use page size options, to see if there's a place referencing this constant.
Assistant
Now let's implement the change with
.superRefineto add proper error path, and export the constant.Assistant
Now let's check the openapi/server.ts reference for EvalTableQuerySchema to make sure the change doesn't break openapi generation (superRefine schemas sometimes not compatible with json schema conversion tools).
Assistant
Let's build/typecheck and run existing tests to verify nothing broke.
Assistant
No type errors. Now run the openapi generation test and the eval-related tests.
Assistant
It's vitest, not jest. Let's use vitest instead.
Assistant
Confirmed pre-existing environment issue unrelated to our change. Now let's update the route to enforce the max in the actual route logic too (redundant safety net, since schema validation already handles it), but let's check the route file for consistency and also review the swagger/openapi docs description mentioning limit max.
Assistant
That's fine, no changes needed there. Let's verify the openapi generation works with the superRefine schema (registerZodSchema might use zod-to-openapi which sometimes doesn't handle superRefine well for
query).Assistant
Good, openapi generation passes. Now let's write a quick reproduction script to verify the fix behavior end-to-end (schema level).
Assistant
This confirms the fix works as expected. Let's clean up the temp script.
Assistant
Now let's double check the route file to see if there's anything else referencing limit that we should adjust (e.g., a comment mentioning the cap, or docs).
Assistant
This already handles validation via
queryResult.successreturning 400 correctly for over-cap non-export requests, since our schema now rejects those. No further route changes needed.Let's also check the CLAUDE.md/docs or CHANGELOG conventions for adding entries, though not required. Let's view git diff for final review.
Assistant
This looks good and minimal. Let's run the broader test suite related to types/api to ensure nothing else references EvalTableQuerySchema in ways that could break.
Assistant
Only used internally within eval.ts (exported as
EvalSchemas.Table.Query). Good. Let's re-run the full relevant test set once more to be safe (excluding the pre-existing broken sqlite test file).Assistant
All relevant tests pass. Now let's review the checklist items.
Assistant
Checklist review:
- Reproduced the reported error: The issue describes a missing upper bound rather than an existing runtime error/exception. I confirmed the gap by testing that
EvalSchemas.Table.Query.safeParse({ limit: '1001' })succeeded before my change (implicitly, since no cap existed) and now correctly fails, matching the suggested validation test in the issue. - Confirmed fix resolves it: Verified via a script that non-export requests with
limitaboveEVAL_TABLE_MAX_PAGE_SIZE(1000) now fail validation (success: false), requests at/below the cap succeed, and export requests (format: 'csv'/'json') remain exempt from the cap (stillsuccess: trueeven above the cap), matching the required behavior for exports. - Edge cases considered: Limit exactly at the cap is allowed (
<=not<); export requests bypass the cap entirely regardless of limit value; default limit (50) and other existing validations (positive,int) are unaffected; the new constant is exported so it can be reused (e.g., by future UI code) per the interface spec. - Ran existing tests: Ran
test/types/apiSchemas.redteam.test.ts,test/server/routes/eval.validation.test.ts,test/server/routes/eval.export.test.ts,test/server/routes-eval.test.ts,test/server/routes/eval.sharing.test.ts, andtest/openapi/serverOpenApi.test.ts— all 80 tests pass.test/server/routes/eval.filteredMetrics.test.tsfails due to a pre-existing environment issue (better-sqlite3native bindings not built), unrelated to this change (confirmed identical failure on the unmodified codebase viagit stash). Also confirmedtsc --noEmitpasses with no type errors.
- Reproduced the reported error: The issue describes a missing upper bound rather than an existing runtime error/exception. I confirmed the gap by testing that
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.
Identical first prompt
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-27BCompare side by side →
Qwen/Qwen3.6-27BCompare side by side →