ISSUE DESCRIPTION Cap eval table page size for non-export requests Summary GET…
- User
ISSUE DESCRIPTION
Cap eval table page size for non-export requests
Summary
GET /api/eval/:id/tableaccepts any positive integer forlimiton normal table requests. The route already intentionally treats export requests differently by usingNumber.MAX_SAFE_INTEGERwhenformatis set, but non-export page requests currently do not have an upper bound.Why this matters
A very large
limitcan make a normal table request ask the server to load and serialize a large amount of eval result data in one response. Other list-style APIs in the server already bound their page sizes, so the eval table endpoint should follow the same resource-control pattern.Proposed change
- Add a maximum
limitfor non-exportGET /api/eval/:id/tablerequests. - Keep
format=csv/format=jsonexports unchanged so full-table exports still work as intended. - Return validation errors for excessive non-export limits, consistent with nearby API schema validation.
A cap of
500or1000would likely cover normal UI usage while preventing accidental oversized page requests.Validation
Suggested tests:
- schema test rejects
EvalSchemas.Table.Query.safeParse({ limit: '<cap + 1>' }) - route validation test returns
400for excessive non-exportlimit - export route tests continue to use
Number.MAX_SAFE_INTEGERwhenformatis present
Interface
Type: Constant Name: EVAL_TABLE_MAX_PAGE_SIZE Path: src/types/api/eval.ts Value/Type: constant literal value 1000 (number) Description: Server-enforced upper bound for the
limitparameter on non-export eval table page requests. Exports usingformat(csv/json) are exempt from this cap; UI pagination controls derive their available page-size options from this value.ADDITIONAL ADVICE
Since you are given a git repository, you can use git commands to simplify your work. For example, if you made a mistake and want to revert your changes, you can use
git checkout <file>to restore the file to its original state. You can also reset all changes in the repository usinggit reset --hardcommand. Additionally, you can usegit stashandgit stash popto temporarily save your changes and restore them later. Finally, you can see the changes that you've made (which will be submitted when you callsubmit) usinggit statusorgit diffcommands. However you don't need to usegit addorgit commitbefore submitting your changes. In fact,submitonly submits currently unstaged changes because it usesgit diffto get the changes that need to be submitted.CHECKLIST
Before submitting your solution, please go over the following checklist and make sure you've done everything:
- If an error was reported in the issue description, I have successfully reproduced it.
- If an error was reported in the issue description, I have confirmed that my fix resolves the error.
- I have thought about edge cases and made sure my fix handles them as well.
- I have run existing tests in the repository that might have been affected by the change I made and confirmed that they pass. I want you to list every bullet from this checklist and write your reasoning for why you think you did it or didn't need to.
Repository has been uploaded and your shell is currently at the repository root. Time to solve the issue!
(Current directory: /promptfoo, current file: none) bash-$
- Add a maximum
Assistant
I'll start by exploring the repository structure to understand the codebase and find the relevant files mentioned in the issue.
Assistant
I'll start by exploring the repository structure to understand the codebase.
Assistant
Let me look at the eval.ts types file and find the relevant schema.
Assistant
Now let me find the route handler that uses this schema and handles the export logic.
Assistant
Let me look at the route handler in detail.
Assistant
Let me look at the existing tests for the eval table schema and route to understand patterns.
Assistant
Let me search more broadly for tests related to the eval schema and route.
Assistant
Let me look at how the schemas are tested. Let me search for test directories that test the schemas (server-side validation).
Assistant
No server tests exist. Let me look at where the schema validation tests might be. Let me search the whole repo.
Assistant
Let me look at how other list-style APIs bound their page sizes, to follow the existing pattern.
Assistant
Let me check the Zod version and how refinements/issues are used in the codebase.
Assistant
Let me look at the existing superRefine pattern in
src/contracts/shared.tsto follow the convention for addIssue.Assistant
Now let me check the existing tests that may reference the limit or page sizes to ensure my change doesn't break them.
Assistant
Let me check how the frontend makes the table request and what page sizes it uses, to ensure my change won't break the UI.
Assistant
Let me look at the ResultsTable pagination page-size selector options to understand what page-size options the UI offers.
Assistant
The UI page-size options are 10, 50, 100, 500, 1000 — so a cap of 1000 matches the max UI option exactly. Now let me look at the OpenAPI
queryhelper to make sure asuperRefine-wrapped schema will work for OpenAPI generation.Assistant
Let me check whether there's a committed OpenAPI spec and a check test that could be affected.
Assistant
Now I understand the structure. Let me reproduce/verify the current behavior first with a quick script before making changes.
Assistant
Confirmed the bug:
limit=1001with no format currently succeeds. Now let me implement the fix by editingsrc/types/api/eval.ts.Assistant
Now let me open the file and make the edit:
Assistant
Now I'll make the edit to add the constant and the conditional cap:
Assistant
Now let me check whether
EVAL_TABLE_MAX_PAGE_SIZEshould be re-exported from the types barrel, and verify the route still works. Let me check the index exports.Assistant
The constant is exported at the top level of the file. Now let me verify the fix with a reproduction script covering all the test scenarios.
Assistant
The
!triggered bash history expansion. Let me write the test to a file to avoid shell quoting issues.Assistant
All scenarios pass. Now let me verify the OpenAPI spec generation still works (since the schema is registered there) and run a typecheck.
Assistant
The spec file isn't at the default path. Let me find where the committed OpenAPI spec is.
Assistant
There's no committed spec, so
openapi:checkisn't part of CI here. Let me confirm the spec still generates without error (to a temp path) and inspect the EvalTableQuery portion.Assistant
OpenAPI generation works correctly with the
superRefinewrapper. Now let me confirm the route still usesNumber.MAX_SAFE_INTEGERfor exports and returns 400 on schema failure (already in place). Let me view the route section.Assistant
The route already returns 400 on schema failure and uses
MAX_SAFE_INTEGERfor exports — no route changes needed. Now let me run the existing tests that could be affected. Let me check the test setup.Assistant
The root vitest config only covers
test/**. The app tests run viatest:app. Let me run the relevant app tests.Assistant
All app tests pass. Let me check the
test/directory for any eval table route/schema tests that might be affected.Assistant
There are dedicated tests. Let me look at the most relevant ones: the schema test, validation test, and export test.
Assistant
Now let me look at the validation and export route tests to understand expectations.
Assistant
Let me look at the export route test and the routes-eval test to ensure my change doesn't break existing export behavior.
Assistant
Let me see the full export test, especially around line 350-400, to check for any limit exceeding the cap with format.
Assistant
Let me run all the relevant test files to confirm my change doesn't break anything.
Assistant
The one failure is an environment issue —
better-sqlite3native bindings aren't compiled for this Node version (node-v137), unrelated to my change. Let me confirm it fails identically without my changes by stashing.Assistant
Confirmed — the
eval.filteredMetrics.test.tsfailure is a pre-existing environment issue (better-sqlite3 native binding for node-v137), identical with or without my change. Let me try to rebuild better-sqlite3 so I can run the DB-backed route tests too.Assistant
Rebuild succeeded. Let me re-run the DB-backed route tests.
Assistant
All 68 tests pass. Now let me write a temporary test to verify the route returns 400 for an excessive non-export limit (matching the issue's suggested route test), then remove it. Let me check the vitest config include pattern first.
Assistant
I'll create a temporary test under
test/(to be removed before submitting) to verify the route behavior for excessive non-export limits.Assistant
globals: falserequires importingbeforeEach. Let me fix the import.Assistant
All three route-level scenarios pass. Now let me remove the temporary test file.
Assistant
Only
src/types/api/eval.tsis modified. Let me review the final diff.Assistant
The diff looks clean. Let me run the repo's linter (biome) on the changed file to ensure formatting compliance.
Assistant
Biome passes. Let me search the test suite for any test that passes a large
limitto the table endpoint without aformat, which could now break.Assistant
The smoke test uses
limit=abc(NaN → 400), unaffected by my change. Let me run the smoke route test and the evalTableUtils test to be thorough.Assistant
All 201 tests pass. Let me verify the constant is accessible via the
@promptfoo/types/api/evalpackage path (as the app imports types).Assistant
Let me check how the app resolves
@promptfoo/types/api/evalto confirm the constant would be importable from the app.Assistant
The app resolves
@promptfootosrc/, so the constant is accessible. The UI already caps page-size options at 1000, matching the constant. Let me verify the prettified error message includeslimit(for consistency with nearby validation that checkserror.toContain(field)).Assistant
Let me place the temp script in the repo root so module resolution works.
Assistant
The error message is clear and includes
limit. The change is complete and verified. Let me do a final review of the changes before submitting.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.
Identical first prompt
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-27BCompare side by side →
Qwen/Qwen3.6-27BCompare side by side →