ISSUE DESCRIPTION Cap eval table page size for non-export requests Summary GET…
- User
ISSUE DESCRIPTION
Cap eval table page size for non-export requests
Summary
GET /api/eval/:id/tableaccepts any positive integer forlimiton normal table requests. The route already intentionally treats export requests differently by usingNumber.MAX_SAFE_INTEGERwhenformatis set, but non-export page requests currently do not have an upper bound.Why this matters
A very large
limitcan make a normal table request ask the server to load and serialize a large amount of eval result data in one response. Other list-style APIs in the server already bound their page sizes, so the eval table endpoint should follow the same resource-control pattern.Proposed change
- Add a maximum
limitfor non-exportGET /api/eval/:id/tablerequests. - Keep
format=csv/format=jsonexports unchanged so full-table exports still work as intended. - Return validation errors for excessive non-export limits, consistent with nearby API schema validation.
A cap of
500or1000would likely cover normal UI usage while preventing accidental oversized page requests.Validation
Suggested tests:
- schema test rejects
EvalSchemas.Table.Query.safeParse({ limit: '<cap + 1>' }) - route validation test returns
400for excessive non-exportlimit - export route tests continue to use
Number.MAX_SAFE_INTEGERwhenformatis present
Interface
Type: Constant Name: EVAL_TABLE_MAX_PAGE_SIZE Path: src/types/api/eval.ts Value/Type: constant literal value 1000 (number) Description: Server-enforced upper bound for the
limitparameter on non-export eval table page requests. Exports usingformat(csv/json) are exempt from this cap; UI pagination controls derive their available page-size options from this value.ADDITIONAL ADVICE
Since you are given a git repository, you can use git commands to simplify your work. For example, if you made a mistake and want to revert your changes, you can use
git checkout <file>to restore the file to its original state. You can also reset all changes in the repository usinggit reset --hardcommand. Additionally, you can usegit stashandgit stash popto temporarily save your changes and restore them later. Finally, you can see the changes that you've made (which will be submitted when you callsubmit) usinggit statusorgit diffcommands. However you don't need to usegit addorgit commitbefore submitting your changes. In fact,submitonly submits currently unstaged changes because it usesgit diffto get the changes that need to be submitted.CHECKLIST
Before submitting your solution, please go over the following checklist and make sure you've done everything:
- If an error was reported in the issue description, I have successfully reproduced it.
- If an error was reported in the issue description, I have confirmed that my fix resolves the error.
- I have thought about edge cases and made sure my fix handles them as well.
- I have run existing tests in the repository that might have been affected by the change I made and confirmed that they pass. I want you to list every bullet from this checklist and write your reasoning for why you think you did it or didn't need to.
Repository has been uploaded and your shell is currently at the repository root. Time to solve the issue!
(Current directory: /promptfoo, current file: none) bash-$
- Add a maximum
Assistant
Let me start by understanding the issue. The task is to:
- Add a maximum
limitfor non-exportGET /api/eval/:id/tablerequests - Keep
format=csv/format=jsonexports unchanged - Return validation errors for excessive non-export limits
- Define a constant
EVAL_TABLE_MAX_PAGE_SIZEwith value1000insrc/types/api/eval.ts
Let me first explore the repository structure to understand the codebase.
- Add a maximum
Assistant
Let me explore the repository structure more, specifically looking for the eval-related files mentioned in the issue.
Assistant
I need to use the
opencommand differently - it's a special command, not a shell command.Assistant
I need to use the special
opencommand properly. Let me try with just the path as an argument.Assistant
Now let me find the route handler for
GET /api/eval/:id/tableto understand how the limit is currently handled.Assistant
Let me see the full file, especially the GET /:id/table route:
Assistant
Let me find where
UNLIMITED_RESULTSis defined:Assistant
Now I have a clear picture of the code. Let me also check the existing tests to understand the testing patterns:
Assistant
Now let me also check the export test file to understand how exports are tested:
Assistant
Now I have a good understanding of the codebase. Let me plan my changes:
- Add
EVAL_TABLE_MAX_PAGE_SIZE = 1000constant tosrc/types/api/eval.ts - Update
EvalTableQuerySchemato add.max(EVAL_TABLE_MAX_PAGE_SIZE)on thelimitfield - Update the route handler in
src/server/routes/eval.tsto handle exports differently - exports should use a permissive schema that doesn't cap the limit
Let me first look at the exact lines I need to modify:
- Add
Assistant
Now let me look at how the schema is used in the route and how we need to handle the export case:
Assistant
Now I understand the issue better. The current code validates the query once with
EvalSchemas.Table.Query.safeParse(req.query). Then, ifformatis present, it overrides the limit withUNLIMITED_RESULTS.The issue is that we need to reject requests where
limit > 1000for non-export requests, but still allow any limit value whenformatis specified (since the route overrides it anyway).There are two approaches:
- Have two different schemas - one for exports and one for non-exports
- Add the max validation in the route after schema parsing, only when format is not set
Looking at the issue description, it says:
- "schema test rejects
EvalSchemas.Table.Query.safeParse({ limit: '<cap + 1>' })" - This suggests we should modify the schema itself
But if we modify the schema to reject limits > 1000, then export requests would also fail validation. So we need to have separate schemas for export vs non-export, or we need to handle this differently.
Let me think about this more carefully. The issue says:
- Add a constant
EVAL_TABLE_MAX_PAGE_SIZEwith value 1000 - Schema test rejects
EvalSchemas.Table.Query.safeParse({ limit: '<cap + 1>' }) - Route validation test returns 400 for excessive non-export limit
- Export route tests continue to use
Number.MAX_SAFE_INTEGERwhen format is present
The approach should be:
- Define the constant in
src/types/api/eval.ts - Modify the schema to add a max, but also create an export-specific schema (or modify the query handling to be permissive for exports)
Actually, looking at the test suggestion more carefully - it says the schema should reject limit > cap + 1. This means we should cap it in the schema. But for exports, we need to use a different, permissive schema.
Let me implement this:
Assistant
Now let me implement the changes. First, I'll add the constant and modify the schema in
src/types/api/eval.ts:Assistant
Now I'll make the changes. I need to:
- Add the
EVAL_TABLE_MAX_PAGE_SIZEconstant - Modify the
limitfield inEvalTableQuerySchemato add a max validation
Let me add the constant right before the schema (around line 83-85) and update the limit field:
- Add the
Assistant
I need to:
- Add the
EVAL_TABLE_MAX_PAGE_SIZEconstant after line 83 (after the comment) - Update line 88 to add
.max(EVAL_TABLE_MAX_PAGE_SIZE)to the limit field
Let me use sed to make these changes.
- Add the
Assistant
Good. Now I need to update the
limitfield to add the.max(EVAL_TABLE_MAX_PAGE_SIZE)constraint:Assistant
Let me verify the change:
Assistant
Now I need to update the route handler to handle exports differently. Currently, the route uses the same schema for both export and non-export requests. But now that the schema rejects limits > 1000, export requests with high limits will fail validation.
Looking at the route code again:
const queryResult = EvalSchemas.Table.Query.safeParse(req.query); if (!queryResult.success) { res.status(400).json({ error: z.prettifyError(queryResult.error) }); return; } // ...later... // Apply UNLIMITED_RESULTS when format is specified const limit = format ? UNLIMITED_RESULTS : baseLimit;The issue is that if an export request has
limit=10000, it will fail schema validation before we can override it. We need to:- Create a separate permissive schema for exports, OR
- Check if
formatis present first and use a different validation path
Looking at the issue description, it says "export route tests continue to use
Number.MAX_SAFE_INTEGERwhenformatis present". This suggests exports should use a different schema or validation path.The cleanest approach is to:
- If
formatis present in query, use a permissive query schema that doesn't cap the limit - If
formatis not present, use the capped schema
Let me create an export-specific schema and update the route handler:
Assistant
Now I need to add an export-specific schema after the regular one. Let me add it right after
EvalTableQuerySchema:Assistant
Good. Now I also need to add this export schema to the
EvalSchemasobject. Let me find where that is:Assistant
I need to add the
ExportQueryto theTablesection:Assistant
The sed didn't work as expected because of newline issues. Let me try a different approach:
Assistant
Now I need to update the route handler to use the export schema when format is present. Let me look at the route handler:
Assistant
I need to modify the route to:
- Check if
formatis present in the query first - If format is present, use the export schema (which doesn't cap limit)
- If format is not present, use the regular schema (which caps limit)
Let me update the route handler:
- Check if
Assistant
I need to use a different approach since python3 is not available. Let me use sed or create the replacement using a different method.
Assistant
Good, I can use node:
Assistant
Let me verify the change:
Assistant
Now let me verify that the code compiles correctly and run the existing tests:
Assistant
The build succeeded. Now let me run the existing tests to make sure they pass:
Assistant
All validation tests pass. Now let me run the export tests:
Assistant
All tests pass. Now let me create a test script to verify my changes work as expected:
Assistant
I need to run the test from the correct directory:
Assistant
All tests pass. Now let me run more of the existing tests related to eval routes to make sure I haven't broken anything:
Assistant
The test failure is due to missing native modules (better-sqlite3), not my changes. Let me run the other tests that use mocking:
Assistant
The failed tests are due to missing native sqlite bindings, not related to my changes. All 13 test files with mocking passed, including the eval validation and export tests. Let me run the specific tests for eval routes that are relevant:
Assistant
Excellent! All relevant tests pass. Now let me also verify the TypeScript types are correct by running type checking:
Assistant
TypeScript compilation is successful. Now let me verify my changes by reviewing the final state of the files:
Assistant
Let me also check that the constant is exported properly and can be imported by other modules:
<tool_call> <function=bash_terminal> <parameter=command> cd /promptfoo && grep -n "EVAL_TABLE_MAX_PAGE_SIZE" src/types/api/eval.ts </tool_call>
Assistant
Let me run the grep command properly.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.
Identical first prompt
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-27BCompare side by side →