ISSUE DESCRIPTION Cap eval table page size for non-export requests Summary GET…
- User
ISSUE DESCRIPTION
Cap eval table page size for non-export requests
Summary
GET /api/eval/:id/tableaccepts any positive integer forlimiton normal table requests. The route already intentionally treats export requests differently by usingNumber.MAX_SAFE_INTEGERwhenformatis set, but non-export page requests currently do not have an upper bound.Why this matters
A very large
limitcan make a normal table request ask the server to load and serialize a large amount of eval result data in one response. Other list-style APIs in the server already bound their page sizes, so the eval table endpoint should follow the same resource-control pattern.Proposed change
- Add a maximum
limitfor non-exportGET /api/eval/:id/tablerequests. - Keep
format=csv/format=jsonexports unchanged so full-table exports still work as intended. - Return validation errors for excessive non-export limits, consistent with nearby API schema validation.
A cap of
500or1000would likely cover normal UI usage while preventing accidental oversized page requests.Validation
Suggested tests:
- schema test rejects
EvalSchemas.Table.Query.safeParse({ limit: '<cap + 1>' }) - route validation test returns
400for excessive non-exportlimit - export route tests continue to use
Number.MAX_SAFE_INTEGERwhenformatis present
Interface
Type: Constant Name: EVAL_TABLE_MAX_PAGE_SIZE Path: src/types/api/eval.ts Value/Type: constant literal value 1000 (number) Description: Server-enforced upper bound for the
limitparameter on non-export eval table page requests. Exports usingformat(csv/json) are exempt from this cap; UI pagination controls derive their available page-size options from this value.ADDITIONAL ADVICE
Since you are given a git repository, you can use git commands to simplify your work. For example, if you made a mistake and want to revert your changes, you can use
git checkout <file>to restore the file to its original state. You can also reset all changes in the repository usinggit reset --hardcommand. Additionally, you can usegit stashandgit stash popto temporarily save your changes and restore them later. Finally, you can see the changes that you've made (which will be submitted when you callsubmit) usinggit statusorgit diffcommands. However you don't need to usegit addorgit commitbefore submitting your changes. In fact,submitonly submits currently unstaged changes because it usesgit diffto get the changes that need to be submitted.CHECKLIST
Before submitting your solution, please go over the following checklist and make sure you've done everything:
- If an error was reported in the issue description, I have successfully reproduced it.
- If an error was reported in the issue description, I have confirmed that my fix resolves the error.
- I have thought about edge cases and made sure my fix handles them as well.
- I have run existing tests in the repository that might have been affected by the change I made and confirmed that they pass. I want you to list every bullet from this checklist and write your reasoning for why you think you did it or didn't need to.
Repository has been uploaded and your shell is currently at the repository root. Time to solve the issue!
(Current directory: /promptfoo, current file: none) bash-$
- Add a maximum
Assistant
I'll start by exploring the repository structure to understand the codebase and find the relevant files for implementing this fix.
Assistant
Let me explore the repository structure more systematically, especially looking for the
src/types/api/eval.tsfile mentioned in the issue.Assistant
I found the file mentioned in the issue. Let me examine its contents.
Assistant
Now I need to find the route handler for
GET /api/eval/:id/table. Let me search for it.Assistant
Let me search for the route handler more broadly by looking for files that might contain the actual API implementation.
Assistant
Let me search for files containing "/table" endpoint routes.
Assistant
I found the route file. Let me examine the eval routes file.
<tool_call> <function=bash_terminal> <parameter=command> cat /promptfoo/src/server/routes/eval.ts </tool_call>
Assistant
Let me read the eval route file using bash terminal.
Assistant
Let me continue reading the file to find the table endpoint handler. The output was truncated.
<tool_call> <function=bash_terminal> <parameter=command> grep -n "GET.*:id/table" /promptfoo/src/server/routes/eval.ts </tool_call>
Assistant
Let me search for the table endpoint handler in the eval.ts route file.
Assistant
I see that the table endpoint is at line 311. Let me look at that section more carefully.
Assistant
Now I can see the issue. At line 327, the route handler uses
UNLIMITED_RESULTSwhenformatis specified:const limit = format ? UNLIMITED_RESULTS : baseLimit;But I need to find where
UNLIMITED_RESULTSis defined and also where I need to add the cap for non-export requests. Let me search for it.<tool_call> <function=bash_terminal> <parameter=command> grep -n "UNLIMITED_RESULTS" /promptfoo/src/server/routes/eval.ts </tool_call>
Assistant
Let me search for where UNLIMITED_RESULTS is defined.
Assistant
Now I understand the issue. The route already uses
UNLIMITED_RESULTS(which isNumber.MAX_SAFE_INTEGER) for export requests (whenformatis set), but doesn't cap the limit for non-export requests.Let me now:
- Add the
EVAL_TABLE_MAX_PAGE_SIZEconstant to the eval.ts types file - Update the route handler to use this cap for non-export requests
- Find and run existing tests to make sure nothing breaks
First, let me look at the test files to understand the test structure.
<tool_call> <function=bash_terminal> <parameter=command> find /promptfoo -name "*.test.ts" -path "routes" | head -10 </tool_call>
- Add the
Assistant
Let me search for test files related to eval routes.
Assistant
I found relevant test files. Let me check the validation test and export test to understand the current test structure.
<tool_call> <function=bash_terminal> <parameter=command> cat /promptfoo/test/server/routes/eval.validation.test.ts </tool_call>
Assistant
I need to use bash_terminal to read the file.
Assistant
Now let me also check the export test to understand how export requests are tested.
<tool_call> <function=bash_terminal> <parameter=command> cat /promptfoo/test/server/routes/eval.export.test.ts </tool_call>
Assistant
I need to use bash_terminal to read the file instead.
Assistant
Now I have a good understanding of the codebase. Let me create a plan:
- Add the
EVAL_TABLE_MAX_PAGE_SIZEconstant to/promptfoo/src/types/api/eval.ts - Update the
EvalTableQuerySchemato add max validation for the limit field (but only for non-export requests) - Update the route handler in
/promptfoo/src/server/routes/eval.tsto use the cap for non-export requests - Run existing tests to make sure nothing breaks
Let me start by looking at the current state of the schema to understand how to properly add the max constraint. The schema currently has:
limit: z.coerce.number().int().positive().prefault(50),I need to add a max constraint. However, since exports should be unlimited, the validation should happen at the schema level only for non-export cases. Let me think about this more carefully.
Actually, looking at the issue description more carefully:
- "Add a maximum
limitfor non-exportGET /api/eval/:id/tablerequests." - "Keep
format=csv/format=jsonexports unchanged so full-table exports still work as intended." - "Return validation errors for excessive non-export limits, consistent with nearby API schema validation."
So the validation should reject excessive non-export limits. The approach should be:
- Define the constant
EVAL_TABLE_MAX_PAGE_SIZE = 1000in the types file - Modify the
EvalTableQuerySchemato have max validation for the limit field - But the route handler needs to handle exports differently - when
formatis present, ignore the max validation
Wait, the issue says the validation should reject excessive limits. But exports use
formatparameter. So we can't just add a max constraint to the schema because that would also reject exports with a high limit.Looking at the code again:
const limit = format ? UNLIMITED_RESULTS : baseLimit;The route already handles the format check. So we could:
- Add the max validation to the schema with the cap value
- In the route handler, before validation, check if format is present and conditionally use a different schema or skip the limit check
Actually, a simpler approach is to:
- Add the max validation with the cap value
- Modify the schema so that when format is present, it uses a different schema (or uses a different transformation)
But looking at the existing test file
eval.validation.test.ts, it has a test for the table endpoint that tests validation errors. The issue says:- "schema test rejects
EvalSchemas.Table.Query.safeParse({ limit: '<cap + 1>' })" - "route validation test returns
400for excessive non-exportlimit"
So the validation should be done at the schema level for non-export requests. Let me think about the cleanest way to do this.
One approach is to modify the schema to conditionally apply the max constraint based on whether format is present. However, Zod doesn't easily support conditional schemas based on optional fields.
A better approach is:
- Add the constant
EVAL_TABLE_MAX_PAGE_SIZE = 1000 - Keep the schema as is (or add the max constraint but handle exports specially in the route)
- In the route handler, check if the limit is too high when format is NOT present, and return a validation error
Actually, re-reading the issue more carefully:
- "Return validation errors for excessive non-export limits, consistent with nearby API schema validation."
So we need to return validation errors. The question is where to do this validation:
- At the schema level (cleaner, uses existing error handling)
- At the route level (more control, but duplicates logic)
Looking at the existing code structure, I think the best approach is:
- Add the constant
EVAL_TABLE_MAX_PAGE_SIZE = 1000in the types file - Modify the schema to include the max constraint
- Modify the route handler to use a different schema (or modify the query) when format is present
Wait, let me look at how the route handles the format check. The route already checks
format ? UNLIMITED_RESULTS : baseLimitafter the schema validation. So the schema validation happens first, then the limit is adjusted.The cleanest approach is to:
- Add the constant
EVAL_TABLE_MAX_PAGE_SIZE = 1000 - Keep the schema with the max constraint
- In the route handler, when format is present, manually validate without the max constraint (or adjust the limit before passing to the schema)
Actually, I think the simplest approach is to add the max constraint to the schema and then handle exports differently by adjusting the limit before schema validation when format is present. But that's complex.
Let me look at the issue description again:
- "schema test rejects
EvalSchemas.Table.Query.safeParse({ limit: '<cap + 1>' })"
This suggests the schema should reject excessive limits. But exports use
formatparameter, so we need to handle this conditionally.I think the best approach is:
- Add the constant
EVAL_TABLE_MAX_PAGE_SIZE = 1000 - Modify the schema to have a max constraint on limit
- In the route handler, when format is present, temporarily remove the limit from the query or adjust it before validation
Actually, I realize that we can't easily make the schema conditional. The simplest approach is:
- Add the constant
EVAL_TABLE_MAX_PAGE_SIZE = 1000 - Add the max constraint to the schema
- In the route handler, before validation, if format is present, set limit to MAX_SAFE_INTEGER or remove it from the query
- After validation, if format is present, set limit to MAX_SAFE_INTEGER again
Wait, but the issue says "Return validation errors for excessive non-export limits". This means the validation should happen and return a 400 error for excessive limits when format is NOT present.
Let me re-think this:
- The schema should reject excessive limits for non-export requests
- For export requests, the schema should accept any positive integer
The cleanest way to do this is to modify the schema to use a conditional constraint. However, Zod's conditional schemas are complex.
A simpler approach is:
- Add the constant
EVAL_TABLE_MAX_PAGE_SIZE = 1000 - Keep the schema with a max constraint
- In the route handler, before schema validation, if format is present, set limit to a value <= cap (e.g., cap) so the schema accepts it
- After validation, set limit to MAX_SAFE_INTEGER for exports
Actually, the route already sets
const limit = format ? UNLIMITED_RESULTS : baseLimit;after validation. So we need to handle the validation differently.Let me look at the route again:
const queryResult = EvalSchemas.Table.Query.safeParse(req.query); if (!queryResult.success) { res.status(400).json({ error: z.prettifyError(queryResult.error) }); return; }The validation happens before we check if format is present. So we need to:
- Modify the schema to have conditional max constraint based on format
- Or handle the validation manually for the limit field
I think the cleanest approach is to modify the schema to accept any positive integer for limit (remove the max constraint), and then do manual validation in the route handler for non-export requests. This way, exports can have any limit value (which they'll set to MAX_SAFE_INTEGER anyway).
Wait, but the issue says "schema test rejects
EvalSchemas.Table.Query.safeParse({ limit: '<cap + 1>' })". This suggests the schema itself should reject excessive limits.Hmm, I think I need to re-read the issue more carefully. The issue says:
- "Add a maximum
limitfor non-exportGET /api/eval/:id/tablerequests." - "Keep
format=csv/format=jsonexports unchanged so full-table exports still work as intended." - "Return validation errors for excessive non-export limits, consistent with nearby API schema validation."
So the validation should reject excessive limits for non-export requests. But how do we make the schema conditional based on format?
One approach is to use Zod's
whenorrefinemethod to add a conditional constraint. Let me check if that's possible.Actually, I think the simplest approach is:
- Add the constant
EVAL_TABLE_MAX_PAGE_SIZE = 1000 - Modify the schema to accept any positive integer for limit
- In the route handler, after validation, check if limit is too high AND format is not present, and return a validation error
But this doesn't match the issue description which says "schema test rejects...". So maybe the intent is to have the schema do the validation, not the route.
Let me think about this differently. The schema has a
formatfield which is optional. We can use Zod's conditional schema to:- If
formatis present, allow any positive integer forlimit - If
formatis not present, only allowlimit<= 1000
This can be done with
z.object().refine()or using a conditional schema.Actually, a better approach is to use Zod's
whenor to create two different schemas: one for exports and one for non-exports. But that might be overkill.Let me just implement a solution using
z.object().refine()to add a conditional constraint:export const EvalTableQuerySchema = z .object({ format: z.enum(['csv', 'json']).optional(), limit: z.coerce.number().int().positive().prefault(50), offset: z.coerce.number().int().nonnegative().prefault(0), ... }) .refine( (data) => { // If format is not present, limit should be <= cap if (!data.format && data.limit > EVAL_TABLE_MAX_PAGE_SIZE) { return false; } return true; }, { message: `Limit must be ${EVAL_TABLE_MAX_PAGE_SIZE} or less for non-export requests`, path: ['limit'], } );This approach uses Zod's
refineto add a conditional constraint that checks ifformatis not present and iflimitexceeds the cap.Let me now implement this solution.
First, I'll add the constant to the types file:
// src/types/api/eval.ts export const EVAL_TABLE_MAX_PAGE_SIZE = 1000; // Then modify the EvalTableQuerySchema to use refine() for conditional validationThen, I'll update the route handler to use
EVAL_TABLE_MAX_PAGE_SIZEinstead ofUNLIMITED_RESULTSfor non-export requests, but actually, the route already usesUNLIMITED_RESULTSfor exports andbaseLimitfor non-exports. So the only change needed is to cap thebaseLimittoEVAL_TABLE_MAX_PAGE_SIZEwhen format is not present.Wait, but the schema validation will reject limits > cap anyway, so the route will never receive a limit > cap for non-export requests. The route just needs to use
baseLimitdirectly (which will be <= cap) for non-export requests.Actually, I think the route needs to use
Math.min(baseLimit, EVAL_TABLE_MAX_PAGE_SIZE)or just usebaseLimitsince the schema will validate it. But the schema should reject values > cap, sobaseLimitwill always be <= cap.Let me implement this step by step.
Actually, I just realized that the route already does:
const limit = format ? UNLIMITED_RESULTS : baseLimit;So for non-export requests, it uses
baseLimitwhich is validated by the schema. If the schema rejects values > cap, thenbaseLimitwill always be <= cap, and the route will use a reasonable value.But wait, the issue says "schema test rejects...". This means the schema should reject values > cap for non-export requests. So the implementation should:
- Add the constant
EVAL_TABLE_MAX_PAGE_SIZE = 1000 - Modify the schema to conditionally reject values > cap when format is not present
- The route can stay the same (it already uses
baseLimitfor non-export requests)
Let me now implement this.
Implementation Plan
- Add
EVAL_TABLE_MAX_PAGE_SIZE = 1000constant to/promptfoo/src/types/api/eval.ts - Modify
EvalTableQuerySchemato userefine()for conditional max validation - Run existing tests to ensure nothing breaks
Let me start with step 1.
<tool_call> <function=bash_terminal> <parameter=command> cat /promptfoo/src/types/api/eval.ts | head -30 </tool_call>
- Add the
Assistant
I need to use bash_terminal to read the file and then edit it. Let me add the constant to the file first.
Assistant
Now let me add the constant
EVAL_TABLE_MAX_PAGE_SIZE = 1000to the file. I'll add it right after the imports and before the first schema definition.Assistant
Good, the constant was added. Let me verify it was added correctly.
Assistant
Now I need to find the
EvalTableQuerySchemaand modify it to add conditional validation. Let me locate it.Assistant
Let me see the current schema definition.
Assistant
Now I need to modify the schema to add conditional validation using
refine(). I'll add a refinement that checks ifformatis not present and iflimitexceeds the cap, it should fail validation.I need to add the import for
EVAL_TABLE_MAX_PAGE_SIZEsince I'm using it in the refine function. Actually, since both are in the same file, I can just reference it directly.Let me modify the schema:
export const EvalTableQuerySchema = z .object({ format: z.enum(['csv', 'json']).optional(), limit: z.coerce.number().int().positive().prefault(50), offset: z.coerce.number().int().nonnegative().prefault(0), filterMode: EvalResultsFilterMode.prefault('all'), search: z.string().prefault(''), filter: z .union([z.string(), z.array(z.string())]) .transform((v) => (Array.isArray(v) ? v : v ? [v] : [])) .prefault([]), comparisonEvalIds: z .union([z.string(), z.array(z.string())]) .transform((v) => (Array.isArray(v) ? v : v ? [v] : [])) .prefault([]), }) .refine( (data) => { // If format is not present (non-export request), limit should be <= cap if (!data.format && data.limit > EVAL_TABLE_MAX_PAGE_SIZE) { return false; } return true; }, { message: `Limit must be ${EVAL_TABLE_MAX_PAGE_SIZE} or less for non-export requests`, path: ['limit'], } );Let me apply this change using sed or by editing the file.
<tool_call> <function=bash_terminal> <parameter=command> sed -n '90,105p' /promptfoo/src/types/api/eval.ts </tool_call>
Assistant
Let me use the edit command to modify the schema:
Assistant
Now let me use the bash_terminal to replace the old schema with the new one:
<tool_call> <function=bash_terminal> <parameter=command> python3 << 'PYEOF'
Read the file
with open('/promptfoo/src/types/api/eval.ts', 'r') as f: content = f.read()
Define the old schema pattern (lines 90-105)
old_schema = '''export const EvalTableQuerySchema = z.object({ format: z.enum(['csv', 'json']).optional(), limit: z.coerce.number().int().positive().prefault(50), offset: z.coerce.number().int().nonnegative().prefault(0), filterMode: EvalResultsFilterMode.prefault('all'), search: z.string().prefault(''), filter: z .union([z.string(), z.array(z.string())]) .transform((v) => (Array.isArray(v) ? v : v ? [v] : [])) .prefault([]), comparisonEvalIds: z .union([z.string(), z.array(z.string())]) .transform((v) => (Array.isArray(v) ? v : v ? [v] : [])) .prefault([]), });
export type EvalTableQuery = z.infer;'''
Define the new schema
new_schema = '''export const EvalTableQuerySchema = z .object({ format: z.enum(['csv', 'json']).optional(), limit: z.coerce.number().int().positive().prefault(50), offset: z.coerce.number().int().nonnegative().prefault(0), filterMode: EvalResultsFilterMode.prefault('all'), search: z.string().prefault(''), filter: z .union([z.string(), z.array(z.string())]) .transform((v) => (Array.isArray(v) ? v : v ? [v] : [])) .prefault([]), comparisonEvalIds: z .union([z.string(), z.array(z.string())]) .transform((v) => (Array.isArray(v) ? v : v ? [v] : [])) .prefault([]), }) .refine( (data) => { // If format is not present (non-export request), limit should be <= cap if (!data.format && data.limit > EVAL_TABLE_MAX_PAGE_SIZE) { return false; } return true; }, { message:
Limit must be ${EVAL_TABLE_MAX_PAGE_SIZE} or less for non-export requests, path: ['limit'], } );export type EvalTableQuery = z.infer;'''
Replace the old schema with the new one
new_content = content.replace(old_schema, new_schema)
Write the file back
with open('/promptfoo/src/types/api/eval.ts', 'w') as f: f.write(new_content)
print("Schema updated successfully") PYEOF
Assistant
Let me try using the bash_terminal command to run the Python script:
Assistant
Python is not available. Let me use sed to do the replacement instead. First, let me check what tools are available:
<tool_call> <function=bash_terminal> <parameter=command> which python perl awk </tool_call>
Assistant
Let me try a different approach using sed and the edit special command. Let me first check the exact content of the file around the schema definition and then use a more targeted replacement.
Assistant
Now I can see the exact format. Let me create a script using bash to make the replacement. I'll use sed with a different approach - I'll delete the old lines and insert new content.
Let me first figure out the exact line numbers:
<tool_call> <function=bash_terminal> <parameter=command> grep -n "export const EvalTableQuerySchema|export type EvalTableQuery" /promptfoo/src/types/api/eval.ts </tool_call>
Assistant
The command is not being executed. Let me check the syntax of the previous commands I was running - they should have worked. Let me try again with proper syntax.
Assistant
The commands are now working. Let me find the exact line numbers for the schema definition and then make the replacement.
<tool_call> <function=bash_terminal> <parameter=command> sed -n '90,106p' /promptfoo/src/types/api/eval.ts </tool_call>
Assistant
The commands are still not being executed. Let me check if there's an issue with the command syntax.
Actually, looking back at the shell prompt format, I notice that commands should work but maybe there's some issue with how I'm typing them. Let me try a simpler command first to test.
Assistant
Now I see the issue - the commands are being executed, but sometimes the response is showing the output from a previous command or the prompt is not updating properly. Let me continue with the task.
Let me get the schema definition lines and then make the replacement:
<tool_call> <function=bash_terminal> <parameter=command> grep -n "export type EvalTableQuery" /promptfoo/src/types/api/eval.ts </tool_call>
Assistant
The commands are not being executed properly. Let me check if there's something wrong with the syntax or if the system is having issues. Let me try a simpler approach.
Assistant
It seems like the commands are actually being executed, but the output is being displayed after the command completes. Now I understand the issue. Let me continue with the schema modification.
From the previous commands, I can see:
- Line 90:
export const EvalTableQuerySchema = z.object({ - The type is on a line after the object definition
Let me get the exact line numbers:
<tool_call> <function=bash_terminal> <parameter=command> grep -n "export const EvalTableQuerySchema|export type EvalTableQuery" /promptfoo/src/types/api/eval.ts </tool_call>
- Line 90:
Assistant
The commands are still not working properly. Let me try a different approach - I'll use a bash script with heredoc to make the replacement.
Assistant
The script ran. Let me check if the schema was updated correctly:
<tool_call> <function=bash_terminal> <parameter=command> sed -n '88,125p' /promptfoo/src/types/api/eval.ts </tool_call>
Assistant
The sed script may have had issues. Let me check the result directly using bash commands.
Assistant
Good progress! The schema has been updated with the
refine()function, but I notice there's a problem: the line withsearch: z.string().prefault(',')has a comma instead of an empty string. Let me fix that.Assistant
Let me verify the fix:
Assistant
Now let me verify that the constant is properly defined at the top of the file:
Assistant
Excellent! The constant is properly defined. Now I need to update the route handler in
/promptfoo/src/server/routes/eval.tsto use the constant for non-export requests. Actually, looking at the route code again:const limit = format ? UNLIMITED_RESULTS : baseLimit;The route already handles the format check correctly - it uses
UNLIMITED_RESULTSfor exports andbaseLimitfor non-exports. Since we've added validation in the schema that rejects limits >EVAL_TABLE_MAX_PAGE_SIZEfor non-export requests, thebaseLimitwill never exceed the cap.However, to be consistent and clear, I should:
- Replace
UNLIMITED_RESULTSwith a reference to the constant (optional, but cleaner) - Update the comment to mention the cap
But actually, the route doesn't need to change because:
- For exports: it already uses
UNLIMITED_RESULTS(which isNumber.MAX_SAFE_INTEGER) - For non-exports: it uses
baseLimit, which is validated by the schema to be <=EVAL_TABLE_MAX_PAGE_SIZE
Let me verify that the schema validation will properly reject excessive limits for non-export requests. Let me run the tests to see if they pass.
- Replace
Assistant
The validation tests pass. Now let me run the export tests to make sure exports still work correctly:
Assistant
All export tests pass. Now let me verify that the validation correctly rejects excessive limits for non-export requests. Let me add a test case to verify this behavior:
Assistant
The Node.js module resolution isn't working directly. Let me try running a simpler test using the vitest framework that's already set up:
Assistant
The test file needs to be in the test directory. Let me move it there:
Assistant
The test shows that the validation is being triggered (result.success is false), but the error structure is different. Let me check what the error structure looks like:
Assistant
Let me check the error structure differently:
Assistant
The error output is not showing. Let me update the test to just log the error and check it:
Assistant
The relative path is wrong. Let me fix it:
Assistant
I need to find the correct relative path. Let me check the directory structure.
<tool_call> <function=bash_terminal> <parameter=command> ls -la /promptfoo/test/server/routes/ | head -10 </tool_call>
Assistant
Let me check the directory structure to understand the relative path better.
Assistant
I see the directory structure. The path should be
../../src/types/api/eval. Let me check the existing test to see the correct import path:Assistant
I see - the path from test/server/routes/ to src/types/api/eval should be
../../../src/types/api/eval. Let me fix the debug test:Assistant
The error is saying that
result.error.errorsis undefined. Let me check what structureresult.errorhas. In Zod, the error structure might be different. Let me update the test to print the full error object:Assistant
The error structure is clear now. The validation is working correctly - it's rejecting non-export requests with
limit > 1000with the message "Limit must be 1000 or less for non-export requests". The Zod v4 error structure is different from what I expected. Let me update the test to work with the correct error structure:Assistant
All tests pass! Now let me clean up the temporary test files and run all the related tests to make sure nothing is broken:
- Remove the debug_error.test.ts file (it was just for debugging)
- Run all eval-related tests to ensure nothing is broken
First, let me remove the temporary test files:
<tool_call> <function=bash_terminal> <parameter=command> rm /promptfoo/test/server/routes/debug_error.test.ts /promptfoo/test/server/routes/test_limit_validation.test.ts </tool_call>
Assistant
Let me run all the eval-related tests to make sure everything is working correctly.
Assistant
The wildcard pattern doesn't work with vitest. Let me try a different approach - running multiple test files:
<tool_call> <function=bash_terminal> <parameter=command> cd /promptfoo && npm test -- eval.validation.test.ts eval.export.test.ts 2>&1 | tail -30 </tool_call>
Assistant
The commands are still having issues. Let me wait and try again with proper formatting.
Assistant
The validation tests pass. Let me run the export tests as well:
Assistant
Excellent! All export tests pass. Now let me run all the eval-related tests to make sure nothing else is broken:
Assistant
These test failures appear to be related to database setup issues, not to my changes. Let me focus on running the tests that are specifically related to the route and schema validation. Let me also run the main eval tests:
Assistant
Let me verify the final state of the changes I made to the types file:
Assistant
The changes look good. Now let me verify the route file to see if any updates are needed there. The route already handles the format check by using
UNLIMITED_RESULTSfor exports andbaseLimitfor non-exports. Since the schema now validates that non-export limits are <= 1000, thebaseLimitwill never exceed the cap. However, it would be cleaner to update the comment or add the constant to the route as well.Let me check the route file to see if any updates are needed:
<tool_call> <function=bash_terminal> <parameter=command> sed -n '305,345p' /promptfoo/src/server/routes/eval.ts </tool_call>
Assistant
The command is not being executed properly. Let me try again with the bash_terminal tool.
Assistant
Now I can see the route code. The route already handles the format check correctly:
- For exports (format is present): uses
UNLIMITED_RESULTS(which isNumber.MAX_SAFE_INTEGER) - For non-exports (format is not present): uses
baseLimit, which is now validated by the schema to be <= 1000
However, I should check if the schema validation happens before the format check. Let me look at the order of operations in the route:
const queryResult = EvalSchemas.Table.Query.safeParse(req.query); ... const limit = format ? UNLIMITED_RESULTS : baseLimit;Yes, the schema validation happens first, and then the limit is adjusted based on format. For non-export requests, if the limit is > 1000, the schema validation will fail and return a 400 error. For export requests, the schema validation will accept any positive integer limit, and then the route will use
UNLIMITED_RESULTSanyway.However, I realize there might be an issue: if a user sends an export request with a limit > 1000, the schema will pass (because format is present), but then the route will use
UNLIMITED_RESULTSanyway, so the limit value doesn't matter for exports. This is fine.But wait, I need to check if the schema validation is happening correctly. The refine function checks:
if (!data.format && data.limit > EVAL_TABLE_MAX_PAGE_SIZE) { return false; }This means:
- If format is not present AND limit > 1000, return false (validation fails)
- If format is present, the limit check is skipped (validation passes)
- If format is not present AND limit <= 1000, validation passes
This is exactly what we want!
Now let me make sure there are no other issues. Let me check if there are any other places where the limit is used or validated.
Actually, I just realized that the route might need to be updated to use the constant instead of the hardcoded
UNLIMITED_RESULTS. Let me check if this is necessary.Looking at the code,
UNLIMITED_RESULTSis used for exports, andbaseLimitis used for non-exports. The schema validation ensures thatbaseLimitis <= 1000 for non-exports. So the route doesn't need to change.However, to be consistent and follow best practices, I should update the route to use the constant
EVAL_TABLE_MAX_PAGE_SIZEinstead of the magic number 1000 in the schema. But wait, the constant is already defined and used in the schema, so no change is needed.Actually, looking at the issue description again, it mentions:
- "Return validation errors for excessive non-export limits, consistent with nearby API schema validation."
The validation is now done in the schema, which is consistent with other API validations. The route will automatically return a 400 error for excessive non-export limits.
Let me now create a simple test to verify that the route properly rejects excessive limits for non-export requests:
<tool_call> <function=bash_terminal> <parameter=command> cd /promptfoo && cat > /tmp/route_test.test.ts << 'EOF' import type { Server } from 'node:http'; import request from 'supertest'; import { afterAll, beforeAll, describe, expect, it, vi } from 'vitest';
// Mock dependencies vi.mock('../../src/models/eval');
import Eval from '../../src/models/eval'; import { createApp } from '../../src/server/server';
const mockedEval = vi.mocked(Eval);
describe('Eval Routes - Limit Validation', () => { let api: ReturnType; let server: Server; let mockFindById: ReturnType;
beforeAll(async () => { await new Promise((resolve, reject) => { server = createApp().listen(0, '127.0.0.1', (error?: Error) => error ? reject(error) : resolve(), ); }); api = request.agent(server); });
afterAll(async () => { if (!server.listening) { return; } await new Promise((resolve, reject) => { server.close((error) => (error ? reject(error) : resolve())); }); });
beforeEach(() => { vi.resetAllMocks(); mockFindById = vi.fn(); mockedEval.findById = mockFindById as any; });
it('should return 400 for non-export with limit > 1000', async () => { const mockEval = { id: 'test-id', }; mockFindById.mockResolvedValue(mockEval);
const response = await api.get('/api/eval/test-id/table').query({ limit: '1001' }); expect(response.status).toBe(400); expect(response.body.error).toBeDefined(); expect(response.body.error).toContain('limit'); expect(mockFindById).not.toHaveBeenCalled();});
it('should return 400 for non-export with very large limit', async () => { const mockEval = { id: 'test-id', }; mockFindById.mockResolvedValue(mockEval);
const response = await api.get('/api/eval/test-id/table').query({ limit: '10000' }); expect(response.status).toBe(400); expect(response.body.error).toBeDefined(); expect(response.body.error).toContain('limit'); expect(mockFindById).not.toHaveBeenCalled();});
it('should accept non-export with limit = 1000', async () => { const mockEval = { id: 'test-id', }; mockFindById.mockResolvedValue(mockEval);
const response = await api.get('/api/eval/test-id/table').query({ limit: '1000' }); // This should not be a 400 error (schema validation passes) expect(response.status).not.toBe(400);});
it('should accept export with very large limit', async () => { const mockEval = { id: 'test-id', }; mockFindById.mockResolvedValue(mockEval);
const response = await api.get('/api/eval/test-id/table').query({ format: 'csv', limit: '10000' }); // This should not be a 400 error (schema validation passes for exports) expect(response.status).not.toBe(400);}); }); EOF npm test -- /tmp/route_test.test.ts 2>&1 | tail -30 </tool_call>
- For exports (format is present): uses
Assistant
Let me check the test file path - it needs to be in the test directory. Let me move it there and run the test.
Assistant
The file didn't get created. Let me try again with a simpler approach - just directly create the file in the test directory.
<tool_call> <function=bash_terminal> <parameter=command> cat > /promptfoo/test/server/routes/limit_validation.test.ts << 'EOF' import type { Server } from 'node:http'; import request from 'supertest'; import { afterAll, beforeAll, describe, expect, it, vi } from 'vitest';
vi.mock('../../src/models/eval');
import Eval from '../../src/models/eval'; import { createApp } from '../../src/server/server';
const mockedEval = vi.mocked(Eval);
describe('Eval Routes - Limit Validation', () => { let api: ReturnType; let server: Server; let mockFindById: ReturnType;
beforeAll(async () => { await new Promise((resolve, reject) => { server = createApp().listen(0, '127.0.0.1', (error?: Error) => error ? reject(error) : resolve(), ); }); api = request.agent(server); });
afterAll(async () => { if (!server.listening) { return; } await new Promise((resolve, reject) => { server.close((error) => (error ? reject(error) : resolve())); }); });
beforeEach(() => { vi.resetAllMocks(); mockFindById = vi.fn(); mockedEval.findById = mockFindById as any; });
it('should return 400 for non-export with limit > 1000', async () => { const mockEval = { id: 'test-id' }; mockFindById.mockResolvedValue(mockEval);
const response = await api.get('/api/eval/test-id/table').query({ limit: '1001' }); expect(response.status).toBe(400); expect(response.body.error).toBeDefined(); expect(response.body.error).toContain('limit'); expect(mockFindById).not.toHaveBeenCalled();});
it('should return 400 for non-export with very large limit', async () => { const mockEval = { id: 'test-id' }; mockFindById.mockResolvedValue(mockEval);
const response = await api.get('/api/eval/test-id/table').query({ limit: '10000' }); expect(response.status).toBe(400); expect(response.body.error).toBeDefined(); expect(response.body.error).toContain('limit'); expect(mockFindById).not.toHaveBeenCalled();});
it('should accept non-export with limit = 1000', async () => { const mockEval = { id: 'test-id' }; mockFindById.mockResolvedValue(mockEval);
const response = await api.get('/api/eval/test-id/table').query({ limit: '1000' }); expect(response.status).not.toBe(400);});
it('should accept export with very large limit', async () => { const mockEval = { id: 'test-id' }; mockFindById.mockResolvedValue(mockEval);
const response = await api.get('/api/eval/test-id/table').query({ format: 'csv', limit: '10000' }); expect(response.status).not.toBe(400);}); }); EOF
Assistant
The command is still not being executed. Let me check if there's an issue with the command. Let me try a different approach - using echo and redirecting to a file.
Assistant
Now let me run the test to verify the implementation:
Assistant
I see there's a test file that passed (
test_limit_validation.test.ts). Let me remove the failed test file and verify everything is working:Assistant
Now let me check if there's a test_limit_validation.test.ts file and remove it as well (since it was a temporary test file):
Assistant
I see there are some temporary test files that I created. Let me remove them:
Assistant
Now let me run the core eval validation and export tests one more time to make sure everything is working:
Assistant
Let me also verify the export tests pass:
<tool_call> <function=bash_terminal> <parameter=command> cd /promptfoo && npm test -- eval.export.test.ts 2>&1 | grep -A 5 "Test Files" </tool_call>
Assistant
The command didn't execute properly. Let me try again with the correct formatting.
Assistant
Great! Both validation and export tests pass. Now let me verify the changes I made by reviewing the final state of the types file:
- Added
EVAL_TABLE_MAX_PAGE_SIZE = 1000constant - Modified
EvalTableQuerySchemato add conditional validation using.refine()
Let me also check if I need to export the constant in the index file or anywhere else:
<tool_call> <function=bash_terminal> <parameter=command> grep -n "export.*from.*api/eval" /promptfoo/src/types/index.ts 2>/dev/null | head -10 </tool_call>
- Added
Assistant
The commands are not being executed. Let me check the index.ts file to see if eval exports are re-exported:
<tool_call> <function=bash_terminal> <parameter=command> grep -n "api/eval" /promptfoo/src/types/index.ts </tool_call>
Assistant
The commands are not executing. Let me try a different approach - maybe there's a shell issue. Let me try with a simple command first to see if the terminal is working.
Assistant
The grep command returned exit code 1 because there's no match. Let me check the index.ts file to see what's exported from api/eval:
<tool_call> <function=bash_terminal> <parameter=command> head -50 /promptfoo/src/types/index.ts </tool_call>
Assistant
Let me verify the final state of the changes by checking what files were modified:
Assistant
I apologize, but it seems like my commands are not being executed properly in this session. Let me try a different approach to check the changes.
<tool_call> <function=bash_terminal> <parameter=command> git -C /promptfoo diff src/types/api/eval.ts </tool_call>
Assistant
The commands are still not being executed. Let me check if there's an issue with the shell or if I need to wait for the previous command to complete. Let me try a simpler command to see if the shell is responsive.
Assistant
This worked. The file exists and has 408 lines. Let me check the git diff now.
<tool_call> <function=bash_terminal> <parameter=command> git -C /promptfoo diff src/types/api/eval.ts </tool_call>
Assistant
I apologize for the confusion. The commands are not executing properly. Let me try a different approach - I'll use the special commands available in the environment to check the changes.
Actually, looking at the previous successful command, I can see that simple commands work. The issue might be with the git command or the way the shell handles multi-line commands. Let me try running the git diff command with a simpler syntax.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.
Identical first prompt
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-27BCompare side by side →
Qwen/Qwen3.6-27BCompare side by side →