Skip to content
TheChatArchiveThe Chat Archive
QwenQwen/Qwen3.5-35B-A3B· alibabaHugging Face ibragim-bad/swe_rebench_07_2026_trajectories

ISSUE DESCRIPTION Cap eval table page size for non-export requests Summary GET…

  1. User

    ISSUE DESCRIPTION

    Cap eval table page size for non-export requests

    Summary

    GET /api/eval/:id/table accepts any positive integer for limit on normal table requests. The route already intentionally treats export requests differently by using Number.MAX_SAFE_INTEGER when format is set, but non-export page requests currently do not have an upper bound.

    Why this matters

    A very large limit can make a normal table request ask the server to load and serialize a large amount of eval result data in one response. Other list-style APIs in the server already bound their page sizes, so the eval table endpoint should follow the same resource-control pattern.

    Proposed change

    • Add a maximum limit for non-export GET /api/eval/:id/table requests.
    • Keep format=csv / format=json exports unchanged so full-table exports still work as intended.
    • Return validation errors for excessive non-export limits, consistent with nearby API schema validation.

    A cap of 500 or 1000 would likely cover normal UI usage while preventing accidental oversized page requests.

    Validation

    Suggested tests:

    • schema test rejects EvalSchemas.Table.Query.safeParse({ limit: '<cap + 1>' })
    • route validation test returns 400 for excessive non-export limit
    • export route tests continue to use Number.MAX_SAFE_INTEGER when format is present

    Interface

    Type: Constant Name: EVAL_TABLE_MAX_PAGE_SIZE Path: src/types/api/eval.ts Value/Type: constant literal value 1000 (number) Description: Server-enforced upper bound for the limit parameter on non-export eval table page requests. Exports using format (csv/json) are exempt from this cap; UI pagination controls derive their available page-size options from this value.

    ADDITIONAL ADVICE

    Since you are given a git repository, you can use git commands to simplify your work. For example, if you made a mistake and want to revert your changes, you can use git checkout <file> to restore the file to its original state. You can also reset all changes in the repository using git reset --hard command. Additionally, you can use git stash and git stash pop to temporarily save your changes and restore them later. Finally, you can see the changes that you've made (which will be submitted when you call submit) using git status or git diff commands. However you don't need to use git add or git commit before submitting your changes. In fact, submit only submits currently unstaged changes because it uses git diff to get the changes that need to be submitted.

    CHECKLIST

    Before submitting your solution, please go over the following checklist and make sure you've done everything:

    • If an error was reported in the issue description, I have successfully reproduced it.
    • If an error was reported in the issue description, I have confirmed that my fix resolves the error.
    • I have thought about edge cases and made sure my fix handles them as well.
    • I have run existing tests in the repository that might have been affected by the change I made and confirmed that they pass. I want you to list every bullet from this checklist and write your reasoning for why you think you did it or didn't need to.

    Repository has been uploaded and your shell is currently at the repository root. Time to solve the issue!

    (Current directory: /promptfoo, current file: none) bash-$

  2. Qwen Assistant

    I'll start by exploring the repository structure to understand the codebase and find the relevant files for implementing this fix.

  3. Qwen Assistant

    Let me explore the repository structure more systematically, especially looking for the src/types/api/eval.ts file mentioned in the issue.

  4. Qwen Assistant

    I found the file mentioned in the issue. Let me examine its contents.

  5. Qwen Assistant

    Now I need to find the route handler for GET /api/eval/:id/table. Let me search for it.

  6. Qwen Assistant

    Let me search for the route handler more broadly by looking for files that might contain the actual API implementation.

  7. Qwen Assistant

    Let me search for files containing "/table" endpoint routes.

  8. Qwen Assistant

    I found the route file. Let me examine the eval routes file.

    <tool_call> <function=bash_terminal> <parameter=command> cat /promptfoo/src/server/routes/eval.ts </tool_call>

  9. Qwen Assistant

    Let me read the eval route file using bash terminal.

  10. Qwen Assistant

    Let me continue reading the file to find the table endpoint handler. The output was truncated.

    <tool_call> <function=bash_terminal> <parameter=command> grep -n "GET.*:id/table" /promptfoo/src/server/routes/eval.ts </tool_call>

  11. Qwen Assistant

    Let me search for the table endpoint handler in the eval.ts route file.

  12. Qwen Assistant

    I see that the table endpoint is at line 311. Let me look at that section more carefully.

  13. Qwen Assistant

    Now I can see the issue. At line 327, the route handler uses UNLIMITED_RESULTS when format is specified:

    const limit = format ? UNLIMITED_RESULTS : baseLimit;
    

    But I need to find where UNLIMITED_RESULTS is defined and also where I need to add the cap for non-export requests. Let me search for it.

    <tool_call> <function=bash_terminal> <parameter=command> grep -n "UNLIMITED_RESULTS" /promptfoo/src/server/routes/eval.ts </tool_call>

  14. Qwen Assistant

    Let me search for where UNLIMITED_RESULTS is defined.

  15. Qwen Assistant

    Now I understand the issue. The route already uses UNLIMITED_RESULTS (which is Number.MAX_SAFE_INTEGER) for export requests (when format is set), but doesn't cap the limit for non-export requests.

    Let me now:

    1. Add the EVAL_TABLE_MAX_PAGE_SIZE constant to the eval.ts types file
    2. Update the route handler to use this cap for non-export requests
    3. Find and run existing tests to make sure nothing breaks

    First, let me look at the test files to understand the test structure.

    <tool_call> <function=bash_terminal> <parameter=command> find /promptfoo -name "*.test.ts" -path "routes" | head -10 </tool_call>

  16. Qwen Assistant

    Let me search for test files related to eval routes.

  17. Qwen Assistant

    I found relevant test files. Let me check the validation test and export test to understand the current test structure.

    <tool_call> <function=bash_terminal> <parameter=command> cat /promptfoo/test/server/routes/eval.validation.test.ts </tool_call>

  18. Qwen Assistant

    I need to use bash_terminal to read the file.

  19. Qwen Assistant

    Now let me also check the export test to understand how export requests are tested.

    <tool_call> <function=bash_terminal> <parameter=command> cat /promptfoo/test/server/routes/eval.export.test.ts </tool_call>

  20. Qwen Assistant

    I need to use bash_terminal to read the file instead.

  21. Qwen Assistant

    Now I have a good understanding of the codebase. Let me create a plan:

    1. Add the EVAL_TABLE_MAX_PAGE_SIZE constant to /promptfoo/src/types/api/eval.ts
    2. Update the EvalTableQuerySchema to add max validation for the limit field (but only for non-export requests)
    3. Update the route handler in /promptfoo/src/server/routes/eval.ts to use the cap for non-export requests
    4. Run existing tests to make sure nothing breaks

    Let me start by looking at the current state of the schema to understand how to properly add the max constraint. The schema currently has:

    limit: z.coerce.number().int().positive().prefault(50),
    

    I need to add a max constraint. However, since exports should be unlimited, the validation should happen at the schema level only for non-export cases. Let me think about this more carefully.

    Actually, looking at the issue description more carefully:

    • "Add a maximum limit for non-export GET /api/eval/:id/table requests."
    • "Keep format=csv / format=json exports unchanged so full-table exports still work as intended."
    • "Return validation errors for excessive non-export limits, consistent with nearby API schema validation."

    So the validation should reject excessive non-export limits. The approach should be:

    1. Define the constant EVAL_TABLE_MAX_PAGE_SIZE = 1000 in the types file
    2. Modify the EvalTableQuerySchema to have max validation for the limit field
    3. But the route handler needs to handle exports differently - when format is present, ignore the max validation

    Wait, the issue says the validation should reject excessive limits. But exports use format parameter. So we can't just add a max constraint to the schema because that would also reject exports with a high limit.

    Looking at the code again:

    const limit = format ? UNLIMITED_RESULTS : baseLimit;
    

    The route already handles the format check. So we could:

    1. Add the max validation to the schema with the cap value
    2. In the route handler, before validation, check if format is present and conditionally use a different schema or skip the limit check

    Actually, a simpler approach is to:

    1. Add the max validation with the cap value
    2. Modify the schema so that when format is present, it uses a different schema (or uses a different transformation)

    But looking at the existing test file eval.validation.test.ts, it has a test for the table endpoint that tests validation errors. The issue says:

    • "schema test rejects EvalSchemas.Table.Query.safeParse({ limit: '<cap + 1>' })"
    • "route validation test returns 400 for excessive non-export limit"

    So the validation should be done at the schema level for non-export requests. Let me think about the cleanest way to do this.

    One approach is to modify the schema to conditionally apply the max constraint based on whether format is present. However, Zod doesn't easily support conditional schemas based on optional fields.

    A better approach is:

    1. Add the constant EVAL_TABLE_MAX_PAGE_SIZE = 1000
    2. Keep the schema as is (or add the max constraint but handle exports specially in the route)
    3. In the route handler, check if the limit is too high when format is NOT present, and return a validation error

    Actually, re-reading the issue more carefully:

    • "Return validation errors for excessive non-export limits, consistent with nearby API schema validation."

    So we need to return validation errors. The question is where to do this validation:

    • At the schema level (cleaner, uses existing error handling)
    • At the route level (more control, but duplicates logic)

    Looking at the existing code structure, I think the best approach is:

    1. Add the constant EVAL_TABLE_MAX_PAGE_SIZE = 1000 in the types file
    2. Modify the schema to include the max constraint
    3. Modify the route handler to use a different schema (or modify the query) when format is present

    Wait, let me look at how the route handles the format check. The route already checks format ? UNLIMITED_RESULTS : baseLimit after the schema validation. So the schema validation happens first, then the limit is adjusted.

    The cleanest approach is to:

    1. Add the constant EVAL_TABLE_MAX_PAGE_SIZE = 1000
    2. Keep the schema with the max constraint
    3. In the route handler, when format is present, manually validate without the max constraint (or adjust the limit before passing to the schema)

    Actually, I think the simplest approach is to add the max constraint to the schema and then handle exports differently by adjusting the limit before schema validation when format is present. But that's complex.

    Let me look at the issue description again:

    • "schema test rejects EvalSchemas.Table.Query.safeParse({ limit: '<cap + 1>' })"

    This suggests the schema should reject excessive limits. But exports use format parameter, so we need to handle this conditionally.

    I think the best approach is:

    1. Add the constant EVAL_TABLE_MAX_PAGE_SIZE = 1000
    2. Modify the schema to have a max constraint on limit
    3. In the route handler, when format is present, temporarily remove the limit from the query or adjust it before validation

    Actually, I realize that we can't easily make the schema conditional. The simplest approach is:

    1. Add the constant EVAL_TABLE_MAX_PAGE_SIZE = 1000
    2. Add the max constraint to the schema
    3. In the route handler, before validation, if format is present, set limit to MAX_SAFE_INTEGER or remove it from the query
    4. After validation, if format is present, set limit to MAX_SAFE_INTEGER again

    Wait, but the issue says "Return validation errors for excessive non-export limits". This means the validation should happen and return a 400 error for excessive limits when format is NOT present.

    Let me re-think this:

    1. The schema should reject excessive limits for non-export requests
    2. For export requests, the schema should accept any positive integer

    The cleanest way to do this is to modify the schema to use a conditional constraint. However, Zod's conditional schemas are complex.

    A simpler approach is:

    1. Add the constant EVAL_TABLE_MAX_PAGE_SIZE = 1000
    2. Keep the schema with a max constraint
    3. In the route handler, before schema validation, if format is present, set limit to a value <= cap (e.g., cap) so the schema accepts it
    4. After validation, set limit to MAX_SAFE_INTEGER for exports

    Actually, the route already sets const limit = format ? UNLIMITED_RESULTS : baseLimit; after validation. So we need to handle the validation differently.

    Let me look at the route again:

    const queryResult = EvalSchemas.Table.Query.safeParse(req.query);
    if (!queryResult.success) {
      res.status(400).json({ error: z.prettifyError(queryResult.error) });
      return;
    }
    

    The validation happens before we check if format is present. So we need to:

    1. Modify the schema to have conditional max constraint based on format
    2. Or handle the validation manually for the limit field

    I think the cleanest approach is to modify the schema to accept any positive integer for limit (remove the max constraint), and then do manual validation in the route handler for non-export requests. This way, exports can have any limit value (which they'll set to MAX_SAFE_INTEGER anyway).

    Wait, but the issue says "schema test rejects EvalSchemas.Table.Query.safeParse({ limit: '<cap + 1>' })". This suggests the schema itself should reject excessive limits.

    Hmm, I think I need to re-read the issue more carefully. The issue says:

    • "Add a maximum limit for non-export GET /api/eval/:id/table requests."
    • "Keep format=csv / format=json exports unchanged so full-table exports still work as intended."
    • "Return validation errors for excessive non-export limits, consistent with nearby API schema validation."

    So the validation should reject excessive limits for non-export requests. But how do we make the schema conditional based on format?

    One approach is to use Zod's when or refine method to add a conditional constraint. Let me check if that's possible.

    Actually, I think the simplest approach is:

    1. Add the constant EVAL_TABLE_MAX_PAGE_SIZE = 1000
    2. Modify the schema to accept any positive integer for limit
    3. In the route handler, after validation, check if limit is too high AND format is not present, and return a validation error

    But this doesn't match the issue description which says "schema test rejects...". So maybe the intent is to have the schema do the validation, not the route.

    Let me think about this differently. The schema has a format field which is optional. We can use Zod's conditional schema to:

    • If format is present, allow any positive integer for limit
    • If format is not present, only allow limit <= 1000

    This can be done with z.object().refine() or using a conditional schema.

    Actually, a better approach is to use Zod's when or to create two different schemas: one for exports and one for non-exports. But that might be overkill.

    Let me just implement a solution using z.object().refine() to add a conditional constraint:

    export const EvalTableQuerySchema = z
      .object({
        format: z.enum(['csv', 'json']).optional(),
        limit: z.coerce.number().int().positive().prefault(50),
        offset: z.coerce.number().int().nonnegative().prefault(0),
        ...
      })
      .refine(
        (data) => {
          // If format is not present, limit should be <= cap
          if (!data.format && data.limit > EVAL_TABLE_MAX_PAGE_SIZE) {
            return false;
          }
          return true;
        },
        {
          message: `Limit must be ${EVAL_TABLE_MAX_PAGE_SIZE} or less for non-export requests`,
          path: ['limit'],
        }
      );
    

    This approach uses Zod's refine to add a conditional constraint that checks if format is not present and if limit exceeds the cap.

    Let me now implement this solution.

    First, I'll add the constant to the types file:

    // src/types/api/eval.ts
    
    export const EVAL_TABLE_MAX_PAGE_SIZE = 1000;
    
    // Then modify the EvalTableQuerySchema to use refine() for conditional validation
    

    Then, I'll update the route handler to use EVAL_TABLE_MAX_PAGE_SIZE instead of UNLIMITED_RESULTS for non-export requests, but actually, the route already uses UNLIMITED_RESULTS for exports and baseLimit for non-exports. So the only change needed is to cap the baseLimit to EVAL_TABLE_MAX_PAGE_SIZE when format is not present.

    Wait, but the schema validation will reject limits > cap anyway, so the route will never receive a limit > cap for non-export requests. The route just needs to use baseLimit directly (which will be <= cap) for non-export requests.

    Actually, I think the route needs to use Math.min(baseLimit, EVAL_TABLE_MAX_PAGE_SIZE) or just use baseLimit since the schema will validate it. But the schema should reject values > cap, so baseLimit will always be <= cap.

    Let me implement this step by step.

    Actually, I just realized that the route already does:

    const limit = format ? UNLIMITED_RESULTS : baseLimit;
    

    So for non-export requests, it uses baseLimit which is validated by the schema. If the schema rejects values > cap, then baseLimit will always be <= cap, and the route will use a reasonable value.

    But wait, the issue says "schema test rejects...". This means the schema should reject values > cap for non-export requests. So the implementation should:

    1. Add the constant EVAL_TABLE_MAX_PAGE_SIZE = 1000
    2. Modify the schema to conditionally reject values > cap when format is not present
    3. The route can stay the same (it already uses baseLimit for non-export requests)

    Let me now implement this.

    Implementation Plan

    1. Add EVAL_TABLE_MAX_PAGE_SIZE = 1000 constant to /promptfoo/src/types/api/eval.ts
    2. Modify EvalTableQuerySchema to use refine() for conditional max validation
    3. Run existing tests to ensure nothing breaks

    Let me start with step 1.

    <tool_call> <function=bash_terminal> <parameter=command> cat /promptfoo/src/types/api/eval.ts | head -30 </tool_call>

  22. Qwen Assistant

    I need to use bash_terminal to read the file and then edit it. Let me add the constant to the file first.

  23. Qwen Assistant

    Now let me add the constant EVAL_TABLE_MAX_PAGE_SIZE = 1000 to the file. I'll add it right after the imports and before the first schema definition.

  24. Qwen Assistant

    Good, the constant was added. Let me verify it was added correctly.

  25. Qwen Assistant

    Now I need to find the EvalTableQuerySchema and modify it to add conditional validation. Let me locate it.

  26. Qwen Assistant

    Let me see the current schema definition.

  27. Qwen Assistant

    Now I need to modify the schema to add conditional validation using refine(). I'll add a refinement that checks if format is not present and if limit exceeds the cap, it should fail validation.

    I need to add the import for EVAL_TABLE_MAX_PAGE_SIZE since I'm using it in the refine function. Actually, since both are in the same file, I can just reference it directly.

    Let me modify the schema:

    export const EvalTableQuerySchema = z
      .object({
        format: z.enum(['csv', 'json']).optional(),
        limit: z.coerce.number().int().positive().prefault(50),
        offset: z.coerce.number().int().nonnegative().prefault(0),
        filterMode: EvalResultsFilterMode.prefault('all'),
        search: z.string().prefault(''),
        filter: z
          .union([z.string(), z.array(z.string())])
          .transform((v) => (Array.isArray(v) ? v : v ? [v] : []))
          .prefault([]),
        comparisonEvalIds: z
          .union([z.string(), z.array(z.string())])
          .transform((v) => (Array.isArray(v) ? v : v ? [v] : []))
          .prefault([]),
      })
      .refine(
        (data) => {
          // If format is not present (non-export request), limit should be <= cap
          if (!data.format && data.limit > EVAL_TABLE_MAX_PAGE_SIZE) {
            return false;
          }
          return true;
        },
        {
          message: `Limit must be ${EVAL_TABLE_MAX_PAGE_SIZE} or less for non-export requests`,
          path: ['limit'],
        }
      );
    

    Let me apply this change using sed or by editing the file.

    <tool_call> <function=bash_terminal> <parameter=command> sed -n '90,105p' /promptfoo/src/types/api/eval.ts </tool_call>

  28. Qwen Assistant

    Let me use the edit command to modify the schema:

  29. Qwen Assistant

    Now let me use the bash_terminal to replace the old schema with the new one:

    <tool_call> <function=bash_terminal> <parameter=command> python3 << 'PYEOF'

    Read the file

    with open('/promptfoo/src/types/api/eval.ts', 'r') as f: content = f.read()

    Define the old schema pattern (lines 90-105)

    old_schema = '''export const EvalTableQuerySchema = z.object({ format: z.enum(['csv', 'json']).optional(), limit: z.coerce.number().int().positive().prefault(50), offset: z.coerce.number().int().nonnegative().prefault(0), filterMode: EvalResultsFilterMode.prefault('all'), search: z.string().prefault(''), filter: z .union([z.string(), z.array(z.string())]) .transform((v) => (Array.isArray(v) ? v : v ? [v] : [])) .prefault([]), comparisonEvalIds: z .union([z.string(), z.array(z.string())]) .transform((v) => (Array.isArray(v) ? v : v ? [v] : [])) .prefault([]), });

    export type EvalTableQuery = z.infer;'''

    Define the new schema

    new_schema = '''export const EvalTableQuerySchema = z .object({ format: z.enum(['csv', 'json']).optional(), limit: z.coerce.number().int().positive().prefault(50), offset: z.coerce.number().int().nonnegative().prefault(0), filterMode: EvalResultsFilterMode.prefault('all'), search: z.string().prefault(''), filter: z .union([z.string(), z.array(z.string())]) .transform((v) => (Array.isArray(v) ? v : v ? [v] : [])) .prefault([]), comparisonEvalIds: z .union([z.string(), z.array(z.string())]) .transform((v) => (Array.isArray(v) ? v : v ? [v] : [])) .prefault([]), }) .refine( (data) => { // If format is not present (non-export request), limit should be <= cap if (!data.format && data.limit > EVAL_TABLE_MAX_PAGE_SIZE) { return false; } return true; }, { message: Limit must be ${EVAL_TABLE_MAX_PAGE_SIZE} or less for non-export requests, path: ['limit'], } );

    export type EvalTableQuery = z.infer;'''

    Replace the old schema with the new one

    new_content = content.replace(old_schema, new_schema)

    Write the file back

    with open('/promptfoo/src/types/api/eval.ts', 'w') as f: f.write(new_content)

    print("Schema updated successfully") PYEOF

  30. Qwen Assistant

    Let me try using the bash_terminal command to run the Python script:

  31. Qwen Assistant

    Python is not available. Let me use sed to do the replacement instead. First, let me check what tools are available:

    <tool_call> <function=bash_terminal> <parameter=command> which python perl awk </tool_call>

  32. Qwen Assistant

    Let me try a different approach using sed and the edit special command. Let me first check the exact content of the file around the schema definition and then use a more targeted replacement.

  33. Qwen Assistant

    Now I can see the exact format. Let me create a script using bash to make the replacement. I'll use sed with a different approach - I'll delete the old lines and insert new content.

    Let me first figure out the exact line numbers:

    <tool_call> <function=bash_terminal> <parameter=command> grep -n "export const EvalTableQuerySchema|export type EvalTableQuery" /promptfoo/src/types/api/eval.ts </tool_call>

  34. Qwen Assistant

    The command is not being executed. Let me check the syntax of the previous commands I was running - they should have worked. Let me try again with proper syntax.

  35. Qwen Assistant

    The commands are now working. Let me find the exact line numbers for the schema definition and then make the replacement.

    <tool_call> <function=bash_terminal> <parameter=command> sed -n '90,106p' /promptfoo/src/types/api/eval.ts </tool_call>

  36. Qwen Assistant

    The commands are still not being executed. Let me check if there's an issue with the command syntax.

    Actually, looking back at the shell prompt format, I notice that commands should work but maybe there's some issue with how I'm typing them. Let me try a simpler command first to test.

  37. Qwen Assistant

    Now I see the issue - the commands are being executed, but sometimes the response is showing the output from a previous command or the prompt is not updating properly. Let me continue with the task.

    Let me get the schema definition lines and then make the replacement:

    <tool_call> <function=bash_terminal> <parameter=command> grep -n "export type EvalTableQuery" /promptfoo/src/types/api/eval.ts </tool_call>

  38. Qwen Assistant

    The commands are not being executed properly. Let me check if there's something wrong with the syntax or if the system is having issues. Let me try a simpler approach.

  39. Qwen Assistant

    It seems like the commands are actually being executed, but the output is being displayed after the command completes. Now I understand the issue. Let me continue with the schema modification.

    From the previous commands, I can see:

    • Line 90: export const EvalTableQuerySchema = z.object({
    • The type is on a line after the object definition

    Let me get the exact line numbers:

    <tool_call> <function=bash_terminal> <parameter=command> grep -n "export const EvalTableQuerySchema|export type EvalTableQuery" /promptfoo/src/types/api/eval.ts </tool_call>

  40. Qwen Assistant

    The commands are still not working properly. Let me try a different approach - I'll use a bash script with heredoc to make the replacement.

  41. Qwen Assistant

    The script ran. Let me check if the schema was updated correctly:

    <tool_call> <function=bash_terminal> <parameter=command> sed -n '88,125p' /promptfoo/src/types/api/eval.ts </tool_call>

  42. Qwen Assistant

    The sed script may have had issues. Let me check the result directly using bash commands.

  43. Qwen Assistant

    Good progress! The schema has been updated with the refine() function, but I notice there's a problem: the line with search: z.string().prefault(',') has a comma instead of an empty string. Let me fix that.

  44. Qwen Assistant

    Let me verify the fix:

  45. Qwen Assistant

    Now let me verify that the constant is properly defined at the top of the file:

  46. Qwen Assistant

    Excellent! The constant is properly defined. Now I need to update the route handler in /promptfoo/src/server/routes/eval.ts to use the constant for non-export requests. Actually, looking at the route code again:

    const limit = format ? UNLIMITED_RESULTS : baseLimit;
    

    The route already handles the format check correctly - it uses UNLIMITED_RESULTS for exports and baseLimit for non-exports. Since we've added validation in the schema that rejects limits > EVAL_TABLE_MAX_PAGE_SIZE for non-export requests, the baseLimit will never exceed the cap.

    However, to be consistent and clear, I should:

    1. Replace UNLIMITED_RESULTS with a reference to the constant (optional, but cleaner)
    2. Update the comment to mention the cap

    But actually, the route doesn't need to change because:

    • For exports: it already uses UNLIMITED_RESULTS (which is Number.MAX_SAFE_INTEGER)
    • For non-exports: it uses baseLimit, which is validated by the schema to be <= EVAL_TABLE_MAX_PAGE_SIZE

    Let me verify that the schema validation will properly reject excessive limits for non-export requests. Let me run the tests to see if they pass.

  47. Qwen Assistant

    The validation tests pass. Now let me run the export tests to make sure exports still work correctly:

  48. Qwen Assistant

    All export tests pass. Now let me verify that the validation correctly rejects excessive limits for non-export requests. Let me add a test case to verify this behavior:

  49. Qwen Assistant

    The Node.js module resolution isn't working directly. Let me try running a simpler test using the vitest framework that's already set up:

  50. Qwen Assistant

    The test file needs to be in the test directory. Let me move it there:

  51. Qwen Assistant

    The test shows that the validation is being triggered (result.success is false), but the error structure is different. Let me check what the error structure looks like:

  52. Qwen Assistant

    Let me check the error structure differently:

  53. Qwen Assistant

    The error output is not showing. Let me update the test to just log the error and check it:

  54. Qwen Assistant

    The relative path is wrong. Let me fix it:

  55. Qwen Assistant

    I need to find the correct relative path. Let me check the directory structure.

    <tool_call> <function=bash_terminal> <parameter=command> ls -la /promptfoo/test/server/routes/ | head -10 </tool_call>

  56. Qwen Assistant

    Let me check the directory structure to understand the relative path better.

  57. Qwen Assistant

    I see the directory structure. The path should be ../../src/types/api/eval. Let me check the existing test to see the correct import path:

  58. Qwen Assistant

    I see - the path from test/server/routes/ to src/types/api/eval should be ../../../src/types/api/eval. Let me fix the debug test:

  59. Qwen Assistant

    The error is saying that result.error.errors is undefined. Let me check what structure result.error has. In Zod, the error structure might be different. Let me update the test to print the full error object:

  60. Qwen Assistant

    The error structure is clear now. The validation is working correctly - it's rejecting non-export requests with limit > 1000 with the message "Limit must be 1000 or less for non-export requests". The Zod v4 error structure is different from what I expected. Let me update the test to work with the correct error structure:

  61. Qwen Assistant

    All tests pass! Now let me clean up the temporary test files and run all the related tests to make sure nothing is broken:

    1. Remove the debug_error.test.ts file (it was just for debugging)
    2. Run all eval-related tests to ensure nothing is broken

    First, let me remove the temporary test files:

    <tool_call> <function=bash_terminal> <parameter=command> rm /promptfoo/test/server/routes/debug_error.test.ts /promptfoo/test/server/routes/test_limit_validation.test.ts </tool_call>

  62. Qwen Assistant

    Let me run all the eval-related tests to make sure everything is working correctly.

  63. Qwen Assistant

    The wildcard pattern doesn't work with vitest. Let me try a different approach - running multiple test files:

    <tool_call> <function=bash_terminal> <parameter=command> cd /promptfoo && npm test -- eval.validation.test.ts eval.export.test.ts 2>&1 | tail -30 </tool_call>

  64. Qwen Assistant

    The commands are still having issues. Let me wait and try again with proper formatting.

  65. Qwen Assistant

    The validation tests pass. Let me run the export tests as well:

  66. Qwen Assistant

    Excellent! All export tests pass. Now let me run all the eval-related tests to make sure nothing else is broken:

  67. Qwen Assistant

    These test failures appear to be related to database setup issues, not to my changes. Let me focus on running the tests that are specifically related to the route and schema validation. Let me also run the main eval tests:

  68. Qwen Assistant

    Let me verify the final state of the changes I made to the types file:

  69. Qwen Assistant

    The changes look good. Now let me verify the route file to see if any updates are needed there. The route already handles the format check by using UNLIMITED_RESULTS for exports and baseLimit for non-exports. Since the schema now validates that non-export limits are <= 1000, the baseLimit will never exceed the cap. However, it would be cleaner to update the comment or add the constant to the route as well.

    Let me check the route file to see if any updates are needed:

    <tool_call> <function=bash_terminal> <parameter=command> sed -n '305,345p' /promptfoo/src/server/routes/eval.ts </tool_call>

  70. Qwen Assistant

    The command is not being executed properly. Let me try again with the bash_terminal tool.

  71. Qwen Assistant

    Now I can see the route code. The route already handles the format check correctly:

    • For exports (format is present): uses UNLIMITED_RESULTS (which is Number.MAX_SAFE_INTEGER)
    • For non-exports (format is not present): uses baseLimit, which is now validated by the schema to be <= 1000

    However, I should check if the schema validation happens before the format check. Let me look at the order of operations in the route:

    const queryResult = EvalSchemas.Table.Query.safeParse(req.query);
    ...
    const limit = format ? UNLIMITED_RESULTS : baseLimit;
    

    Yes, the schema validation happens first, and then the limit is adjusted based on format. For non-export requests, if the limit is > 1000, the schema validation will fail and return a 400 error. For export requests, the schema validation will accept any positive integer limit, and then the route will use UNLIMITED_RESULTS anyway.

    However, I realize there might be an issue: if a user sends an export request with a limit > 1000, the schema will pass (because format is present), but then the route will use UNLIMITED_RESULTS anyway, so the limit value doesn't matter for exports. This is fine.

    But wait, I need to check if the schema validation is happening correctly. The refine function checks:

    if (!data.format && data.limit > EVAL_TABLE_MAX_PAGE_SIZE) {
      return false;
    }
    

    This means:

    • If format is not present AND limit > 1000, return false (validation fails)
    • If format is present, the limit check is skipped (validation passes)
    • If format is not present AND limit <= 1000, validation passes

    This is exactly what we want!

    Now let me make sure there are no other issues. Let me check if there are any other places where the limit is used or validated.

    Actually, I just realized that the route might need to be updated to use the constant instead of the hardcoded UNLIMITED_RESULTS. Let me check if this is necessary.

    Looking at the code, UNLIMITED_RESULTS is used for exports, and baseLimit is used for non-exports. The schema validation ensures that baseLimit is <= 1000 for non-exports. So the route doesn't need to change.

    However, to be consistent and follow best practices, I should update the route to use the constant EVAL_TABLE_MAX_PAGE_SIZE instead of the magic number 1000 in the schema. But wait, the constant is already defined and used in the schema, so no change is needed.

    Actually, looking at the issue description again, it mentions:

    • "Return validation errors for excessive non-export limits, consistent with nearby API schema validation."

    The validation is now done in the schema, which is consistent with other API validations. The route will automatically return a 400 error for excessive non-export limits.

    Let me now create a simple test to verify that the route properly rejects excessive limits for non-export requests:

    <tool_call> <function=bash_terminal> <parameter=command> cd /promptfoo && cat > /tmp/route_test.test.ts << 'EOF' import type { Server } from 'node:http'; import request from 'supertest'; import { afterAll, beforeAll, describe, expect, it, vi } from 'vitest';

    // Mock dependencies vi.mock('../../src/models/eval');

    import Eval from '../../src/models/eval'; import { createApp } from '../../src/server/server';

    const mockedEval = vi.mocked(Eval);

    describe('Eval Routes - Limit Validation', () => { let api: ReturnType; let server: Server; let mockFindById: ReturnType;

    beforeAll(async () => { await new Promise((resolve, reject) => { server = createApp().listen(0, '127.0.0.1', (error?: Error) => error ? reject(error) : resolve(), ); }); api = request.agent(server); });

    afterAll(async () => { if (!server.listening) { return; } await new Promise((resolve, reject) => { server.close((error) => (error ? reject(error) : resolve())); }); });

    beforeEach(() => { vi.resetAllMocks(); mockFindById = vi.fn(); mockedEval.findById = mockFindById as any; });

    it('should return 400 for non-export with limit > 1000', async () => { const mockEval = { id: 'test-id', }; mockFindById.mockResolvedValue(mockEval);

    const response = await api.get('/api/eval/test-id/table').query({ limit: '1001' });
    
    expect(response.status).toBe(400);
    expect(response.body.error).toBeDefined();
    expect(response.body.error).toContain('limit');
    expect(mockFindById).not.toHaveBeenCalled();
    

    });

    it('should return 400 for non-export with very large limit', async () => { const mockEval = { id: 'test-id', }; mockFindById.mockResolvedValue(mockEval);

    const response = await api.get('/api/eval/test-id/table').query({ limit: '10000' });
    
    expect(response.status).toBe(400);
    expect(response.body.error).toBeDefined();
    expect(response.body.error).toContain('limit');
    expect(mockFindById).not.toHaveBeenCalled();
    

    });

    it('should accept non-export with limit = 1000', async () => { const mockEval = { id: 'test-id', }; mockFindById.mockResolvedValue(mockEval);

    const response = await api.get('/api/eval/test-id/table').query({ limit: '1000' });
    
    // This should not be a 400 error (schema validation passes)
    expect(response.status).not.toBe(400);
    

    });

    it('should accept export with very large limit', async () => { const mockEval = { id: 'test-id', }; mockFindById.mockResolvedValue(mockEval);

    const response = await api.get('/api/eval/test-id/table').query({ 
      format: 'csv', 
      limit: '10000' 
    });
    
    // This should not be a 400 error (schema validation passes for exports)
    expect(response.status).not.toBe(400);
    

    }); }); EOF npm test -- /tmp/route_test.test.ts 2>&1 | tail -30 </tool_call>

  72. Qwen Assistant

    Let me check the test file path - it needs to be in the test directory. Let me move it there and run the test.

  73. Qwen Assistant

    The file didn't get created. Let me try again with a simpler approach - just directly create the file in the test directory.

    <tool_call> <function=bash_terminal> <parameter=command> cat > /promptfoo/test/server/routes/limit_validation.test.ts << 'EOF' import type { Server } from 'node:http'; import request from 'supertest'; import { afterAll, beforeAll, describe, expect, it, vi } from 'vitest';

    vi.mock('../../src/models/eval');

    import Eval from '../../src/models/eval'; import { createApp } from '../../src/server/server';

    const mockedEval = vi.mocked(Eval);

    describe('Eval Routes - Limit Validation', () => { let api: ReturnType; let server: Server; let mockFindById: ReturnType;

    beforeAll(async () => { await new Promise((resolve, reject) => { server = createApp().listen(0, '127.0.0.1', (error?: Error) => error ? reject(error) : resolve(), ); }); api = request.agent(server); });

    afterAll(async () => { if (!server.listening) { return; } await new Promise((resolve, reject) => { server.close((error) => (error ? reject(error) : resolve())); }); });

    beforeEach(() => { vi.resetAllMocks(); mockFindById = vi.fn(); mockedEval.findById = mockFindById as any; });

    it('should return 400 for non-export with limit > 1000', async () => { const mockEval = { id: 'test-id' }; mockFindById.mockResolvedValue(mockEval);

    const response = await api.get('/api/eval/test-id/table').query({ limit: '1001' });
    
    expect(response.status).toBe(400);
    expect(response.body.error).toBeDefined();
    expect(response.body.error).toContain('limit');
    expect(mockFindById).not.toHaveBeenCalled();
    

    });

    it('should return 400 for non-export with very large limit', async () => { const mockEval = { id: 'test-id' }; mockFindById.mockResolvedValue(mockEval);

    const response = await api.get('/api/eval/test-id/table').query({ limit: '10000' });
    
    expect(response.status).toBe(400);
    expect(response.body.error).toBeDefined();
    expect(response.body.error).toContain('limit');
    expect(mockFindById).not.toHaveBeenCalled();
    

    });

    it('should accept non-export with limit = 1000', async () => { const mockEval = { id: 'test-id' }; mockFindById.mockResolvedValue(mockEval);

    const response = await api.get('/api/eval/test-id/table').query({ limit: '1000' });
    
    expect(response.status).not.toBe(400);
    

    });

    it('should accept export with very large limit', async () => { const mockEval = { id: 'test-id' }; mockFindById.mockResolvedValue(mockEval);

    const response = await api.get('/api/eval/test-id/table').query({ format: 'csv', limit: '10000' });
    
    expect(response.status).not.toBe(400);
    

    }); }); EOF

  74. Qwen Assistant

    The command is still not being executed. Let me check if there's an issue with the command. Let me try a different approach - using echo and redirecting to a file.

  75. Qwen Assistant

    Now let me run the test to verify the implementation:

  76. Qwen Assistant

    I see there's a test file that passed (test_limit_validation.test.ts). Let me remove the failed test file and verify everything is working:

  77. Qwen Assistant

    Now let me check if there's a test_limit_validation.test.ts file and remove it as well (since it was a temporary test file):

  78. Qwen Assistant

    I see there are some temporary test files that I created. Let me remove them:

  79. Qwen Assistant

    Now let me run the core eval validation and export tests one more time to make sure everything is working:

  80. Qwen Assistant

    Let me also verify the export tests pass:

    <tool_call> <function=bash_terminal> <parameter=command> cd /promptfoo && npm test -- eval.export.test.ts 2>&1 | grep -A 5 "Test Files" </tool_call>

  81. Qwen Assistant

    The command didn't execute properly. Let me try again with the correct formatting.

  82. Qwen Assistant

    Great! Both validation and export tests pass. Now let me verify the changes I made by reviewing the final state of the types file:

    1. Added EVAL_TABLE_MAX_PAGE_SIZE = 1000 constant
    2. Modified EvalTableQuerySchema to add conditional validation using .refine()

    Let me also check if I need to export the constant in the index file or anywhere else:

    <tool_call> <function=bash_terminal> <parameter=command> grep -n "export.*from.*api/eval" /promptfoo/src/types/index.ts 2>/dev/null | head -10 </tool_call>

  83. Qwen Assistant

    The commands are not being executed. Let me check the index.ts file to see if eval exports are re-exported:

    <tool_call> <function=bash_terminal> <parameter=command> grep -n "api/eval" /promptfoo/src/types/index.ts </tool_call>

  84. Qwen Assistant

    The commands are not executing. Let me try a different approach - maybe there's a shell issue. Let me try with a simple command first to see if the terminal is working.

  85. Qwen Assistant

    The grep command returned exit code 1 because there's no match. Let me check the index.ts file to see what's exported from api/eval:

    <tool_call> <function=bash_terminal> <parameter=command> head -50 /promptfoo/src/types/index.ts </tool_call>

  86. Qwen Assistant

    Let me verify the final state of the changes by checking what files were modified:

  87. Qwen Assistant

    I apologize, but it seems like my commands are not being executed properly in this session. Let me try a different approach to check the changes.

    <tool_call> <function=bash_terminal> <parameter=command> git -C /promptfoo diff src/types/api/eval.ts </tool_call>

  88. Qwen Assistant

    The commands are still not being executed. Let me check if there's an issue with the shell or if I need to wait for the previous command to complete. Let me try a simpler command to see if the shell is responsive.

  89. Qwen Assistant

    This worked. The file exists and has 408 lines. Let me check the git diff now.

    <tool_call> <function=bash_terminal> <parameter=command> git -C /promptfoo diff src/types/api/eval.ts </tool_call>

  90. Qwen Assistant

    I apologize for the confusion. The commands are not executing properly. Let me try a different approach - I'll use the special commands available in the environment to check the changes.

    Actually, looking at the previous successful command, I can see that simple commands work. The issue might be with the git command or the way the shell handles multi-line commands. Let me try running the git diff command with a simpler syntax.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.