Skip to content
TheChatArchiveThe Chat Archive
QwenQwen/Qwen3.6-27B· alibabaHugging Face ibragim-bad/swe_rebench_07_2026_trajectories

ISSUE DESCRIPTION Cap eval table page size for non-export requests Summary GET…

  1. User

    ISSUE DESCRIPTION

    Cap eval table page size for non-export requests

    Summary

    GET /api/eval/:id/table accepts any positive integer for limit on normal table requests. The route already intentionally treats export requests differently by using Number.MAX_SAFE_INTEGER when format is set, but non-export page requests currently do not have an upper bound.

    Why this matters

    A very large limit can make a normal table request ask the server to load and serialize a large amount of eval result data in one response. Other list-style APIs in the server already bound their page sizes, so the eval table endpoint should follow the same resource-control pattern.

    Proposed change

    • Add a maximum limit for non-export GET /api/eval/:id/table requests.
    • Keep format=csv / format=json exports unchanged so full-table exports still work as intended.
    • Return validation errors for excessive non-export limits, consistent with nearby API schema validation.

    A cap of 500 or 1000 would likely cover normal UI usage while preventing accidental oversized page requests.

    Validation

    Suggested tests:

    • schema test rejects EvalSchemas.Table.Query.safeParse({ limit: '<cap + 1>' })
    • route validation test returns 400 for excessive non-export limit
    • export route tests continue to use Number.MAX_SAFE_INTEGER when format is present

    Interface

    Type: Constant Name: EVAL_TABLE_MAX_PAGE_SIZE Path: src/types/api/eval.ts Value/Type: constant literal value 1000 (number) Description: Server-enforced upper bound for the limit parameter on non-export eval table page requests. Exports using format (csv/json) are exempt from this cap; UI pagination controls derive their available page-size options from this value.

    ADDITIONAL ADVICE

    Since you are given a git repository, you can use git commands to simplify your work. For example, if you made a mistake and want to revert your changes, you can use git checkout <file> to restore the file to its original state. You can also reset all changes in the repository using git reset --hard command. Additionally, you can use git stash and git stash pop to temporarily save your changes and restore them later. Finally, you can see the changes that you've made (which will be submitted when you call submit) using git status or git diff commands. However you don't need to use git add or git commit before submitting your changes. In fact, submit only submits currently unstaged changes because it uses git diff to get the changes that need to be submitted.

    CHECKLIST

    Before submitting your solution, please go over the following checklist and make sure you've done everything:

    • If an error was reported in the issue description, I have successfully reproduced it.
    • If an error was reported in the issue description, I have confirmed that my fix resolves the error.
    • I have thought about edge cases and made sure my fix handles them as well.
    • I have run existing tests in the repository that might have been affected by the change I made and confirmed that they pass. I want you to list every bullet from this checklist and write your reasoning for why you think you did it or didn't need to.

    Repository has been uploaded and your shell is currently at the repository root. Time to solve the issue!

    (Current directory: /promptfoo, current file: none) bash-$

  2. Qwen Assistant

    Let me start by understanding the issue. The task is to:

    1. Add a maximum limit for non-export GET /api/eval/:id/table requests
    2. Keep format=csv / format=json exports unchanged
    3. Return validation errors for excessive non-export limits
    4. Define a constant EVAL_TABLE_MAX_PAGE_SIZE with value 1000 in src/types/api/eval.ts

    Let me first explore the repository structure to understand the codebase.

  3. Qwen Assistant

    Let me explore the repository structure more, specifically looking for the eval-related files mentioned in the issue.

  4. Qwen Assistant

    I need to use the open command differently - it's a special command, not a shell command.

  5. Qwen Assistant

    I need to use the special open command properly. Let me try with just the path as an argument.

  6. Qwen Assistant

    Now let me find the route handler for GET /api/eval/:id/table to understand how the limit is currently handled.

  7. Qwen Assistant

    Let me see the full file, especially the GET /:id/table route:

  8. Qwen Assistant

    Let me find where UNLIMITED_RESULTS is defined:

  9. Qwen Assistant

    Now I have a clear picture of the code. Let me also check the existing tests to understand the testing patterns:

  10. Qwen Assistant

    Now let me also check the export test file to understand how exports are tested:

  11. Qwen Assistant

    Now I have a good understanding of the codebase. Let me plan my changes:

    1. Add EVAL_TABLE_MAX_PAGE_SIZE = 1000 constant to src/types/api/eval.ts
    2. Update EvalTableQuerySchema to add .max(EVAL_TABLE_MAX_PAGE_SIZE) on the limit field
    3. Update the route handler in src/server/routes/eval.ts to handle exports differently - exports should use a permissive schema that doesn't cap the limit

    Let me first look at the exact lines I need to modify:

  12. Qwen Assistant

    Now let me look at how the schema is used in the route and how we need to handle the export case:

  13. Qwen Assistant

    Now I understand the issue better. The current code validates the query once with EvalSchemas.Table.Query.safeParse(req.query). Then, if format is present, it overrides the limit with UNLIMITED_RESULTS.

    The issue is that we need to reject requests where limit > 1000 for non-export requests, but still allow any limit value when format is specified (since the route overrides it anyway).

    There are two approaches:

    1. Have two different schemas - one for exports and one for non-exports
    2. Add the max validation in the route after schema parsing, only when format is not set

    Looking at the issue description, it says:

    • "schema test rejects EvalSchemas.Table.Query.safeParse({ limit: '<cap + 1>' })"
    • This suggests we should modify the schema itself

    But if we modify the schema to reject limits > 1000, then export requests would also fail validation. So we need to have separate schemas for export vs non-export, or we need to handle this differently.

    Let me think about this more carefully. The issue says:

    1. Add a constant EVAL_TABLE_MAX_PAGE_SIZE with value 1000
    2. Schema test rejects EvalSchemas.Table.Query.safeParse({ limit: '<cap + 1>' })
    3. Route validation test returns 400 for excessive non-export limit
    4. Export route tests continue to use Number.MAX_SAFE_INTEGER when format is present

    The approach should be:

    1. Define the constant in src/types/api/eval.ts
    2. Modify the schema to add a max, but also create an export-specific schema (or modify the query handling to be permissive for exports)

    Actually, looking at the test suggestion more carefully - it says the schema should reject limit > cap + 1. This means we should cap it in the schema. But for exports, we need to use a different, permissive schema.

    Let me implement this:

  14. Qwen Assistant

    Now let me implement the changes. First, I'll add the constant and modify the schema in src/types/api/eval.ts:

  15. Qwen Assistant

    Now I'll make the changes. I need to:

    1. Add the EVAL_TABLE_MAX_PAGE_SIZE constant
    2. Modify the limit field in EvalTableQuerySchema to add a max validation

    Let me add the constant right before the schema (around line 83-85) and update the limit field:

  16. Qwen Assistant

    I need to:

    1. Add the EVAL_TABLE_MAX_PAGE_SIZE constant after line 83 (after the comment)
    2. Update line 88 to add .max(EVAL_TABLE_MAX_PAGE_SIZE) to the limit field

    Let me use sed to make these changes.

  17. Qwen Assistant

    Good. Now I need to update the limit field to add the .max(EVAL_TABLE_MAX_PAGE_SIZE) constraint:

  18. Qwen Assistant

    Let me verify the change:

  19. Qwen Assistant

    Now I need to update the route handler to handle exports differently. Currently, the route uses the same schema for both export and non-export requests. But now that the schema rejects limits > 1000, export requests with high limits will fail validation.

    Looking at the route code again:

    const queryResult = EvalSchemas.Table.Query.safeParse(req.query);
    if (!queryResult.success) {
      res.status(400).json({ error: z.prettifyError(queryResult.error) });
      return;
    }
    
    // ...later...
    // Apply UNLIMITED_RESULTS when format is specified
    const limit = format ? UNLIMITED_RESULTS : baseLimit;
    

    The issue is that if an export request has limit=10000, it will fail schema validation before we can override it. We need to:

    1. Create a separate permissive schema for exports, OR
    2. Check if format is present first and use a different validation path

    Looking at the issue description, it says "export route tests continue to use Number.MAX_SAFE_INTEGER when format is present". This suggests exports should use a different schema or validation path.

    The cleanest approach is to:

    1. If format is present in query, use a permissive query schema that doesn't cap the limit
    2. If format is not present, use the capped schema

    Let me create an export-specific schema and update the route handler:

  20. Qwen Assistant

    Now I need to add an export-specific schema after the regular one. Let me add it right after EvalTableQuerySchema:

  21. Qwen Assistant

    Good. Now I also need to add this export schema to the EvalSchemas object. Let me find where that is:

  22. Qwen Assistant

    I need to add the ExportQuery to the Table section:

  23. Qwen Assistant

    The sed didn't work as expected because of newline issues. Let me try a different approach:

  24. Qwen Assistant

    Now I need to update the route handler to use the export schema when format is present. Let me look at the route handler:

  25. Qwen Assistant

    I need to modify the route to:

    1. Check if format is present in the query first
    2. If format is present, use the export schema (which doesn't cap limit)
    3. If format is not present, use the regular schema (which caps limit)

    Let me update the route handler:

  26. Qwen Assistant

    I need to use a different approach since python3 is not available. Let me use sed or create the replacement using a different method.

  27. Qwen Assistant

    Good, I can use node:

  28. Qwen Assistant

    Let me verify the change:

  29. Qwen Assistant

    Now let me verify that the code compiles correctly and run the existing tests:

  30. Qwen Assistant

    The build succeeded. Now let me run the existing tests to make sure they pass:

  31. Qwen Assistant

    All validation tests pass. Now let me run the export tests:

  32. Qwen Assistant

    All tests pass. Now let me create a test script to verify my changes work as expected:

  33. Qwen Assistant

    I need to run the test from the correct directory:

  34. Qwen Assistant

    All tests pass. Now let me run more of the existing tests related to eval routes to make sure I haven't broken anything:

  35. Qwen Assistant

    The test failure is due to missing native modules (better-sqlite3), not my changes. Let me run the other tests that use mocking:

  36. Qwen Assistant

    The failed tests are due to missing native sqlite bindings, not related to my changes. All 13 test files with mocking passed, including the eval validation and export tests. Let me run the specific tests for eval routes that are relevant:

  37. Qwen Assistant

    Excellent! All relevant tests pass. Now let me also verify the TypeScript types are correct by running type checking:

  38. Qwen Assistant

    TypeScript compilation is successful. Now let me verify my changes by reviewing the final state of the files:

  39. Qwen Assistant

    Let me also check that the constant is exported properly and can be imported by other modules:

    <tool_call> <function=bash_terminal> <parameter=command> cd /promptfoo && grep -n "EVAL_TABLE_MAX_PAGE_SIZE" src/types/api/eval.ts </tool_call>

  40. Qwen Assistant

    Let me run the grep command properly.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.