Skip to content
TheChatArchiveThe Chat Archive
DeepSeekdeepseek/deepseek-v4-pro· deepseekHugging Face ibragim-bad/swe_rebench_07_2026_trajectories

ISSUE DESCRIPTION generator: typed-table emit exceeds SQLite column cap on wide config…

  1. User

    ISSUE DESCRIPTION

    generator: typed-table emit exceeds SQLite column cap on wide config singletons

    Filed by /printing-press-retro from a run of HaloPSA (scorecard 91/100 Grade A, 100% verify pass).

    Summary

    The typed-table emit path generates one column per leaf JSON field in a resource's response schema with no upper bound. On HaloPSA's tenant-global /Control config singleton, this produced a 3,747-column CREATE TABLE that exceeded SQLite's default SQLITE_MAX_COLUMN=2000. Generation passed every quality gate; sync --full --dry-run died cold with SQL logic error: too many columns on control (1). The failure mode is silent until first sync.

    Where to look

    • Component: generator
    • Likely area: Typed-table-emit path in internal/generator/templates/store/ — wherever CREATE TABLE text is composed per discovered resource.
    • Triggered when: A spec resource's response schema flattens to >~1500 leaf fields. Enterprise config singletons are the obvious class.

    What we observed

    Run-1 output (post-generate):

    PASS go mod tidy / govulncheck / go vet / go build
    PASS halopsa-pp-cli --help / version / doctor
    Generated halopsa at .../working/halopsa-pp-cli
    

    First sync --full --dry-run:

    Error: opening local database: running migrations:
    migration failed: SQL logic error: too many columns on control (1)
    

    Per-table column counts (top 5 widest, recorded by Python introspection of internal/store/store.go):

    tablecolumns
    control3747
    tickets847
    projects847
    opportunities847
    client447

    Tickets / projects / opportunities sit just under the 2000 cap but with no headroom; the next major-version Halo response schema could push them over.

    Workaround applied this run: hand-stripped /Control* paths from the spec (5 paths) and added a stub Control schema so residual $ref targets resolved. Regenerated successfully.

    Suspected root cause

    Confident. The typed-table-emit path walks the resource's response schema and emits one column per leaf field with no width sanity check. The store package has no JSON-only fallback for resources whose schema exceeds a safe column count.

    Suggested direction

    In the typed-table-emit path, compute the column count before writing CREATE TABLE. If the count exceeds a threshold (1500 leaves headroom under SQLite's default 2000; tunable), fall back to JSON-only schema: id TEXT PRIMARY KEY, data JSON NOT NULL, synced_at DATETIME. Emit a generation-time log line naming the resource and the fallback (e.g., store-fallback: control (3747 cols) → JSON-only).

    Preserve typed-column behavior below the threshold — the typed columns power offline FTS5 and cross-entity SQL joins; the fallback should only fire on pathological widths.

    Acceptance criteria

    • positive: feed a synthetic spec where one resource has 3000+ leaf fields; generated store.go for that resource has only 3 columns; sync against an empty DB succeeds.
    • negative: feed a normal resource (<1500 cols); typed-table emission unchanged; flat columns still emitted.

    Frequency

    subclass:wide-config-singleton — complete-functional-failure when it fires.

    Complexity

    small (column-count check + JSON-only emit branch).

    Scope boundary

    Does NOT cover per-column truncation, schema-aware projection, or per-API column allowlists. Single threshold + JSON-only fallback only.

    Dependencies

    None.

    Related issues

    None — first sighting of this specific failure mode.

    Artifacts

    ArtifactLink
    Retro documenthttps://files.catbox.moe/ico9uo.md
    Manuscripts (research + proofs)https://files.catbox.moe/rn4hf4.zip
    Generated CLI source codehttps://files.catbox.moe/4v6myv.zip

    Generated by /printing-press-retro · CLI Printing Press

    Evaluation contract

    Use 1500 columns as the safety cap for a generated typed SQLite resource table, including dependent-parent columns. Resources exceeding the cap must retain their per-resource table but fall back to a JSON-only schema containing id, data, and synced_at; do not generate typed-column indexes, FTS structures, or typed upsert code for that fallback table. Batch store generation must dispatch wide resources to the JSON-only path while preserving the existing typed behavior for ordinary narrow resources.

    Interface

    No new interfaces are introduced.

    ADDITIONAL ADVICE

    Since you are given a git repository, you can use git commands to simplify your work. For example, if you made a mistake and want to revert your changes, you can use git checkout <file> to restore the file to its original state. You can also reset all changes in the repository using git reset --hard command. Additionally, you can use git stash and git stash pop to temporarily save your changes and restore them later. Finally, you can see the changes that you've made (which will be submitted when you call submit) using git status or git diff commands. However you don't need to use git add or git commit before submitting your changes. In fact, submit only submits currently unstaged changes because it uses git diff to get the changes that need to be submitted.

    CHECKLIST

    Before submitting your solution, please go over the following checklist and make sure you've done everything:

    • If an error was reported in the issue description, I have successfully reproduced it.
    • If an error was reported in the issue description, I have confirmed that my fix resolves the error.
    • I have thought about edge cases and made sure my fix handles them as well.
    • I have run existing tests in the repository that might have been affected by the change I made and confirmed that they pass. I want you to list every bullet from this checklist and write your reasoning for why you think you did it or didn't need to.

    Repository has been uploaded and your shell is currently at the repository root. Time to solve the issue!

    (Current directory: /cli-printing-press, current file: none) bash-$

  2. DeepSeek Assistant

    Let me start by understanding the repository structure and the issue at hand.

    The issue is about a "typed-table emit path" in a generator that creates SQLite tables with one column per leaf JSON field. When a resource has too many leaf fields (>1500), it exceeds SQLite's column limit (2000). The fix should:

    1. Compute column count before writing CREATE TABLE
    2. If count exceeds 1500, fall back to JSON-only schema: id TEXT PRIMARY KEY, data JSON NOT NULL, synced_at DATETIME
    3. Emit a log line when falling back
    4. Keep typed columns for resources below the threshold

    Let me explore the repository structure first

  3. DeepSeek Assistant

    Let me explore the repository structure more carefully, focusing on the generator part.

  4. DeepSeek Assistant

    Let me explore the generator's template and store-related files:

  5. DeepSeek Assistant

    Found the store template. Let me examine it and the surrounding code

  6. DeepSeek Assistant

    Let me look at the store template and the generator code that uses it:

  7. DeepSeek Assistant

    Let me look at the CREATE TABLE section and how columns are built

  8. DeepSeek Assistant

    Now let me find the Go code that drives this template — where Tables and their Columns are constructed:

  9. DeepSeek Assistant

    Let me search more broadly for the type definitions for Table.

  10. DeepSeek Assistant

    Let me search for the template data structures.

  11. DeepSeek Assistant

    Let me look at the generator.go to understand how the store template data is built:

  12. DeepSeek Assistant

    Let me find the TableDef type definition and the `hasTypedTable

  13. DeepSeek Assistant

    The types are probably defined elsewhere. Let me search more broadly.

  14. DeepSeek Assistant

    Let me look at the schema_builder.go

  15. DeepSeek Assistant

    Let me see the rest of schema_builder.go and find `hasTypedTable

  16. DeepSeek Assistant

    Let me find the hasTypedTable function definition in generator.go

  17. DeepSeek Assistant

    Let me see the full hasTypedTable function.

  18. DeepSeek Assistant

    Now let me look at schemaWithDependentParents to understand how dependent-parent columns are added (they could push a table over the limit):

  19. DeepSeek Assistant

    Let me look at more of the store template to understand the full picture for non-typed tables

  20. DeepSeek Assistant

    Let me look at the sections around those gates

  21. DeepSeek Assistant

    Let me look at lines around 930-1000 in the store template

  22. DeepSeek Assistant

    Let me see the template section around the CREATE TABLE more carefully and understand the full structure

  23. DeepSeek Assistant

    Now I have a clear understanding of the architecture. Let me look at the existing tests to understand the patterns

  24. DeepSeek Assistant

    Let me look at buildSubResourceTable in schema_builder.go

  25. DeepSeek Assistant

    Let me look at the buildSubResourceTable function.

  26. DeepSeek Assistant

    Let me check how logging is done in the generator

  27. DeepSeek Assistant

    Let me check the logger used in the generator and main entry point

  28. DeepSeek Assistant

    Let me check how the main pipeline calls the generator and handles

  29. DeepSeek Assistant

    Let me look at how the generate pipeline logs output

  30. DeepSeek Assistant

    Let me check how the pipeline uses the generator and where output happens

  31. DeepSeek Assistant

    Let me find how the generate pipeline produces

  32. DeepSeek Assistant

    Let me look at plan_generate.go to understand how the generator outputs messages

  33. DeepSeek Assistant

    Let me find where the generator's output is controlled and look at how to add a log line

  34. DeepSeek Assistant

    Now let me look at collectResponseFields to understand field

  35. DeepSeek Assistant

    Let me see the collectResponseFields function:

  36. DeepSeek Assistant

    Now I have a thorough understanding. Let me look at how the template function map is built in

  37. DeepSeek Assistant

    Let me see the FuncMap definition

  38. DeepSeek Assistant

    Let me check the existing golden store tests to understand what the expected output looks like

  39. DeepSeek Assistant

    Let me look at the schema_builder test and store-related

  40. DeepSeek Assistant

    Now let me now implement the changes. First, let me add the changes to schema_builder.go:

    1. Add constant and Fallback field
    2. Add column count check in BuildSchema
    3. Also check in schemaWithDependentParents

    Then modify the template and generator.

    Let me start implementing.Now I have a comprehensive understanding. Let me implement the changes step by step.

    Step 1: Add Fallback field to TableDef and the column cap constant in `schema_builder

  41. DeepSeek Assistant

    The replace didn't work because of whitespace differences. Let me go to that line and use edit.

  42. DeepSeek Assistant

    Let me edit the file to add the constant and Fallback field, and the column cap

  43. DeepSeek Assistant

    Let me look at the current state of the file

  44. DeepSeek Assistant

    Let me see the current state of the file around the gravity area

  45. DeepSeek Assistant

    Let me add fmt and os to the imports

  46. DeepSeek Assistant

    I see a duplicate ) at line 11. Let me fix

  47. DeepSeek Assistant

    Now let me add the column cap check. Let me see the area after the gravity block

  48. DeepSeek Assistant

    Now let me add the column cap check after the gravity block

  49. DeepSeek Assistant

    Now let me add the column cap check in schemaWithDependentParents in generator.go

  50. DeepSeek Assistant

    Now let me add the column cap check in schemaWithDependentParents

  51. DeepSeek Assistant

    Let me check the indentation around the edit area

  52. DeepSeek Assistant

    Let me verify the indentation and formatting

  53. DeepSeek Assistant

    The edit accidentally removed a closing brace. Let me fix

  54. DeepSeek Assistant

    Now I need to modify the store template to handle fallback

  55. DeepSeek Assistant

    Now let me modify the store template to add the fallback table branch

  56. DeepSeek Assistant

    Let me verify the changes compile and check the backfillColumns section

  57. DeepSeek Assistant

    Let me check the build environment

  58. DeepSeek Assistant

    Let me set up the Go environment

  59. DeepSeek Assistant

    Now let me try building again

  60. DeepSeek Assistant

    Let me set up the full Go

  61. DeepSeek Assistant

    Good, it compiles. Let me run the existing tests for the generator

  62. DeepSeek Assistant

    All BuildSchema tests pass. Let me run the broader generator tests

  63. DeepSeek Assistant

    Let me look at the existing golden store.go to understand the expected format

  64. DeepSeek Assistant

    Let me run a single failing test with more output to see the actual

  65. DeepSeek Assistant

    Let me look at the existing golden store.go to understand the structure around that line

  66. DeepSeek Assistant

    Let me verify the braces in generator.go are correct by looking at the full function

  67. DeepSeek Assistant

    Let me write a quick test to see what the template generates with a fallback

  68. DeepSeek Assistant

    Let me find and look at the generated store.go from the test

  69. DeepSeek Assistant

    Let me look at the test helper to understand the output directory

  70. DeepSeek Assistant

    Let me check the template diff more

  71. DeepSeek Assistant

    I found the bug! The closing } for the migrations slice was accidentally removed. Let me fix

  72. DeepSeek Assistant

    Let me add back the closing } for the migrations slice

  73. DeepSeek Assistant

    Now let me run the tests again

  74. DeepSeek Assistant

    Tests pass. Let me run the full generator test suite

  75. DeepSeek Assistant

    The failing test is a govulncheck vulnerability scan unrelated to our changes. Let me verify by running the schema builder and store-related tests

  76. DeepSeek Assistant

    Let me run specific tests

  77. DeepSeek Assistant

    Let me run a broader set of tests to confirm nothing is broken

  78. DeepSeek Assistant

    All tests pass. Let me now test the fallback mechanism with a synthetic spec. Let me create a temporary script

  79. DeepSeek Assistant

    Let me write a temporary test inside the module instead

  80. DeepSeek Assistant

    Let me find the correct type name

  81. DeepSeek Assistant

    Let me look at the Endpoint type

  82. DeepSeek Assistant

    Let me look at the ResponseDef struct

  83. DeepSeek Assistant

    Let me fix and run the

  84. DeepSeek Assistant

    Both tests pass. Let me clean up the temporary test file and run the full suite again

  85. DeepSeek Assistant

    Let me run the full test suite again (excluding the unrelated vulncheck test

  86. DeepSeek Assistant

    I found an edge case: when a fallback table gets parent_id added, hasTypedTable could still return true. Let me update hasTypedTable to exclude fallback tables

  87. DeepSeek Assistant

    Let me update hasTypedTable to exclude fallback tables

  88. DeepSeek Assistant

    Let me use replace

  89. DeepSeek Assistant

    Now let me rebuild and run the tests again

  90. DeepSeek Assistant

    Let me run the tests again

  91. DeepSeek Assistant

    Let me run a broader set of tests one more time

  92. DeepSeek Assistant

    Everything works. Let me now review all the changes made

  93. DeepSeek Assistant

    Let me review all diffs in detail

  94. DeepSeek Assistant

    Let me verify the backfillColumns section handles fallback tables

  95. DeepSeek Assistant

    Let me check isStoreBackfillColumn

  96. DeepSeek Assistant

    Let me check how the backfill section is gated in the template

  97. DeepSeek Assistant

    Let me check the backfill section in the template

  98. DeepSeek Assistant

    Everything looks correct. Let me run a final comprehensive test pass

  99. DeepSeek Assistant

    Let me check how the per-resource tables are used in queries

  100. DeepSeek Assistant

    Let me check the context around line 1038

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.