Skip to content
TheChatArchiveThe Chat Archive
GLMz-ai/glm-5.2· unknownHugging Face ibragim-bad/swe_rebench_07_2026_trajectories

ISSUE DESCRIPTION generator: typed-table emit exceeds SQLite column cap on wide config…

  1. User

    ISSUE DESCRIPTION

    generator: typed-table emit exceeds SQLite column cap on wide config singletons

    Filed by /printing-press-retro from a run of HaloPSA (scorecard 91/100 Grade A, 100% verify pass).

    Summary

    The typed-table emit path generates one column per leaf JSON field in a resource's response schema with no upper bound. On HaloPSA's tenant-global /Control config singleton, this produced a 3,747-column CREATE TABLE that exceeded SQLite's default SQLITE_MAX_COLUMN=2000. Generation passed every quality gate; sync --full --dry-run died cold with SQL logic error: too many columns on control (1). The failure mode is silent until first sync.

    Where to look

    • Component: generator
    • Likely area: Typed-table-emit path in internal/generator/templates/store/ — wherever CREATE TABLE text is composed per discovered resource.
    • Triggered when: A spec resource's response schema flattens to >~1500 leaf fields. Enterprise config singletons are the obvious class.

    What we observed

    Run-1 output (post-generate):

    PASS go mod tidy / govulncheck / go vet / go build
    PASS halopsa-pp-cli --help / version / doctor
    Generated halopsa at .../working/halopsa-pp-cli
    

    First sync --full --dry-run:

    Error: opening local database: running migrations:
    migration failed: SQL logic error: too many columns on control (1)
    

    Per-table column counts (top 5 widest, recorded by Python introspection of internal/store/store.go):

    tablecolumns
    control3747
    tickets847
    projects847
    opportunities847
    client447

    Tickets / projects / opportunities sit just under the 2000 cap but with no headroom; the next major-version Halo response schema could push them over.

    Workaround applied this run: hand-stripped /Control* paths from the spec (5 paths) and added a stub Control schema so residual $ref targets resolved. Regenerated successfully.

    Suspected root cause

    Confident. The typed-table-emit path walks the resource's response schema and emits one column per leaf field with no width sanity check. The store package has no JSON-only fallback for resources whose schema exceeds a safe column count.

    Suggested direction

    In the typed-table-emit path, compute the column count before writing CREATE TABLE. If the count exceeds a threshold (1500 leaves headroom under SQLite's default 2000; tunable), fall back to JSON-only schema: id TEXT PRIMARY KEY, data JSON NOT NULL, synced_at DATETIME. Emit a generation-time log line naming the resource and the fallback (e.g., store-fallback: control (3747 cols) → JSON-only).

    Preserve typed-column behavior below the threshold — the typed columns power offline FTS5 and cross-entity SQL joins; the fallback should only fire on pathological widths.

    Acceptance criteria

    • positive: feed a synthetic spec where one resource has 3000+ leaf fields; generated store.go for that resource has only 3 columns; sync against an empty DB succeeds.
    • negative: feed a normal resource (<1500 cols); typed-table emission unchanged; flat columns still emitted.

    Frequency

    subclass:wide-config-singleton — complete-functional-failure when it fires.

    Complexity

    small (column-count check + JSON-only emit branch).

    Scope boundary

    Does NOT cover per-column truncation, schema-aware projection, or per-API column allowlists. Single threshold + JSON-only fallback only.

    Dependencies

    None.

    Related issues

    None — first sighting of this specific failure mode.

    Artifacts

    ArtifactLink
    Retro documenthttps://files.catbox.moe/ico9uo.md
    Manuscripts (research + proofs)https://files.catbox.moe/rn4hf4.zip
    Generated CLI source codehttps://files.catbox.moe/4v6myv.zip

    Generated by /printing-press-retro · CLI Printing Press

    Evaluation contract

    Use 1500 columns as the safety cap for a generated typed SQLite resource table, including dependent-parent columns. Resources exceeding the cap must retain their per-resource table but fall back to a JSON-only schema containing id, data, and synced_at; do not generate typed-column indexes, FTS structures, or typed upsert code for that fallback table. Batch store generation must dispatch wide resources to the JSON-only path while preserving the existing typed behavior for ordinary narrow resources.

    Interface

    No new interfaces are introduced.

    ADDITIONAL ADVICE

    Since you are given a git repository, you can use git commands to simplify your work. For example, if you made a mistake and want to revert your changes, you can use git checkout <file> to restore the file to its original state. You can also reset all changes in the repository using git reset --hard command. Additionally, you can use git stash and git stash pop to temporarily save your changes and restore them later. Finally, you can see the changes that you've made (which will be submitted when you call submit) using git status or git diff commands. However you don't need to use git add or git commit before submitting your changes. In fact, submit only submits currently unstaged changes because it uses git diff to get the changes that need to be submitted.

    CHECKLIST

    Before submitting your solution, please go over the following checklist and make sure you've done everything:

    • If an error was reported in the issue description, I have successfully reproduced it.
    • If an error was reported in the issue description, I have confirmed that my fix resolves the error.
    • I have thought about edge cases and made sure my fix handles them as well.
    • I have run existing tests in the repository that might have been affected by the change I made and confirmed that they pass. I want you to list every bullet from this checklist and write your reasoning for why you think you did it or didn't need to.

    Repository has been uploaded and your shell is currently at the repository root. Time to solve the issue!

    (Current directory: /cli-printing-press, current file: none) bash-$

  2. GLM Assistant

    I'll start by exploring the repository structure to understand the codebase and locate the relevant files for the typed-table emit path.

  3. GLM Assistant

    Let me explore the internal/generator directory structure.

  4. GLM Assistant

    The issue mentions internal/generator/templates/store/ but let me find the actual store-related files.

  5. GLM Assistant

    Let me look at the store.go.tmpl more, specifically the part where CREATE TABLE is composed for resources.

  6. GLM Assistant

    Let me view the key sections of store.go.tmpl, especially the migration block (lines 250-330) and the upsert area.

  7. GLM Assistant

    Now I understand the template structure. Let me find the Go code that builds the Tables and Columns data for the template, and the hasTypedTable helper.

  8. GLM Assistant

    Now let me look at the hasTypedTable helper definition in generator.go.

  9. GLM Assistant

    Let me understand the dependent-parent columns concept and the hasTypedTable flow more. Let me look at schemaWithDependentParents and how columns/Tables flow into the template.

  10. GLM Assistant

    Let me check who calls BuildSchema vs schemaWithDependentParents and where the schema flows into rendering.

  11. GLM Assistant

    Let me look at the related store test templates and the backfill logic to ensure the fallback doesn't break them.

  12. GLM Assistant

    Let me look at the store test templates to ensure the fallback doesn't break them.

  13. GLM Assistant

    Let me check the schema version test template and the MCP tools template for any per-table/per-column emit that could break.

  14. GLM Assistant

    Let me look at the MCP tools rendering to see if it uses Tables/columns.

  15. GLM Assistant

    Let me check the search and sync templates for typed-table references.

  16. GLM Assistant

    Let me look at how the generator currently emits status/log lines during generation.

  17. GLM Assistant

    Let me check how the golden/verify tests capture generator output, to determine if a stderr log line would break them.

  18. GLM Assistant

    Let me look at the golden test infrastructure to understand how generation output is verified.

  19. GLM Assistant

    Let me find the golden test runner that drives full generation and checks output.

  20. GLM Assistant

    Let me look at the golden.sh script and the verify test approach.

  21. GLM Assistant

    The golden tests compare stdout.txt AND stderr.txt byte-for-byte. Let me check what golden cases look like and their stderr.txt content.

  22. GLM Assistant

    Let me check how many times schemaWithDependentParents is called during a single Generate().

  23. GLM Assistant

    So schemaWithDependentParents() is called once at line 2408 (in renderVisionAndRootFiles), and the resulting schema is reused for store, vision, and mcp. But there's ALSO a call at line 1976: g.renderMCPToolFiles(g.schemaWithDependentParents()). Let me check what calls that.

    Let me look at line 1960-1980.

  24. GLM Assistant

    Let me find which function contains line 1976. Let me look above it.

  25. GLM Assistant

    Let me check whether any tests call schemaWithDependentParents directly, and look at existing schema_builder tests.

  26. GLM Assistant

    Now I have a clear understanding. Let me look at the exact template lines to make the precise edit.

  27. GLM Assistant

    I now have a clear plan. Let me implement the changes. First, add the JSONOnly field and the cap helper to schema_builder.go.

  28. GLM Assistant

    Now let me add the maxTypedTableColumns constant and applyColumnCap helper. Let me add them right after baseTableColumns.

  29. GLM Assistant

    Now let me update schemaWithDependentParents in generator.go to apply the cap and emit the log line.

  30. GLM Assistant

    Now let me update the store.go.tmpl template to emit CREATE TABLE for JSON-only fallback tables. Let me open the file at the relevant lines.

  31. GLM Assistant

    Now let me edit the gate comment and condition.

  32. GLM Assistant

    Now let me verify the code compiles.

  33. GLM Assistant

    The Go module cache isn't set up. Let me check the environment and configure it.

  34. GLM Assistant

    I need to set HOME, GOPATH, GOCACHE. Let me configure the Go environment.

  35. GLM Assistant

    Let me look at the error area in generator.go.

  36. GLM Assistant

    The \n in my Python string became a literal newline in the Go source. Let me fix it with the editor.

  37. GLM Assistant

    Let me verify the fix and rebuild.

  38. GLM Assistant

    The build passes. Now let me write a reproduction test. Let me first look at how existing tests set up a Generator and run full generation.

  39. GLM Assistant

    Let me look at the New constructor and VisionSet/Store defaults.

  40. GLM Assistant

    Let me look at fts_rowid_test.go as a template for my reproduction test (small, Store-enabled).

  41. GLM Assistant

    I have all the pieces. Let me write a temporary reproduction/verification test.

  42. GLM Assistant

    My regex helpers are fragile. Let me rewrite the test with simpler, robust extraction.

  43. GLM Assistant

    Let me check the runGoCommand helper signature before running.

  44. GLM Assistant

    Let me check goBuildCacheDir to ensure the build works in this environment.

  45. GLM Assistant

    Now let me run my reproduction test (with the Go env configured).

  46. GLM Assistant

    The full build fails because the MCP main is generated without proper module setup. Let me check the MCP main generation gate and how the build is expected to work.

  47. GLM Assistant

    The MCP main is always generated, so I need MCP: true in VisionSet. Let me also add a migration-open test to confirm "sync against an empty DB succeeds". Let me rewrite the test.

  48. GLM Assistant

    Let me fix the double-Close in the migration test snippet.

  49. GLM Assistant

    Let me look at the actual content and fix it with the editor.

  50. GLM Assistant

    The edit left the embedded snippet incomplete. Let me fix it with a replace.

  51. GLM Assistant

    Now the embedded snippet is valid. Let me run the test.

  52. GLM Assistant

    The extractCreateTable is matching the wrong delimiter (double-quote instead of backtick). Let me fix it to find the closing backtick of the raw string literal.

  53. GLM Assistant

    The test passes. Now let me confirm the fix actually addresses the issue by temporarily disabling the cap and verifying the test fails (reproducing the original error).

  54. GLM Assistant

    I've reproduced the exact error from the issue: SQL logic error: too many columns on control (1). Now let me restore the fix.

  55. GLM Assistant

    The fix works and the original error is reproduced. Now let me verify the "store-fallback" log line is emitted to stderr during generation. Let me check the existing captureStderr helper.

  56. GLM Assistant

    Let me view the full captureStderr helper to use it.

  57. GLM Assistant

    Let me add a stderr-capture check to confirm the fallback log line fires during generation.

  58. GLM Assistant

    Let me run the test to confirm the fallback log line fires.

  59. GLM Assistant

    captureStderr is defined in client_url_redaction_test.go — but maybe with a build tag or in a different package? Let me check. It's package generator? Let me verify it's accessible. The error says "undefined: captureStderr". So it's not in package generator, or it has a build constraint. Let me check the file's package and tags.

  60. GLM Assistant

    Let me check for build constraints on that file.

  61. GLM Assistant

    Let me check the very first bytes of the file for a hidden build tag.

  62. GLM Assistant

    Let me check if the test package builds without my file (to confirm captureStderr is normally available).

  63. GLM Assistant

    That's strange — captureStderr works without my file. Let me see the full build error with my file present.

  64. GLM Assistant

    Let me search for all captureStderr definitions across the generator test files.

  65. GLM Assistant

    Let me verify my test file's package declaration and imports.

  66. GLM Assistant

    This is puzzling. Let me sidestep by defining a local stderr-capture helper in my test file instead.

  67. GLM Assistant

    Now let me add the captureGenStderr helper with a unique name and the io import.

  68. GLM Assistant

    Let me use a Python script to add the helper and import reliably.

  69. GLM Assistant

    All assertions pass, including the stderr fallback log line. Now let me run the broader test suite to ensure nothing is broken. First, build and vet the whole project.

  70. GLM Assistant

    Exit 141 was from the pipe to head. Let me run the build without piping.

  71. GLM Assistant

    Build passes. Now let me run go vet and the schema_builder tests.

  72. GLM Assistant

    Vet and schema_builder tests pass. Now let me run the golden suite to confirm byte-identical output is preserved. First build the binary.

  73. GLM Assistant

    jq is missing (needed by the golden script). Let me try to install it.

  74. GLM Assistant

    Now jq is installed. Let me run the golden verify suite.

  75. GLM Assistant

    Many golden cases failed. Let me determine if these are caused by my change or pre-existing. Let me first check a store.go diff and then compare against a clean baseline.

  76. GLM Assistant

    The store.go diff shows github.com/mvanhorn<REPO> — an environmental path-normalization artifact (repo dir /cli-printing-press collides with the URL slug), not caused by my change. Let me confirm by stashing my changes and re-running golden to get the baseline. First, let me check the go.mod change.

  77. GLM Assistant

    The go.mod change is an unintended side-effect of go build -mod=mod (reorganizing an indirect dep). Let me restore it, then verify the golden baseline by stashing my real changes.

  78. GLM Assistant

    The baseline (without my changes) fails with the same 16 cases. Let me confirm the exact same case set fails in both, to prove my change introduces no new diffs.

  79. GLM Assistant

    Identical case sets — my change introduces no new golden diffs. Let me verify the store.go diffs are purely environmental (the <REPO> normalization) and not my CREATE TABLE change.

  80. GLM Assistant

    The store.go diff is purely the environmental <REPO> normalization in the header — no CREATE TABLE/column changes. My change is golden-safe. Now let me restore my changes.

  81. GLM Assistant

    My changes are restored. Let me review the complete source diff.

  82. GLM Assistant

    The diff is clean and correct. Let me recreate my verification test and run it once more to confirm everything still works after the stash dance.

  83. GLM Assistant

    The test passes. Let me run the broader set of store/schema-related generator tests to ensure nothing regressed.

  84. GLM Assistant

    Let me run the full generated-CLI compile test (TestGenerateProjectsCompile) to confirm end-to-end compilation with the template change.

  85. GLM Assistant

    All full-CLI compile tests pass. Let me run gofmt check and go vet on the changed files.

  86. GLM Assistant

    The .tmpl errors are expected (templates aren't valid Go). Let me confirm the .go files are gofmt-clean.

  87. GLM Assistant

    Everything is clean. Now let me remove my temporary test file before submitting (per the instructions, I must not add tests to the repository).

  88. GLM Assistant

    Let me do a final build to confirm everything compiles after removing the temp test.

  89. GLM Assistant

    Build and vet pass. Let me do a final review of the complete diff.

  90. GLM Assistant

    git diff --stat shows go.mod modified again (from go build -mod=mod). Let me check and revert it, since it's an unintended side-effect.

  91. GLM Assistant

    This go.mod change is an unintended build side-effect. Let me revert it.

  92. GLM Assistant

    Now only my 3 source files are modified. Let me confirm the build works in readonly mode (without modifying go.mod).

  93. GLM Assistant

    All changes are in place, the build is clean in readonly mode, and only my 3 source files are modified. Let me do a final summary review of the complete diff before submitting.

  94. GLM Assistant

    The implementation is complete. Let me summarize the work and submit.

    Summary

    Problem: The typed-table emit path in the generator walks a resource's response schema and emits one SQLite column per leaf field with no upper bound. On wide config singletons (e.g., HaloPSA /Control with 3,747 columns), the generated CREATE TABLE exceeds SQLite's default SQLITE_MAX_COLUMN=2000, failing silently at first sync with SQL logic error: too many columns on control (1).

    Fix (3 files):

    1. internal/generator/schema_builder.go — Added the JSONOnly flag to TableDef, the maxTypedTableColumns = 1500 safety cap, and the applyColumnCap helper that resets an over-cap table in place to the JSON-only fallback shape (id/data/synced_at, no indexes, no FTS5) and flags it JSONOnly.

    2. internal/generator/generator.go — In schemaWithDependentParents (the single canonical schema-construction path), after dependent-parent (parent_id) columns are added, iterate every table and apply the cap so the count includes dependent-parent columns. Emit a generation-time stderr log line store-fallback: <name> (<n> cols) -> JSON-only.

    3. internal/generator/templates/store.go.tmpl — Changed the per-resource CREATE TABLE gate from {{if hasTypedTable .}} to {{if or (hasTypedTable .) .JSONOnly}} so the fallback table is retained (per the contract) with its 3 columns, while all typed surfaces (columns, indexes, FTS5, typed upsert, UpsertBatch dispatch case) stay suppressed because hasTypedTable is false for JSONOnly tables (Columns <= 3).

    Verification:

    • Reproduced the exact issue error (too many columns on control (1)) with the cap disabled; confirmed it's resolved with the cap enabled.
    • Positive criterion: a 3000-field control resource generates a 3-column store.go table, no typed upsert/FTS/indexes, the generated CLI builds, and store.Open migration succeeds against an empty DB.
    • Negative criterion: an ordinary narrow items resource still emits typed columns + upsertItemsTx + UpsertBatch dispatch case.
    • The fallback log line fires to stderr.
    • go build ./..., go vet, gofmt clean; schema_builder, store/FTS/typed/upsert/dependent tests pass; full-CLI compile tests (stytch/clerk/loops) pass.
    • Golden suite: identical pre-existing failure set with and without my change (the failures are environmental repo-path normalization artifacts); no new golden diffs introduced — store.go diffs are purely the header URL normalization that also fails on the untouched baseline.

    Checklist reasoning:

    • Reproduced the error: Yes — synthetic 3000-field spec with cap disabled produced SQL logic error: too many columns on control (1).
    • Confirmed fix resolves the error: Yes — with cap enabled, migration succeeds and the table has 3 columns.
    • Edge cases considered: Yes — cap includes dependent-parent columns (applied post parent_id); sub-resource tables are tiny and unaffected; JSONOnly tables retain their CREATE TABLE but suppress typed columns/indexes/FTS/upsert/dispatch via the existing hasTypedTable gating; backfill generation is a no-op for JSONOnly (only id/data/synced_at, all excluded); narrow resources unchanged; low-gravity "dead" tables still skipped (JSONOnly=false + hasTypedTable=false).
    • Ran affected existing tests: Yes — schema_builder, store/FTS/typed/upsert/dependent tests, full-CLI compile tests, and golden suite all pass/unchanged.

    I removed the temporary verification test before submitting.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.