generator: typed-table emit exceeds SQLite column cap on wide config singletons Filed by…
- User
generator: typed-table emit exceeds SQLite column cap on wide config singletons
Filed by
/printing-press-retrofrom a run of HaloPSA (scorecard 91/100 Grade A, 100% verify pass).Summary
The typed-table emit path generates one column per leaf JSON field in a resource's response schema with no upper bound. On HaloPSA's tenant-global
/Controlconfig singleton, this produced a 3,747-columnCREATE TABLEthat exceeded SQLite's defaultSQLITE_MAX_COLUMN=2000. Generation passed every quality gate;sync --full --dry-rundied cold withSQL logic error: too many columns on control (1). The failure mode is silent until first sync.Where to look
- Component: generator
- Likely area: Typed-table-emit path in
internal/generator/templates/store/— whereverCREATE TABLEtext is composed per discovered resource. - Triggered when: A spec resource's response schema flattens to >~1500 leaf fields. Enterprise config singletons are the obvious class.
What we observed
Run-1 output (post-generate):
PASS go mod tidy / govulncheck / go vet / go build PASS halopsa-pp-cli --help / version / doctor Generated halopsa at .../working/halopsa-pp-cliFirst
sync --full --dry-run:Error: opening local database: running migrations: migration failed: SQL logic error: too many columns on control (1)Per-table column counts (top 5 widest, recorded by Python introspection of
internal/store/store.go):table columns control 3747 tickets 847 projects 847 opportunities 847 client 447 Tickets / projects / opportunities sit just under the 2000 cap but with no headroom; the next major-version Halo response schema could push them over.
Workaround applied this run: hand-stripped
/Control*paths from the spec (5 paths) and added a stubControlschema so residual$reftargets resolved. Regenerated successfully.Suspected root cause
Confident. The typed-table-emit path walks the resource's response schema and emits one column per leaf field with no width sanity check. The store package has no JSON-only fallback for resources whose schema exceeds a safe column count.
Suggested direction
In the typed-table-emit path, compute the column count before writing
CREATE TABLE. If the count exceeds a threshold (1500 leaves headroom under SQLite's default 2000; tunable), fall back to JSON-only schema:id TEXT PRIMARY KEY, data JSON NOT NULL, synced_at DATETIME. Emit a generation-time log line naming the resource and the fallback (e.g.,store-fallback: control (3747 cols) → JSON-only).Preserve typed-column behavior below the threshold — the typed columns power offline FTS5 and cross-entity SQL joins; the fallback should only fire on pathological widths.
Acceptance criteria
- positive: feed a synthetic spec where one resource has 3000+ leaf fields; generated
store.gofor that resource has only 3 columns; sync against an empty DB succeeds. - negative: feed a normal resource (<1500 cols); typed-table emission unchanged; flat columns still emitted.
Frequency
subclass:wide-config-singleton — complete-functional-failure when it fires.
Complexity
small (column-count check + JSON-only emit branch).
Scope boundary
Does NOT cover per-column truncation, schema-aware projection, or per-API column allowlists. Single threshold + JSON-only fallback only.
Dependencies
None.
Related issues
None — first sighting of this specific failure mode.
Artifacts
Artifact Link Retro document https://files.catbox.moe/ico9uo.md Manuscripts (research + proofs) https://files.catbox.moe/rn4hf4.zip Generated CLI source code https://files.catbox.moe/4v6myv.zip
Generated by
/printing-press-retro· CLI Printing PressEvaluation contract
Use 1500 columns as the safety cap for a generated typed SQLite resource table, including dependent-parent columns. Resources exceeding the cap must retain their per-resource table but fall back to a JSON-only schema containing
id,data, andsynced_at; do not generate typed-column indexes, FTS structures, or typed upsert code for that fallback table. Batch store generation must dispatch wide resources to the JSON-only path while preserving the existing typed behavior for ordinary narrow resources.Interface
No new interfaces are introduced.
- Assistant
Now let me look at how the template consumes
TableDef— particularly typed upserts: - Assistant
Now let me look at the second test in
store_upsert_batch_test.go.tmpland the schema builder tests for style: - Assistant
Let me check the
columnNames/columnPlaceholdershelpers andhasDomainUpsertusage: - Assistant
Now I'll implement. First,
schema_builder.go: - Assistant
Now the store template. First the CREATE TABLE gate:
- Assistant
Now add the JSON-only upsert code after the typed upsert block:
- Assistant
Now the
$hasTypedcomputation and the UpsertBatch dispatch switch: - Assistant
Now the generated batch-test template gates:
- Assistant
Now add tests. Let me look at how existing generator tests render/generate to inspect output:
- Assistant
Now the tests. First, unit tests for
enforceTypedColumnCapinschema_builder_test.go: - Assistant
Now the generator-level end-to-end test:
- Assistant
End-to-end passes — including the generated module's own
internal/storetests (empty-DB migration + batch upsert into the fallback table). Now the full suite and goldens: - Assistant
Passes in isolation — the batch failures may be contention. Let me look at the actual error output in a full run:
- Assistant
Those failures are
fork/exec: resource temporarily unavailable— the sandbox's process limit being hit by parallel generated-module compiles, not code failures. Re-run serialized: - Assistant
Only the pre-existing govulncheck/toolchain failure remains (confirmed identical on clean tree). Now the full repo suite and golden verification:
- Assistant
Pre-existing on clean tree as well (likely a root-user permission quirk). Now golden verification:
- Assistant
The diffs are all
https://github.com/mvanhorn/cli-printing-press→mvanhorn<REPO>— a normalization artifact. Let me confirm: - Assistant
Golden verify passes cleanly (23/23) with my change when run from a non-colliding path. Cleanup and final checks:
- Assistant
All done. Here's a summary of the fix:
Machine change: typed-column cap with JSON-only fallback
Root cause confirmed as filed:
BuildSchemaemitted one column per scalar leaf field with no bound, so wide config singletons (HaloPSA/Control, 3,747 cols) generatedCREATE TABLEstatements past SQLite'sSQLITE_MAX_COLUMN=2000, passing every gate and dying at first sync's migrations.Changes
internal/generator/schema_builder.go— addedmaxTypedTableColumns = 1500andenforceTypedColumnCap(): tables wider than the cap drop to theid/data/synced_atbase shape, clear all indexes and FTS structures, set a newTableDef.JSONOnlyflag, and logstore-fallback: <name> (N cols) -> JSON-onlyat generation time.internal/generator/generator.go— the cap runs at the end ofschemaWithDependentParents(), after dependentparent_idcolumns are added, so the cap reflects final table width (both call sites — full generate and mcp-sync — flow through it).hasTypedTablenow excludesJSONOnlytables so they never get typed upserts.templates/store.go.tmpl—CREATE TABLEgate widened toor (hasTypedTable .) .JSONOnly(the table survives, per contract); newupsert<X>JSONTx+ publicUpsert<X>for JSON-only tables (raw id/data/synced_at write alongside the generic resources row);UpsertBatch's dispatch switch routes wide resources to the JSON-only path under the same savepoint machinery.templates/sync.go.tmpl—upsertSingleObject(the path config singletons actually take) dispatches JSON-only tables throughUpsert<X>instead of the generic-only fallback.templates/store_upsert_batch_test.go.tmpl— generated populate-table tests now cover JSON-only tables too, so every created table stays pinned as written-to.
Tests
TestEnforceTypedColumnCap(unit): at-cap stays typed; one-over-cap (modeling the dependent-parent column) falls back with exactlyid/data/synced_at, no indexes/FTS; narrow untouched.TestGenerate_WideResourceStoreFallsBackToJSONOnly(e2e, matches acceptance criteria): 3,203-leaf resource → generatedstore.gohas a 3-columncontroltable, noidx_control_*, nocontrol_fts, no typed upsert; batch and single-object dispatch go throughupsertControlJSONTx/UpsertControl; narrowitemsresource keeps full typed emission. The test then compiles and runs the generated module'sinternal/storetests, which open an empty DB (migration — where the old code died) and batch-upsert into the fallback table.
Verification
go test ./internal/generatorpasses except pre-existingTestGenerateWithNoAuth— govulncheck flags the installed go1.26.4 toolchain's stdlib crypto/tls vuln (fixed in 1.26.5); fails identically on a clean tree. Oneinternal/pipelinetest also fails pre-existing on clean tree. Batch-modefork/exec: resource temporarily unavailablenoise is a sandbox process-limit; everything passes with-p 1 -parallel 2.scripts/golden.sh verify: 23/23 PASS. Note: run from the workspace path it false-fails because this checkout lives at/cli-printing-press, which collides with the harness's<REPO>path-token normalization (manglesgithub.com/mvanhorn/cli-printing-pressURLs); verified green from a copy at a non-colliding path.go fmt/go vetclean;golangci-lintisn't installed in this environment.
Golden fixture decision: no new fixture — a >1500-leaf spec fixture would be impractically large for the golden tree, and the e2e generator test pins the emitted contract more strongly (it also compiles and executes the generated store against an empty DB). Scope honored: no per-column truncation, projection, or allowlists — single threshold + JSON-only fallback only.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.
Identical first prompt
- composer-2.5Compare side by side →
- composer-2.5Compare side by side →
- composer-2.5Compare side by side →
- composer-2.5Compare side by side →
- composer-2.5Compare side by side →
- gpt-5.6-solCompare side by side →
- gpt-5.6-solCompare side by side →
- gpt-5.6-solCompare side by side →
- gpt-5.6-solCompare side by side →
- gpt-5.6-solCompare side by side →
- fable-5Compare side by side →
- fable-5Compare side by side →