generator: typed-table emit exceeds SQLite column cap on wide config singletons Filed by…
- User
generator: typed-table emit exceeds SQLite column cap on wide config singletons
Filed by
/printing-press-retrofrom a run of HaloPSA (scorecard 91/100 Grade A, 100% verify pass).Summary
The typed-table emit path generates one column per leaf JSON field in a resource's response schema with no upper bound. On HaloPSA's tenant-global
/Controlconfig singleton, this produced a 3,747-columnCREATE TABLEthat exceeded SQLite's defaultSQLITE_MAX_COLUMN=2000. Generation passed every quality gate;sync --full --dry-rundied cold withSQL logic error: too many columns on control (1). The failure mode is silent until first sync.Where to look
- Component: generator
- Likely area: Typed-table-emit path in
internal/generator/templates/store/— whereverCREATE TABLEtext is composed per discovered resource. - Triggered when: A spec resource's response schema flattens to >~1500 leaf fields. Enterprise config singletons are the obvious class.
What we observed
Run-1 output (post-generate):
PASS go mod tidy / govulncheck / go vet / go build PASS halopsa-pp-cli --help / version / doctor Generated halopsa at .../working/halopsa-pp-cliFirst
sync --full --dry-run:Error: opening local database: running migrations: migration failed: SQL logic error: too many columns on control (1)Per-table column counts (top 5 widest, recorded by Python introspection of
internal/store/store.go):table columns control 3747 tickets 847 projects 847 opportunities 847 client 447 Tickets / projects / opportunities sit just under the 2000 cap but with no headroom; the next major-version Halo response schema could push them over.
Workaround applied this run: hand-stripped
/Control*paths from the spec (5 paths) and added a stubControlschema so residual$reftargets resolved. Regenerated successfully.Suspected root cause
Confident. The typed-table-emit path walks the resource's response schema and emits one column per leaf field with no width sanity check. The store package has no JSON-only fallback for resources whose schema exceeds a safe column count.
Suggested direction
In the typed-table-emit path, compute the column count before writing
CREATE TABLE. If the count exceeds a threshold (1500 leaves headroom under SQLite's default 2000; tunable), fall back to JSON-only schema:id TEXT PRIMARY KEY, data JSON NOT NULL, synced_at DATETIME. Emit a generation-time log line naming the resource and the fallback (e.g.,store-fallback: control (3747 cols) → JSON-only).Preserve typed-column behavior below the threshold — the typed columns power offline FTS5 and cross-entity SQL joins; the fallback should only fire on pathological widths.
Acceptance criteria
- positive: feed a synthetic spec where one resource has 3000+ leaf fields; generated
store.gofor that resource has only 3 columns; sync against an empty DB succeeds. - negative: feed a normal resource (<1500 cols); typed-table emission unchanged; flat columns still emitted.
Frequency
subclass:wide-config-singleton — complete-functional-failure when it fires.
Complexity
small (column-count check + JSON-only emit branch).
Scope boundary
Does NOT cover per-column truncation, schema-aware projection, or per-API column allowlists. Single threshold + JSON-only fallback only.
Dependencies
None.
Related issues
None — first sighting of this specific failure mode.
Artifacts
Artifact Link Retro document https://files.catbox.moe/ico9uo.md Manuscripts (research + proofs) https://files.catbox.moe/rn4hf4.zip Generated CLI source code https://files.catbox.moe/4v6myv.zip
Generated by
/printing-press-retro· CLI Printing PressEvaluation contract
Use 1500 columns as the safety cap for a generated typed SQLite resource table, including dependent-parent columns. Resources exceeding the cap must retain their per-resource table but fall back to a JSON-only schema containing
id,data, andsynced_at; do not generate typed-column indexes, FTS structures, or typed upsert code for that fallback table. Batch store generation must dispatch wide resources to the JSON-only path while preserving the existing typed behavior for ordinary narrow resources.Interface
No new interfaces are introduced.
- Assistant
I'll implement the 1500-column SQLite cap with a JSON-only fallback in the typed-table emit path. Exploring the store generator templates and related code first.
- Assistant
Implementing the column-cap fallback: adding
JSONOnlytoTableDef, enforcing the 1500-column limit in the schema builder, and updating store templates for JSON-only domain tables. - Assistant
Implemented the SQLite column-cap guard in the typed-table emit path.
Behavior
maxSQLiteResourceColumns = 1500— safety cap with headroom under SQLite’s default 2000-column limit.- When a resource table exceeds the cap (including after
parent_idinjection for dependent sync), it falls back to JSON-only:id,data,synced_at. - Fallback clears typed columns, indexes, and FTS5 structures.
- Emits
store-fallback: <table> (<N> cols) → JSON-onlyto stderr at generation time. - Narrow resources are unchanged — typed columns, indexes, and FTS still emit when under the cap.
Code changes
internal/generator/schema_builder.go- Added
JSONOnlyonTableDef. - Added
applyJSONOnlyFallbackIfWide()called fromBuildSchemaand again fromschemaWithDependentParents(post-parent_id).
internal/generator/generator.go- Added
hasDomainTabletemplate func (typed tables + JSON-only wide fallbacks). hasTypedTablenow excludesJSONOnlytables.
internal/generator/templates/store.go.tmplCREATE TABLEandUpsertBatchdispatch usehasDomainTable.- JSON-only tables get a simple 3-column upsert (no field extraction, no FTS).
internal/generator/templates/sync.go.tmplupsertSingleObjectroutes JSON-only domain tables through theirUpsert*methods.
Tests
TestBuildSchema_WideResourceFallsBackToJSONOnly— 3000+ fields → 3 columns +JSONOnly.TestBuildSchema_NarrowResourceKeepsTypedColumns— normal resources keep typed columns.TestGenerateStoreWideResourceJSONOnlyFallback— full generate, stderr log, 3-columnCREATE TABLE, compiles, andinternal/storetests pass (migration succeeds).
HaloPSA’s
controltable (3747 columns) would now generate as a 3-column JSON-only table instead of failing at first sync.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.
Identical first prompt
- composer-2.5Compare side by side →
- composer-2.5Compare side by side →
- composer-2.5Compare side by side →
- composer-2.5Compare side by side →
- gpt-5.6-solCompare side by side →
- gpt-5.6-solCompare side by side →
- gpt-5.6-solCompare side by side →
- gpt-5.6-solCompare side by side →
- gpt-5.6-solCompare side by side →
- fable-5Compare side by side →
- fable-5Compare side by side →