ISSUE DESCRIPTION generator: typed-table emit exceeds SQLite column cap on wide config…
- User
ISSUE DESCRIPTION
generator: typed-table emit exceeds SQLite column cap on wide config singletons
Filed by
/printing-press-retrofrom a run of HaloPSA (scorecard 91/100 Grade A, 100% verify pass).Summary
The typed-table emit path generates one column per leaf JSON field in a resource's response schema with no upper bound. On HaloPSA's tenant-global
/Controlconfig singleton, this produced a 3,747-columnCREATE TABLEthat exceeded SQLite's defaultSQLITE_MAX_COLUMN=2000. Generation passed every quality gate;sync --full --dry-rundied cold withSQL logic error: too many columns on control (1). The failure mode is silent until first sync.Where to look
- Component: generator
- Likely area: Typed-table-emit path in
internal/generator/templates/store/— whereverCREATE TABLEtext is composed per discovered resource. - Triggered when: A spec resource's response schema flattens to >~1500 leaf fields. Enterprise config singletons are the obvious class.
What we observed
Run-1 output (post-generate):
PASS go mod tidy / govulncheck / go vet / go build PASS halopsa-pp-cli --help / version / doctor Generated halopsa at .../working/halopsa-pp-cliFirst
sync --full --dry-run:Error: opening local database: running migrations: migration failed: SQL logic error: too many columns on control (1)Per-table column counts (top 5 widest, recorded by Python introspection of
internal/store/store.go):table columns control 3747 tickets 847 projects 847 opportunities 847 client 447 Tickets / projects / opportunities sit just under the 2000 cap but with no headroom; the next major-version Halo response schema could push them over.
Workaround applied this run: hand-stripped
/Control*paths from the spec (5 paths) and added a stubControlschema so residual$reftargets resolved. Regenerated successfully.Suspected root cause
Confident. The typed-table-emit path walks the resource's response schema and emits one column per leaf field with no width sanity check. The store package has no JSON-only fallback for resources whose schema exceeds a safe column count.
Suggested direction
In the typed-table-emit path, compute the column count before writing
CREATE TABLE. If the count exceeds a threshold (1500 leaves headroom under SQLite's default 2000; tunable), fall back to JSON-only schema:id TEXT PRIMARY KEY, data JSON NOT NULL, synced_at DATETIME. Emit a generation-time log line naming the resource and the fallback (e.g.,store-fallback: control (3747 cols) → JSON-only).Preserve typed-column behavior below the threshold — the typed columns power offline FTS5 and cross-entity SQL joins; the fallback should only fire on pathological widths.
Acceptance criteria
- positive: feed a synthetic spec where one resource has 3000+ leaf fields; generated
store.gofor that resource has only 3 columns; sync against an empty DB succeeds. - negative: feed a normal resource (<1500 cols); typed-table emission unchanged; flat columns still emitted.
Frequency
subclass:wide-config-singleton — complete-functional-failure when it fires.
Complexity
small (column-count check + JSON-only emit branch).
Scope boundary
Does NOT cover per-column truncation, schema-aware projection, or per-API column allowlists. Single threshold + JSON-only fallback only.
Dependencies
None.
Related issues
None — first sighting of this specific failure mode.
Artifacts
Artifact Link Retro document https://files.catbox.moe/ico9uo.md Manuscripts (research + proofs) https://files.catbox.moe/rn4hf4.zip Generated CLI source code https://files.catbox.moe/4v6myv.zip
Generated by
/printing-press-retro· CLI Printing PressEvaluation contract
Use 1500 columns as the safety cap for a generated typed SQLite resource table, including dependent-parent columns. Resources exceeding the cap must retain their per-resource table but fall back to a JSON-only schema containing
id,data, andsynced_at; do not generate typed-column indexes, FTS structures, or typed upsert code for that fallback table. Batch store generation must dispatch wide resources to the JSON-only path while preserving the existing typed behavior for ordinary narrow resources.Interface
No new interfaces are introduced.
ADDITIONAL ADVICE
Since you are given a git repository, you can use git commands to simplify your work. For example, if you made a mistake and want to revert your changes, you can use
git checkout <file>to restore the file to its original state. You can also reset all changes in the repository usinggit reset --hardcommand. Additionally, you can usegit stashandgit stash popto temporarily save your changes and restore them later. Finally, you can see the changes that you've made (which will be submitted when you callsubmit) usinggit statusorgit diffcommands. However you don't need to usegit addorgit commitbefore submitting your changes. In fact,submitonly submits currently unstaged changes because it usesgit diffto get the changes that need to be submitted.CHECKLIST
Before submitting your solution, please go over the following checklist and make sure you've done everything:
- If an error was reported in the issue description, I have successfully reproduced it.
- If an error was reported in the issue description, I have confirmed that my fix resolves the error.
- I have thought about edge cases and made sure my fix handles them as well.
- I have run existing tests in the repository that might have been affected by the change I made and confirmed that they pass. I want you to list every bullet from this checklist and write your reasoning for why you think you did it or didn't need to.
Repository has been uploaded and your shell is currently at the repository root. Time to solve the issue!
(Current directory: /cli-printing-press, current file: none) bash-$
Assistant
Now let's look at test around line 127 to understand fixture conventions for generating spec resources with many fields.
Assistant
Now let's add the imports and the constant/helper function, then wire it into BuildSchema and buildSubResourceTable.
Assistant
Now add the constant and helper function after
baseTableColumnsvar block:Assistant
Now add the
JSONOnlyfield to TableDef struct:Assistant
Now wire
enforceColumnCapintoBuildSchemafor the top-level table, right after FTS5 assignment and beforetables = append(tables, table), and also intobuildSubResourceTable.Assistant
Now update
buildSubResourceTableto also enforce the cap (for completeness/generality):Assistant
Now let's verify build compiles so far:
Assistant
Good, it builds. Now let's update the
hasTypedTablepredicate usage / store.go.tmpl to gate CREATE TABLE onhasTypedTable OR JSONOnly. Let's check thehasTypedTablefunc and the generator.go section again.Assistant
Now update store.go.tmpl's CREATE TABLE gate:
Assistant
Fine, unaffected. Now let's check
hasDomainUpsertanddomainUpsertMethodNameused, and check other places relying on.Columnslength e.g. mcp_tools.go.tmpl doesn't. Let's search entire templates dir for.Tablesusage broadly to ensure nothing else assumes hasTypedTable == table exists.Assistant
Let's check store_schema_version_test.go.tmpl and line 1024-1040 area of store.go.tmpl (the second occurrence with
.FTS5) already inspected. Let's check store_schema_version_test.go.tmpl usage.Assistant
Now let's write a quick script to reproduce and test the acceptance criteria: generate a synthetic spec resource with 3000+ fields and verify
BuildSchematruncates to 3 columns, plus check normal resources unaffected.Assistant
Now let's test that the generated
store.gocompiles for such a wide spec, and that a normal narrow resource is unaffected. Let's check how generator tests build full CLI output - check generator_test.go for a harness we could reuse.Assistant
Excellent, full E2E test passes, including
go buildof the generated CLI. Now let's clean up temp test files and run the existing test suite for generator and store-related packages to make sure nothing broke.Assistant
Confirmed this is a pre-existing environment issue (Go toolchain version vulnerability), unrelated to our change. Let's run the full generator test suite skipping this known-failing test, and also run store-related tests in testdata/golden.
Assistant
All generator tests pass (except the pre-existing environment-only govulncheck failure). Now let's run broader test suite for pipeline and other areas that reference store.go templates (golden tests, dogfood, etc.) but skip ones that call govulncheck.
Assistant
Let's run the golden verify script to confirm our change doesn't break existing golden outputs.
Assistant
The diff is empty — identical set of failing golden cases before and after our change, confirming these failures are pre-existing environment issues unrelated to our fix. Let's clean up the built binary and temp artifacts.
Assistant
Let's view the final diff to double-check everything looks correct.
Assistant
This looks good. Now let's also verify the schema_builder_test.go existing tests around thresholds still pass and add a quick sanity check by running the full generator package test suite once more (already ran). Let's also check
go vet.Assistant
All clean. Let's also double check the whole repo build/test for
internal/storepackage (the shared, non-generated store package, if any) isn't relevant here since store.go is generated per-CLI from template — good. Let's check for any other place referencingTableDeffields count assumptions, e.g. in scorecard.go / verify.go (found earlier grep for "CREATE TABLE").Assistant
Good, no conflicts. Now let's run the full generator test suite once more (final check) and gofmt.
Assistant
No formatting issues. Let's run the generator tests once more (skip the known env-only failure) as a final confirmation.
Assistant
All tests pass. Let's do a final
git statuscheck and review the diff once more.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.
Identical first prompt
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-27BCompare side by side →
Qwen/Qwen3.6-27BCompare side by side →