ISSUE DESCRIPTION generator: typed-table emit exceeds SQLite column cap on wide config…
- User
ISSUE DESCRIPTION
generator: typed-table emit exceeds SQLite column cap on wide config singletons
Filed by
/printing-press-retrofrom a run of HaloPSA (scorecard 91/100 Grade A, 100% verify pass).Summary
The typed-table emit path generates one column per leaf JSON field in a resource's response schema with no upper bound. On HaloPSA's tenant-global
/Controlconfig singleton, this produced a 3,747-columnCREATE TABLEthat exceeded SQLite's defaultSQLITE_MAX_COLUMN=2000. Generation passed every quality gate;sync --full --dry-rundied cold withSQL logic error: too many columns on control (1). The failure mode is silent until first sync.Where to look
- Component: generator
- Likely area: Typed-table-emit path in
internal/generator/templates/store/— whereverCREATE TABLEtext is composed per discovered resource. - Triggered when: A spec resource's response schema flattens to >~1500 leaf fields. Enterprise config singletons are the obvious class.
What we observed
Run-1 output (post-generate):
PASS go mod tidy / govulncheck / go vet / go build PASS halopsa-pp-cli --help / version / doctor Generated halopsa at .../working/halopsa-pp-cliFirst
sync --full --dry-run:Error: opening local database: running migrations: migration failed: SQL logic error: too many columns on control (1)Per-table column counts (top 5 widest, recorded by Python introspection of
internal/store/store.go):table columns control 3747 tickets 847 projects 847 opportunities 847 client 447 Tickets / projects / opportunities sit just under the 2000 cap but with no headroom; the next major-version Halo response schema could push them over.
Workaround applied this run: hand-stripped
/Control*paths from the spec (5 paths) and added a stubControlschema so residual$reftargets resolved. Regenerated successfully.Suspected root cause
Confident. The typed-table-emit path walks the resource's response schema and emits one column per leaf field with no width sanity check. The store package has no JSON-only fallback for resources whose schema exceeds a safe column count.
Suggested direction
In the typed-table-emit path, compute the column count before writing
CREATE TABLE. If the count exceeds a threshold (1500 leaves headroom under SQLite's default 2000; tunable), fall back to JSON-only schema:id TEXT PRIMARY KEY, data JSON NOT NULL, synced_at DATETIME. Emit a generation-time log line naming the resource and the fallback (e.g.,store-fallback: control (3747 cols) → JSON-only).Preserve typed-column behavior below the threshold — the typed columns power offline FTS5 and cross-entity SQL joins; the fallback should only fire on pathological widths.
Acceptance criteria
- positive: feed a synthetic spec where one resource has 3000+ leaf fields; generated
store.gofor that resource has only 3 columns; sync against an empty DB succeeds. - negative: feed a normal resource (<1500 cols); typed-table emission unchanged; flat columns still emitted.
Frequency
subclass:wide-config-singleton — complete-functional-failure when it fires.
Complexity
small (column-count check + JSON-only emit branch).
Scope boundary
Does NOT cover per-column truncation, schema-aware projection, or per-API column allowlists. Single threshold + JSON-only fallback only.
Dependencies
None.
Related issues
None — first sighting of this specific failure mode.
Artifacts
Artifact Link Retro document https://files.catbox.moe/ico9uo.md Manuscripts (research + proofs) https://files.catbox.moe/rn4hf4.zip Generated CLI source code https://files.catbox.moe/4v6myv.zip
Generated by
/printing-press-retro· CLI Printing PressEvaluation contract
Use 1500 columns as the safety cap for a generated typed SQLite resource table, including dependent-parent columns. Resources exceeding the cap must retain their per-resource table but fall back to a JSON-only schema containing
id,data, andsynced_at; do not generate typed-column indexes, FTS structures, or typed upsert code for that fallback table. Batch store generation must dispatch wide resources to the JSON-only path while preserving the existing typed behavior for ordinary narrow resources.Interface
No new interfaces are introduced.
ADDITIONAL ADVICE
Since you are given a git repository, you can use git commands to simplify your work. For example, if you made a mistake and want to revert your changes, you can use
git checkout <file>to restore the file to its original state. You can also reset all changes in the repository usinggit reset --hardcommand. Additionally, you can usegit stashandgit stash popto temporarily save your changes and restore them later. Finally, you can see the changes that you've made (which will be submitted when you callsubmit) usinggit statusorgit diffcommands. However you don't need to usegit addorgit commitbefore submitting your changes. In fact,submitonly submits currently unstaged changes because it usesgit diffto get the changes that need to be submitted.CHECKLIST
Before submitting your solution, please go over the following checklist and make sure you've done everything:
- If an error was reported in the issue description, I have successfully reproduced it.
- If an error was reported in the issue description, I have confirmed that my fix resolves the error.
- I have thought about edge cases and made sure my fix handles them as well.
- I have run existing tests in the repository that might have been affected by the change I made and confirmed that they pass. I want you to list every bullet from this checklist and write your reasoning for why you think you did it or didn't need to.
Repository has been uploaded and your shell is currently at the repository root. Time to solve the issue!
(Current directory: /cli-printing-press, current file: none) bash-$
Assistant
I'll start by exploring the repository structure to understand the codebase and locate the relevant files for the typed-table emit path.
Assistant
Let me explore the internal/generator directory structure.
Assistant
The issue mentions
internal/generator/templates/store/but let me find the actual store-related files.Assistant
Let me look at the store.go.tmpl more, specifically the part where
CREATE TABLEis composed for resources.Assistant
Let me view the key sections of store.go.tmpl, especially the migration block (lines 250-330) and the upsert area.
Assistant
Now I understand the template structure. Let me find the Go code that builds the
TablesandColumnsdata for the template, and thehasTypedTablehelper.Assistant
Now let me look at the
hasTypedTablehelper definition in generator.go.Assistant
Let me understand the dependent-parent columns concept and the
hasTypedTableflow more. Let me look atschemaWithDependentParentsand how columns/Tables flow into the template.Assistant
Let me check who calls
BuildSchemavsschemaWithDependentParentsand where the schema flows into rendering.Assistant
Let me look at the related store test templates and the backfill logic to ensure the fallback doesn't break them.
Assistant
Let me look at the store test templates to ensure the fallback doesn't break them.
Assistant
Let me check the schema version test template and the MCP tools template for any per-table/per-column emit that could break.
Assistant
Let me look at the MCP tools rendering to see if it uses Tables/columns.
Assistant
Let me check the search and sync templates for typed-table references.
Assistant
Let me look at how the generator currently emits status/log lines during generation.
Assistant
Let me check how the golden/verify tests capture generator output, to determine if a stderr log line would break them.
Assistant
Let me look at the golden test infrastructure to understand how generation output is verified.
Assistant
Let me find the golden test runner that drives full generation and checks output.
Assistant
Let me look at the golden.sh script and the verify test approach.
Assistant
The golden tests compare stdout.txt AND stderr.txt byte-for-byte. Let me check what golden cases look like and their stderr.txt content.
Assistant
Let me check how many times
schemaWithDependentParentsis called during a singleGenerate().Assistant
So
schemaWithDependentParents()is called once at line 2408 (in renderVisionAndRootFiles), and the resultingschemais reused for store, vision, and mcp. But there's ALSO a call at line 1976:g.renderMCPToolFiles(g.schemaWithDependentParents()). Let me check what calls that.Let me look at line 1960-1980.
Assistant
Let me find which function contains line 1976. Let me look above it.
Assistant
Let me check whether any tests call
schemaWithDependentParentsdirectly, and look at existing schema_builder tests.Assistant
Now I have a clear understanding. Let me look at the exact template lines to make the precise edit.
Assistant
I now have a clear plan. Let me implement the changes. First, add the
JSONOnlyfield and the cap helper toschema_builder.go.Assistant
Now let me add the
maxTypedTableColumnsconstant andapplyColumnCaphelper. Let me add them right afterbaseTableColumns.Assistant
Now let me update
schemaWithDependentParentsin generator.go to apply the cap and emit the log line.Assistant
Now let me update the
store.go.tmpltemplate to emitCREATE TABLEfor JSON-only fallback tables. Let me open the file at the relevant lines.Assistant
Now let me edit the gate comment and condition.
Assistant
Now let me verify the code compiles.
Assistant
The Go module cache isn't set up. Let me check the environment and configure it.
Assistant
I need to set HOME, GOPATH, GOCACHE. Let me configure the Go environment.
Assistant
Let me look at the error area in generator.go.
Assistant
The
\nin my Python string became a literal newline in the Go source. Let me fix it with the editor.Assistant
Let me verify the fix and rebuild.
Assistant
The build passes. Now let me write a reproduction test. Let me first look at how existing tests set up a
Generatorand run full generation.Assistant
Let me look at the
Newconstructor andVisionSet/Store defaults.Assistant
Let me look at
fts_rowid_test.goas a template for my reproduction test (small, Store-enabled).Assistant
I have all the pieces. Let me write a temporary reproduction/verification test.
Assistant
My regex helpers are fragile. Let me rewrite the test with simpler, robust extraction.
Assistant
Let me check the
runGoCommandhelper signature before running.Assistant
Let me check
goBuildCacheDirto ensure the build works in this environment.Assistant
Now let me run my reproduction test (with the Go env configured).
Assistant
The full build fails because the MCP main is generated without proper module setup. Let me check the MCP main generation gate and how the build is expected to work.
Assistant
The MCP main is always generated, so I need
MCP: truein VisionSet. Let me also add a migration-open test to confirm "sync against an empty DB succeeds". Let me rewrite the test.Assistant
Let me fix the double-Close in the migration test snippet.
Assistant
Let me look at the actual content and fix it with the editor.
Assistant
The edit left the embedded snippet incomplete. Let me fix it with a replace.
Assistant
Now the embedded snippet is valid. Let me run the test.
Assistant
The
extractCreateTableis matching the wrong delimiter (double-quote instead of backtick). Let me fix it to find the closing backtick of the raw string literal.Assistant
The test passes. Now let me confirm the fix actually addresses the issue by temporarily disabling the cap and verifying the test fails (reproducing the original error).
Assistant
I've reproduced the exact error from the issue:
SQL logic error: too many columns on control (1). Now let me restore the fix.Assistant
The fix works and the original error is reproduced. Now let me verify the "store-fallback" log line is emitted to stderr during generation. Let me check the existing
captureStderrhelper.Assistant
Let me view the full
captureStderrhelper to use it.Assistant
Let me add a stderr-capture check to confirm the fallback log line fires during generation.
Assistant
Let me run the test to confirm the fallback log line fires.
Assistant
captureStderris defined inclient_url_redaction_test.go— but maybe with a build tag or in a different package? Let me check. It'spackage generator? Let me verify it's accessible. The error says "undefined: captureStderr". So it's not in package generator, or it has a build constraint. Let me check the file's package and tags.Assistant
Let me check for build constraints on that file.
Assistant
Let me check the very first bytes of the file for a hidden build tag.
Assistant
Let me check if the test package builds without my file (to confirm
captureStderris normally available).Assistant
That's strange —
captureStderrworks without my file. Let me see the full build error with my file present.Assistant
Let me search for all
captureStderrdefinitions across the generator test files.Assistant
Let me verify my test file's package declaration and imports.
Assistant
This is puzzling. Let me sidestep by defining a local stderr-capture helper in my test file instead.
Assistant
Now let me add the
captureGenStderrhelper with a unique name and theioimport.Assistant
Let me use a Python script to add the helper and import reliably.
Assistant
All assertions pass, including the stderr fallback log line. Now let me run the broader test suite to ensure nothing is broken. First, build and vet the whole project.
Assistant
Exit 141 was from the pipe to
head. Let me run the build without piping.Assistant
Build passes. Now let me run
go vetand the schema_builder tests.Assistant
Vet and schema_builder tests pass. Now let me run the golden suite to confirm byte-identical output is preserved. First build the binary.
Assistant
jqis missing (needed by the golden script). Let me try to install it.Assistant
Now
jqis installed. Let me run the golden verify suite.Assistant
Many golden cases failed. Let me determine if these are caused by my change or pre-existing. Let me first check a store.go diff and then compare against a clean baseline.
Assistant
The store.go diff shows
github.com/mvanhorn<REPO>— an environmental path-normalization artifact (repo dir/cli-printing-presscollides with the URL slug), not caused by my change. Let me confirm by stashing my changes and re-running golden to get the baseline. First, let me check thego.modchange.Assistant
The
go.modchange is an unintended side-effect ofgo build -mod=mod(reorganizing an indirect dep). Let me restore it, then verify the golden baseline by stashing my real changes.Assistant
The baseline (without my changes) fails with the same 16 cases. Let me confirm the exact same case set fails in both, to prove my change introduces no new diffs.
Assistant
Identical case sets — my change introduces no new golden diffs. Let me verify the store.go diffs are purely environmental (the
<REPO>normalization) and not my CREATE TABLE change.Assistant
The store.go diff is purely the environmental
<REPO>normalization in the header — no CREATE TABLE/column changes. My change is golden-safe. Now let me restore my changes.Assistant
My changes are restored. Let me review the complete source diff.
Assistant
The diff is clean and correct. Let me recreate my verification test and run it once more to confirm everything still works after the stash dance.
Assistant
The test passes. Let me run the broader set of store/schema-related generator tests to ensure nothing regressed.
Assistant
Let me run the full generated-CLI compile test (
TestGenerateProjectsCompile) to confirm end-to-end compilation with the template change.Assistant
All full-CLI compile tests pass. Let me run
gofmtcheck andgo veton the changed files.Assistant
The
.tmplerrors are expected (templates aren't valid Go). Let me confirm the.gofiles are gofmt-clean.Assistant
Everything is clean. Now let me remove my temporary test file before submitting (per the instructions, I must not add tests to the repository).
Assistant
Let me do a final build to confirm everything compiles after removing the temp test.
Assistant
Build and vet pass. Let me do a final review of the complete diff.
Assistant
git diff --statshowsgo.modmodified again (fromgo build -mod=mod). Let me check and revert it, since it's an unintended side-effect.Assistant
This
go.modchange is an unintended build side-effect. Let me revert it.Assistant
Now only my 3 source files are modified. Let me confirm the build works in readonly mode (without modifying go.mod).
Assistant
All changes are in place, the build is clean in readonly mode, and only my 3 source files are modified. Let me do a final summary review of the complete diff before submitting.
Assistant
The implementation is complete. Let me summarize the work and submit.
Summary
Problem: The typed-table emit path in the generator walks a resource's response schema and emits one SQLite column per leaf field with no upper bound. On wide config singletons (e.g., HaloPSA
/Controlwith 3,747 columns), the generatedCREATE TABLEexceeds SQLite's defaultSQLITE_MAX_COLUMN=2000, failing silently at firstsyncwithSQL logic error: too many columns on control (1).Fix (3 files):
-
internal/generator/schema_builder.go— Added theJSONOnlyflag toTableDef, themaxTypedTableColumns = 1500safety cap, and theapplyColumnCaphelper that resets an over-cap table in place to the JSON-only fallback shape (id/data/synced_at, no indexes, no FTS5) and flags itJSONOnly. -
internal/generator/generator.go— InschemaWithDependentParents(the single canonical schema-construction path), after dependent-parent (parent_id) columns are added, iterate every table and apply the cap so the count includes dependent-parent columns. Emit a generation-time stderr log linestore-fallback: <name> (<n> cols) -> JSON-only. -
internal/generator/templates/store.go.tmpl— Changed the per-resourceCREATE TABLEgate from{{if hasTypedTable .}}to{{if or (hasTypedTable .) .JSONOnly}}so the fallback table is retained (per the contract) with its 3 columns, while all typed surfaces (columns, indexes, FTS5, typed upsert,UpsertBatchdispatch case) stay suppressed becausehasTypedTableis false forJSONOnlytables (Columns <= 3).
Verification:
- Reproduced the exact issue error (
too many columns on control (1)) with the cap disabled; confirmed it's resolved with the cap enabled. - Positive criterion: a 3000-field
controlresource generates a 3-columnstore.gotable, no typed upsert/FTS/indexes, the generated CLI builds, andstore.Openmigration succeeds against an empty DB. - Negative criterion: an ordinary narrow
itemsresource still emits typed columns +upsertItemsTx+UpsertBatchdispatch case. - The fallback log line fires to stderr.
go build ./...,go vet,gofmtclean;schema_builder, store/FTS/typed/upsert/dependent tests pass; full-CLI compile tests (stytch/clerk/loops) pass.- Golden suite: identical pre-existing failure set with and without my change (the failures are environmental repo-path normalization artifacts); no new golden diffs introduced —
store.godiffs are purely the header URL normalization that also fails on the untouched baseline.
Checklist reasoning:
- Reproduced the error: Yes — synthetic 3000-field spec with cap disabled produced
SQL logic error: too many columns on control (1). - Confirmed fix resolves the error: Yes — with cap enabled, migration succeeds and the table has 3 columns.
- Edge cases considered: Yes — cap includes dependent-parent columns (applied post parent_id); sub-resource tables are tiny and unaffected; JSONOnly tables retain their CREATE TABLE but suppress typed columns/indexes/FTS/upsert/dispatch via the existing
hasTypedTablegating; backfill generation is a no-op for JSONOnly (only id/data/synced_at, all excluded); narrow resources unchanged; low-gravity "dead" tables still skipped (JSONOnly=false+hasTypedTable=false). - Ran affected existing tests: Yes — schema_builder, store/FTS/typed/upsert/dependent tests, full-CLI compile tests, and golden suite all pass/unchanged.
I removed the temporary verification test before submitting.
-
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.
Identical first prompt
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-27BCompare side by side →
Qwen/Qwen3.6-27BCompare side by side →