ISSUE DESCRIPTION generator: typed-table emit exceeds SQLite column cap on wide config…
- User
ISSUE DESCRIPTION
generator: typed-table emit exceeds SQLite column cap on wide config singletons
Filed by
/printing-press-retrofrom a run of HaloPSA (scorecard 91/100 Grade A, 100% verify pass).Summary
The typed-table emit path generates one column per leaf JSON field in a resource's response schema with no upper bound. On HaloPSA's tenant-global
/Controlconfig singleton, this produced a 3,747-columnCREATE TABLEthat exceeded SQLite's defaultSQLITE_MAX_COLUMN=2000. Generation passed every quality gate;sync --full --dry-rundied cold withSQL logic error: too many columns on control (1). The failure mode is silent until first sync.Where to look
- Component: generator
- Likely area: Typed-table-emit path in
internal/generator/templates/store/— whereverCREATE TABLEtext is composed per discovered resource. - Triggered when: A spec resource's response schema flattens to >~1500 leaf fields. Enterprise config singletons are the obvious class.
What we observed
Run-1 output (post-generate):
PASS go mod tidy / govulncheck / go vet / go build PASS halopsa-pp-cli --help / version / doctor Generated halopsa at .../working/halopsa-pp-cliFirst
sync --full --dry-run:Error: opening local database: running migrations: migration failed: SQL logic error: too many columns on control (1)Per-table column counts (top 5 widest, recorded by Python introspection of
internal/store/store.go):table columns control 3747 tickets 847 projects 847 opportunities 847 client 447 Tickets / projects / opportunities sit just under the 2000 cap but with no headroom; the next major-version Halo response schema could push them over.
Workaround applied this run: hand-stripped
/Control*paths from the spec (5 paths) and added a stubControlschema so residual$reftargets resolved. Regenerated successfully.Suspected root cause
Confident. The typed-table-emit path walks the resource's response schema and emits one column per leaf field with no width sanity check. The store package has no JSON-only fallback for resources whose schema exceeds a safe column count.
Suggested direction
In the typed-table-emit path, compute the column count before writing
CREATE TABLE. If the count exceeds a threshold (1500 leaves headroom under SQLite's default 2000; tunable), fall back to JSON-only schema:id TEXT PRIMARY KEY, data JSON NOT NULL, synced_at DATETIME. Emit a generation-time log line naming the resource and the fallback (e.g.,store-fallback: control (3747 cols) → JSON-only).Preserve typed-column behavior below the threshold — the typed columns power offline FTS5 and cross-entity SQL joins; the fallback should only fire on pathological widths.
Acceptance criteria
- positive: feed a synthetic spec where one resource has 3000+ leaf fields; generated
store.gofor that resource has only 3 columns; sync against an empty DB succeeds. - negative: feed a normal resource (<1500 cols); typed-table emission unchanged; flat columns still emitted.
Frequency
subclass:wide-config-singleton — complete-functional-failure when it fires.
Complexity
small (column-count check + JSON-only emit branch).
Scope boundary
Does NOT cover per-column truncation, schema-aware projection, or per-API column allowlists. Single threshold + JSON-only fallback only.
Dependencies
None.
Related issues
None — first sighting of this specific failure mode.
Artifacts
Artifact Link Retro document https://files.catbox.moe/ico9uo.md Manuscripts (research + proofs) https://files.catbox.moe/rn4hf4.zip Generated CLI source code https://files.catbox.moe/4v6myv.zip
Generated by
/printing-press-retro· CLI Printing PressEvaluation contract
Use 1500 columns as the safety cap for a generated typed SQLite resource table, including dependent-parent columns. Resources exceeding the cap must retain their per-resource table but fall back to a JSON-only schema containing
id,data, andsynced_at; do not generate typed-column indexes, FTS structures, or typed upsert code for that fallback table. Batch store generation must dispatch wide resources to the JSON-only path while preserving the existing typed behavior for ordinary narrow resources.Interface
No new interfaces are introduced.
ADDITIONAL ADVICE
Since you are given a git repository, you can use git commands to simplify your work. For example, if you made a mistake and want to revert your changes, you can use
git checkout <file>to restore the file to its original state. You can also reset all changes in the repository usinggit reset --hardcommand. Additionally, you can usegit stashandgit stash popto temporarily save your changes and restore them later. Finally, you can see the changes that you've made (which will be submitted when you callsubmit) usinggit statusorgit diffcommands. However you don't need to usegit addorgit commitbefore submitting your changes. In fact,submitonly submits currently unstaged changes because it usesgit diffto get the changes that need to be submitted.CHECKLIST
Before submitting your solution, please go over the following checklist and make sure you've done everything:
- If an error was reported in the issue description, I have successfully reproduced it.
- If an error was reported in the issue description, I have confirmed that my fix resolves the error.
- I have thought about edge cases and made sure my fix handles them as well.
- I have run existing tests in the repository that might have been affected by the change I made and confirmed that they pass. I want you to list every bullet from this checklist and write your reasoning for why you think you did it or didn't need to.
Repository has been uploaded and your shell is currently at the repository root. Time to solve the issue!
(Current directory: /cli-printing-press, current file: none) bash-$
- Assistant
I’ll first map the generator/store implementation and existing tests, then identify how typed resources, indexes, FTS, and upserts are selected so the fallback is applied consistently across all generated code paths.
- Assistant
The store is a single template driven by table metadata. I’ll inspect that metadata construction and every template loop keyed on table eligibility; the cleanest fix should mark wide tables once during generation and let all schema/upsert/index/FTS branches follow that classification.
- Assistant
A key existing invariant is that ordinary three-column “low-gravity” definitions are intentionally not emitted, while a wide fallback must be emitted despite also having three columns. I’ll add explicit fallback metadata rather than infer it from column length. The 1500 cap applies to the final column list (therefore includes base and dependent-parent columns), and fallback metadata will suppress typed indexes, FTS, helpers, dispatch, and tests while retaining DDL. Next I’ll inspect sub-resource construction and generation’s logging conventions.
- Assistant
BuildSchemacurrently has no output channel, and changing its public signature would violate the stated interface constraint. I’ll keep classification pure there and emit the requested generation-time line in the existing generation flow after schema construction, using the generator’s configured writer/logging mechanism if available. - Assistant
The evaluation wording adds an important requirement beyond merely surviving migrations: batch writes must dispatch wide resources to a JSON-only per-resource path. I’ll implement one reusable JSON-only transaction helper (not a generated typed helper) and route fallback cases to it, preserving the generic
resourcesrow as today. Single-object sync will likewise write the fallback table through a generic JSON-only method. No resource-specific typed upsert, index, FTS, or search code will be emitted. - Assistant
I’ll now add the explicit metadata and finalization helper. The fallback is triggered only when the final SQLite column count is strictly greater than 1500 (1500 remains typed), records the pre-fallback count for diagnostics, and resets all typed/index/FTS metadata atomically. The finalizer is idempotent so it can run both in
BuildSchemaand after dependent-parent augmentation. - Assistant
Next I’ll update the store template: DDL uses
hasStoreTable, while all typed loops remain gated byhasTypedTable. A generated resource-to-SQL switch will safely use fixed quoted table names, avoiding dynamic identifiers. Batch projections use the existing savepoint isolation for both typed and JSON-only tables. - Assistant
To preserve byte-level behavior for normal resources as much as possible, the JSON-only helper and generic
Upserthook will be emitted only when at least one fallback exists. Existing typed variable/helper names remain untouched, avoiding collateral changes to established generated tests; the fallback branch itself calls only the generic JSON projection helper. - Assistant
The implementation paths are in place. Before broad tests, I’ll run the focused schema-builder and generator suites to catch template parsing, formatting, and assumptions about unchanged narrow output.
- Assistant
The focused tests did not start because this environment has neither
GOPATHnorGOMODCACHEconfigured. I’ll rerun with an isolated temporary GOPATH (without altering repository files), allowing Go to use/download the declared dependencies. - Assistant
The focused run reached and passed generation/template work but one broad
TestGenerateWithNoAuthfailed solely because the installed Go 1.26.4 standard library is flagged bygovulncheck(fixed in 1.26.5); this is environmental and unrelated. I’ll now create a temporary, removed-before-submit test in the generator package to reproduce the 3000-field case end-to-end: generate source, assert three-column DDL/no typed artifacts, compile generated store, open SQLite, batch upsert, and verify the fallback row. - Assistant
I can reuse the package’s existing
minimalSpectest helper andrunGoCommand, then augment it with one wide response type/resource. I’ll keep this as a temporary verification artifact and remove it immediately after the run. - Assistant
I’ll implement the temporary test now. To avoid unrelated full-generation quality gates, it will exercise
BuildSchema, render only store/sync files through the package-private methods, create a tiny temporary generated module, and run a real SQLite store test. - Assistant
The temporary test will set an empty profiler profile, render the store package, and use the repository’s own dependency versions in a minimal module. It will explicitly verify migration and
UpsertBatch, not merely inspect strings. - Assistant
The first synthetic run reproduced the generated fallback correctly but my temporary assertion counted a template whitespace line as a fourth “column,” so it stopped before runtime verification. I’ll rerun with structural column-name assertions and then execute the SQLite test.
- Assistant
Migration succeeded, but the temporary runtime fixture accidentally emitted backslashes into the JSON payload (
{\"id\":...}), so ID extraction correctly rejected invalid JSON and no row was written. I’ll correct the fixture to use a Go raw string JSON literal and rerun the same real database check. - Assistant
The 3001-leaf reproduction now passes end-to-end: generated DDL has only
id/data/synced_at, generated store compiles, empty SQLite opens/migrates, batch upsert succeeds, and the per-resource fallback table contains the row. I’ll inspect remaining table-dependent templates for any unguarded assumptions and clean up wording before running the full relevant package tests. - Assistant
No additional unguarded typed assumptions affect fallback tables: schema-version backfill tests only emit when non-base columns exist, and search loops require
.FTS5(cleared by fallback). I also noticed test execution wrote into the repository’s tracked.gocache; I’ll restore/clean only those test artifacts, update the stale DDL comment to describe both typed and fallback tables, then run focused tests with/tmpas the cache to avoid repository noise. - Assistant
I’ll run the complete schema-builder tests plus the existing store/typed-dispatch regression tests and template-format tests. These directly cover the negative criterion (normal resources retain typed columns/upserts/FTS) and ensure the new condition did not alter ordinary dispatch.
- Assistant
While reviewing the dependent edge case, I found an important flaw: a table already collapsed by
BuildSchemacould haveparent_idappended afterward, yielding four fallback columns. I’ll preserve whether the original wide schema already had that column, then count (but never physically append) a newly required dependent-parent column for an existing fallback. This keeps the DDL exactly three columns and makes the diagnostic’s original width include the dependent column. Tables that reach 1501 only after augmentation continue to collapse in the final pass. - Assistant
The dependent edge is now handled idempotently. I’ll rerun the relevant generator suite and OpenAPI parser tests (which consume
BuildSchema) using only temporary caches, then rungo veton the changed package. - Assistant
I’ll perform one final temporary boundary check for the exact contract: 1500 total columns remains typed; 1501 collapses; and a 1500-column table that gains a dependent
parent_idalso collapses to exactly three columns with no indexes. - Assistant
The focused suites and end-to-end reproduction cover the requested behavior. I’ll now review the final patch for accidental changes, verify no temporary artifacts remain, and run
git diff --checkplus a final package compile test. - Assistant
The final diff is clean and compiles. I’ll make one small documentation cleanup so the cap comment is attached to the cap constant rather than sharing the base-column comment, then rerun formatting/diff checks and submit.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.
Identical first prompt
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-27BCompare side by side →
Qwen/Qwen3.6-27BCompare side by side →