ISSUE DESCRIPTION generator: typed-table emit exceeds SQLite column cap on wide config…
- User
ISSUE DESCRIPTION
generator: typed-table emit exceeds SQLite column cap on wide config singletons
Filed by
/printing-press-retrofrom a run of HaloPSA (scorecard 91/100 Grade A, 100% verify pass).Summary
The typed-table emit path generates one column per leaf JSON field in a resource's response schema with no upper bound. On HaloPSA's tenant-global
/Controlconfig singleton, this produced a 3,747-columnCREATE TABLEthat exceeded SQLite's defaultSQLITE_MAX_COLUMN=2000. Generation passed every quality gate;sync --full --dry-rundied cold withSQL logic error: too many columns on control (1). The failure mode is silent until first sync.Where to look
- Component: generator
- Likely area: Typed-table-emit path in
internal/generator/templates/store/— whereverCREATE TABLEtext is composed per discovered resource. - Triggered when: A spec resource's response schema flattens to >~1500 leaf fields. Enterprise config singletons are the obvious class.
What we observed
Run-1 output (post-generate):
PASS go mod tidy / govulncheck / go vet / go build PASS halopsa-pp-cli --help / version / doctor Generated halopsa at .../working/halopsa-pp-cliFirst
sync --full --dry-run:Error: opening local database: running migrations: migration failed: SQL logic error: too many columns on control (1)Per-table column counts (top 5 widest, recorded by Python introspection of
internal/store/store.go):table columns control 3747 tickets 847 projects 847 opportunities 847 client 447 Tickets / projects / opportunities sit just under the 2000 cap but with no headroom; the next major-version Halo response schema could push them over.
Workaround applied this run: hand-stripped
/Control*paths from the spec (5 paths) and added a stubControlschema so residual$reftargets resolved. Regenerated successfully.Suspected root cause
Confident. The typed-table-emit path walks the resource's response schema and emits one column per leaf field with no width sanity check. The store package has no JSON-only fallback for resources whose schema exceeds a safe column count.
Suggested direction
In the typed-table-emit path, compute the column count before writing
CREATE TABLE. If the count exceeds a threshold (1500 leaves headroom under SQLite's default 2000; tunable), fall back to JSON-only schema:id TEXT PRIMARY KEY, data JSON NOT NULL, synced_at DATETIME. Emit a generation-time log line naming the resource and the fallback (e.g.,store-fallback: control (3747 cols) → JSON-only).Preserve typed-column behavior below the threshold — the typed columns power offline FTS5 and cross-entity SQL joins; the fallback should only fire on pathological widths.
Acceptance criteria
- positive: feed a synthetic spec where one resource has 3000+ leaf fields; generated
store.gofor that resource has only 3 columns; sync against an empty DB succeeds. - negative: feed a normal resource (<1500 cols); typed-table emission unchanged; flat columns still emitted.
Frequency
subclass:wide-config-singleton — complete-functional-failure when it fires.
Complexity
small (column-count check + JSON-only emit branch).
Scope boundary
Does NOT cover per-column truncation, schema-aware projection, or per-API column allowlists. Single threshold + JSON-only fallback only.
Dependencies
None.
Related issues
None — first sighting of this specific failure mode.
Artifacts
Artifact Link Retro document https://files.catbox.moe/ico9uo.md Manuscripts (research + proofs) https://files.catbox.moe/rn4hf4.zip Generated CLI source code https://files.catbox.moe/4v6myv.zip
Generated by
/printing-press-retro· CLI Printing PressEvaluation contract
Use 1500 columns as the safety cap for a generated typed SQLite resource table, including dependent-parent columns. Resources exceeding the cap must retain their per-resource table but fall back to a JSON-only schema containing
id,data, andsynced_at; do not generate typed-column indexes, FTS structures, or typed upsert code for that fallback table. Batch store generation must dispatch wide resources to the JSON-only path while preserving the existing typed behavior for ordinary narrow resources.Interface
No new interfaces are introduced.
ADDITIONAL ADVICE
Since you are given a git repository, you can use git commands to simplify your work. For example, if you made a mistake and want to revert your changes, you can use
git checkout <file>to restore the file to its original state. You can also reset all changes in the repository usinggit reset --hardcommand. Additionally, you can usegit stashandgit stash popto temporarily save your changes and restore them later. Finally, you can see the changes that you've made (which will be submitted when you callsubmit) usinggit statusorgit diffcommands. However you don't need to usegit addorgit commitbefore submitting your changes. In fact,submitonly submits currently unstaged changes because it usesgit diffto get the changes that need to be submitted.CHECKLIST
Before submitting your solution, please go over the following checklist and make sure you've done everything:
- If an error was reported in the issue description, I have successfully reproduced it.
- If an error was reported in the issue description, I have confirmed that my fix resolves the error.
- I have thought about edge cases and made sure my fix handles them as well.
- I have run existing tests in the repository that might have been affected by the change I made and confirmed that they pass. I want you to list every bullet from this checklist and write your reasoning for why you think you did it or didn't need to.
Repository has been uploaded and your shell is currently at the repository root. Time to solve the issue!
(Current directory: /cli-printing-press, current file: none) bash-$
- Assistant
I'll start by exploring the repository structure to understand the codebase and locate the relevant files mentioned in the issue. Let's first check what's in the repository root.
- Assistant
Let me explore the repository structure more systematically.
- Assistant
Let me explore the internal directory structure to find the generator templates.
- Assistant
Let me explore the generator directory to find the store templates.
- Assistant
Let me look at the templates directory to find store-related templates.
- Assistant
Let me look at the store templates specifically.
- Assistant
Let me look at the store template and the main generator code.
- Assistant
Let me look at the store template for CREATE TABLE generation.
- Assistant
Let me look at line 279 to see how the CREATE TABLE is generated for each resource.
- Assistant
Now I need to understand how the
.Tablesdata structure is built in the generator code. Let me search for where the Tables field is populated. code</tool_call> - Assistant
Let me search for the Tables field in the generator code to understand how it's populated.
- Assistant
Let me look at the TableDef structure and the schema building code.
- Assistant
Let me look for TableDef more broadly.
- Assistant
Let me look at the schema_builder.go file to understand the structure and how columns are built.
- Assistant
Now I understand the structure. Let me see the rest of this file to understand the gravity computation and how the columns are added.
- Assistant
Let me continue reading the schema_builder.go to understand the full picture.
- Assistant
Let me continue reading the file to understand the full structure.
- Assistant
Now let me look at
hasTypedTablein the template and the upsert code to understand how the dispatch works. - Assistant
Let me look at the
hasTypedTablefunction implementation. - Assistant
Let me also look at how the template handles the Upsert code to understand the typed vs JSON-only paths.
- Assistant
Let me look at the
domainUpsertMethodNamefunction. - Assistant
Now let me check how the existing tests work and understand the test structure for schema_builder.
- Assistant
Let me continue reading the schema_builder_test.go to understand how the existing tests work.
- Assistant
Let me continue reading the rest of the schema_builder_test.go.
- Assistant
Let me keep reading to see the findTable helper function and understand the pattern.
- Assistant
Now let me look at how the UpsertBatch path works and what
hasDomainUpsertdoes. - Assistant
Now let me look at the UpsertBatch template code to understand how it dispatches:
- Assistant
Let me look at the UpsertBatch dispatch to understand how typed vs JSON-only tables are handled.
- Assistant
Now let me look at
buildSubResourceTablesince sub-resources can also get columns. - Assistant
Let me look at the buildSubResourceTable function.
- Assistant
Now let me check how logging is done in the generator codebase to understand the pattern for emitting warnings.
- Assistant
Now I have a clear understanding of the code. Let me first look at the existing tests to see how they work before making changes.
- Assistant
Let me check the go environment.
- Assistant
GOPATH and GOMODCACHE are empty. Let me set them.
- Assistant
Let me try running the test again with the environment set.
- Assistant
Need to set GOCACHE and HOME too.
- Assistant
Let me now look at the typed upsert methods in the template to understand what they do for tables.
- Assistant
Let me look at the full migration section of the template more carefully.
- Assistant
Now let me look at the UpsertBatch dispatch and the hasDomainUpsert related code.
- Assistant
Let me look at the remaining template sections including ListField and other generated code.
- Assistant
Now let me look at the test that checks dead table skipping.
- Assistant
Now I have a clear picture. Let me plan my approach:
-
schema_builder.go: Add aJSONOnlyfield toTableDef, add aMaxTypedColumns = 1500constant, and after building typed columns, check if count exceeds the limit. If so, log a fallback message, setJSONOnly = true, reset columns tobaseTableColumns, and clear indexes/FTS. -
store.go.tmpl: Modify the CREATE TABLE block to emit the table for JSON-only tables too (but without indexes/FTS).
Let me first look at the template structure around the
hasTypedTablegating for the created table vs the Upsert code to make sure I understand the full scope. -
- Assistant
Let me now understand the full picture. I need to look at the
hasTypedTablefunction and the template structure to plan my changes. Let me check a few more details: - Assistant
Now I have a full understanding of the code. Let me implement the fix. My plan:
-
In
schema_builder.go: AddJSONOnly boolfield toTableDef, addmaxTypedColumnsconstant, and inBuildSchemawhen column count exceeds the threshold, setJSONOnly = trueand reset columns/FTS/indexes. -
In
generator.go: Add ahasPerResourceTabletemplate function that returns true for both typed tables AND JSON-only tables. -
In
store.go.tmpl: Change the CREATE TABLE gate fromhasTypedTabletohasPerResourceTable, keeping indexes/FTS insidehasTypedTableonly.
Let me start with the schema_builder.go changes:
-
- Assistant
Now let me implement the changes. First, let me add the
JSONOnlyfield toTableDefand themaxTypedColumnsconstant in schema_builder.go: - Assistant
Now I'll implement the changes. First, let me add the
JSONOnlyfield toTableDefand themaxTypedColumnsconstant: - Assistant
Let me use the correct edit syntax:
- Assistant
Let me use the correct syntax for the edit command:
- Assistant
I need to use the edit command with the correct syntax. Let me try the heredoc approach.
- Assistant
Let me try opening the file first, then editing.
- Assistant
Now let me add the
JSONOnlyfield toTableDefand themaxTypedColumnsconstant: - Assistant
Let me use sed instead to make the changes:
- Assistant
Let me verify the change was applied correctly.
- Assistant
Now let me add the
maxTypedColumnsconstant and the import forfmt(for logging): - Assistant
I need to add "fmt" and "os" to the imports and add the maxTypedColumns constant. Let me do this:
- Assistant
Let me verify the changes and add the maxTypedColumns constant:
- Assistant
Now let me add the
maxTypedColumnsconstant after thebaseTableColumnsvariable: - Assistant
Let me look at lines around 47-52 to add the maxTypedColumns constant:
- Assistant
Now let me add the
maxTypedColumnsconstant after line 51: - Assistant
Let me verify the changes:
- Assistant
Now I need to add the column count check in
BuildSchemaafter the high-gravity column loop. Let me find the right location: - Assistant
Let me see more context:
- Assistant
Now I need to add the JSON-only fallback check after line 127 (after the gravity >= 2 block). Let me insert it:
- Assistant
Let me verify the changes:
- Assistant
I notice two issues:
- The unicode arrow was written as literal bytes instead of the actual UTF-8 character
- The FTS5 logic still runs after the JSON-only check - I need to make sure FTS5 is not set for JSON-only tables
Let me fix both issues:
- Assistant
Let me check the actual content:
- Assistant
Good, the arrow is now correct. Now I need to fix the FTS5 condition to not set FTS5 for JSON-only tables:
- Assistant
I need to add
!table.JSONOnlyto the FTS5 condition to prevent FTS for JSON-only tables: - Assistant
Let me verify the change:
- Assistant
Good. Now I need to update the template and generator.go to handle JSON-only tables properly. Let me first add a
hasPerResourceTabletemplate function in generator.go: - Assistant
Let me look at the code around line 408 to understand how hasTypedTable is defined and add hasPerResourceTable right after it:
- Assistant
Now let me add the
hasPerResourceTablefunction right afterhasTypedTable: - Assistant
Let me verify the changes:
- Assistant
Good. Now I need to update the
store.go.tmpltemplate to usehasPerResourceTablefor the CREATE TABLE block, while keepinghasTypedTablefor the indexes and FTS5 sections. Let me look at the template structure: - Assistant
Now I need to modify the template. I'll replace the gate from
hasTypedTabletohasPerResourceTablefor the CREATE TABLE, and add a nestedhasTypedTablecheck for indexes and FTS5. Let me make this change: - Assistant
Let me verify the template changes:
- Assistant
The template changes look correct. Now let me verify the code compiles and run the existing tests:
- Assistant
Good, it compiles. Now let me run the existing schema_builder tests:
- Assistant
Good, all existing tests pass. Now let me run the store-related test:
- Assistant
Great, existing tests pass. Let me run a broader set of tests to make sure nothing is broken:
- Assistant
Let me run all generator tests to make sure nothing is broken:
- Assistant
Let me check if this test failure is pre-existing and not related to my changes:
- Assistant
This test failure is pre-existing - it's a govulncheck failure about a Go standard library vulnerability (
GO-2026-5856). Not related to my changes.Now let me write a reproduction test. Let me first check how the existing tests create synthetic specs:
- Assistant
Now let me first verify my changes are correct by running the existing tests (excluding the pre-existing govulncheck failure). Let me also check if there are any golden file tests that might need updating:
- Assistant
All existing tests pass. Now let me create a reproduction script to verify my fix works correctly. First, let me write a test script:
- Assistant
Wait, the instructions say I shouldn't add new tests to the repository. Let me remove the test file and instead run it from a temporary location:
- Assistant
Let me create a temporary test script outside the repository:
- Assistant
Now let me run this test script:
- Assistant
I need to put the test file inside the package. Let me create it as a test file inside the generator package temporarily:
- Assistant
Now let me run the test:
- Assistant
All three tests pass! The fix is working correctly. Now let me clean up the temporary test file:
- Assistant
Now let me run the broader test suite to make sure my changes don't break anything. Let me run some key tests:
- Assistant
All tests pass. Let me also check the templates test and the mutation_store_writeback test which are very relevant:
- Assistant
Let me also run a broader set of tests, especially the ones related to store generation:
- Assistant
All tests pass. Let me verify my changes are complete by reviewing the final state of each modified file:
- Assistant
Let me also check the generator.go changes and the store.go.tmpl changes:
- Assistant
Let me also verify the template changes are correct:
- Assistant
The template structure looks correct. Let me also run one more test to ensure the generated store.go compiles and the generated code is valid:
- Assistant
Let me run a broader set of tests excluding the known govulncheck failure:
- Assistant
Let me try a different approach:
- Assistant
Let me run more tests to ensure nothing is broken:
- Assistant
All tests pass. Let me now review my complete diff:
- Assistant
The diff looks clean and correct. Let me run a final comprehensive test to make sure everything works:
- Assistant
All tests pass. Let me also check if there are golden file tests that might need updating:
- Assistant
Let me check the golden file to make sure it's not affected by my changes:
- Assistant
Let me run the golden file tests to make sure they still pass:
- Assistant
Let me look for where the golden tests are:
- Assistant
Let me check if there are any golden tests in the main test suite:
- Assistant
Let me verify my diff one final time and make sure it's clean:
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.
Identical first prompt
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-27BCompare side by side →
Qwen/Qwen3.6-27BCompare side by side →