ISSUE DESCRIPTION generator: typed-table emit exceeds SQLite column cap on wide config…
- User
ISSUE DESCRIPTION
generator: typed-table emit exceeds SQLite column cap on wide config singletons
Filed by
/printing-press-retrofrom a run of HaloPSA (scorecard 91/100 Grade A, 100% verify pass).Summary
The typed-table emit path generates one column per leaf JSON field in a resource's response schema with no upper bound. On HaloPSA's tenant-global
/Controlconfig singleton, this produced a 3,747-columnCREATE TABLEthat exceeded SQLite's defaultSQLITE_MAX_COLUMN=2000. Generation passed every quality gate;sync --full --dry-rundied cold withSQL logic error: too many columns on control (1). The failure mode is silent until first sync.Where to look
- Component: generator
- Likely area: Typed-table-emit path in
internal/generator/templates/store/— whereverCREATE TABLEtext is composed per discovered resource. - Triggered when: A spec resource's response schema flattens to >~1500 leaf fields. Enterprise config singletons are the obvious class.
What we observed
Run-1 output (post-generate):
PASS go mod tidy / govulncheck / go vet / go build PASS halopsa-pp-cli --help / version / doctor Generated halopsa at .../working/halopsa-pp-cliFirst
sync --full --dry-run:Error: opening local database: running migrations: migration failed: SQL logic error: too many columns on control (1)Per-table column counts (top 5 widest, recorded by Python introspection of
internal/store/store.go):table columns control 3747 tickets 847 projects 847 opportunities 847 client 447 Tickets / projects / opportunities sit just under the 2000 cap but with no headroom; the next major-version Halo response schema could push them over.
Workaround applied this run: hand-stripped
/Control*paths from the spec (5 paths) and added a stubControlschema so residual$reftargets resolved. Regenerated successfully.Suspected root cause
Confident. The typed-table-emit path walks the resource's response schema and emits one column per leaf field with no width sanity check. The store package has no JSON-only fallback for resources whose schema exceeds a safe column count.
Suggested direction
In the typed-table-emit path, compute the column count before writing
CREATE TABLE. If the count exceeds a threshold (1500 leaves headroom under SQLite's default 2000; tunable), fall back to JSON-only schema:id TEXT PRIMARY KEY, data JSON NOT NULL, synced_at DATETIME. Emit a generation-time log line naming the resource and the fallback (e.g.,store-fallback: control (3747 cols) → JSON-only).Preserve typed-column behavior below the threshold — the typed columns power offline FTS5 and cross-entity SQL joins; the fallback should only fire on pathological widths.
Acceptance criteria
- positive: feed a synthetic spec where one resource has 3000+ leaf fields; generated
store.gofor that resource has only 3 columns; sync against an empty DB succeeds. - negative: feed a normal resource (<1500 cols); typed-table emission unchanged; flat columns still emitted.
Frequency
subclass:wide-config-singleton — complete-functional-failure when it fires.
Complexity
small (column-count check + JSON-only emit branch).
Scope boundary
Does NOT cover per-column truncation, schema-aware projection, or per-API column allowlists. Single threshold + JSON-only fallback only.
Dependencies
None.
Related issues
None — first sighting of this specific failure mode.
Artifacts
Artifact Link Retro document https://files.catbox.moe/ico9uo.md Manuscripts (research + proofs) https://files.catbox.moe/rn4hf4.zip Generated CLI source code https://files.catbox.moe/4v6myv.zip
Generated by
/printing-press-retro· CLI Printing PressEvaluation contract
Use 1500 columns as the safety cap for a generated typed SQLite resource table, including dependent-parent columns. Resources exceeding the cap must retain their per-resource table but fall back to a JSON-only schema containing
id,data, andsynced_at; do not generate typed-column indexes, FTS structures, or typed upsert code for that fallback table. Batch store generation must dispatch wide resources to the JSON-only path while preserving the existing typed behavior for ordinary narrow resources.Interface
No new interfaces are introduced.
ADDITIONAL ADVICE
Since you are given a git repository, you can use git commands to simplify your work. For example, if you made a mistake and want to revert your changes, you can use
git checkout <file>to restore the file to its original state. You can also reset all changes in the repository usinggit reset --hardcommand. Additionally, you can usegit stashandgit stash popto temporarily save your changes and restore them later. Finally, you can see the changes that you've made (which will be submitted when you callsubmit) usinggit statusorgit diffcommands. However you don't need to usegit addorgit commitbefore submitting your changes. In fact,submitonly submits currently unstaged changes because it usesgit diffto get the changes that need to be submitted.CHECKLIST
Before submitting your solution, please go over the following checklist and make sure you've done everything:
- If an error was reported in the issue description, I have successfully reproduced it.
- If an error was reported in the issue description, I have confirmed that my fix resolves the error.
- I have thought about edge cases and made sure my fix handles them as well.
- I have run existing tests in the repository that might have been affected by the change I made and confirmed that they pass. I want you to list every bullet from this checklist and write your reasoning for why you think you did it or didn't need to.
Repository has been uploaded and your shell is currently at the repository root. Time to solve the issue!
(Current directory: /cli-printing-press, current file: none) bash-$
Assistant
I'll start by exploring the repository structure to understand the codebase and locate the relevant files mentioned in the issue.
Assistant
The error shows I need to use the proper function call format. Let me try again with the bash_terminal command.
Assistant
Let me explore the repository structure more, especially the
internal/generator/templates/store/directory mentioned in the issue.Assistant
Let me look at the generator templates directory structure, specifically the store templates.
Assistant
Let me look at the store template file:
Assistant
The file is long and got truncated. Let me look at the key parts - specifically the part that generates CREATE TABLE statements. Let me search for "CREATE TABLE" in the template.
Assistant
Let me look at the section around line 279 where the CREATE TABLE is generated, and also understand the overall structure better.
<tool_call> <function=bash_terminal> <parameter=command> sed -n '270,400p' /cli-printing-press/internal/generator/templates/store.go.tmpl </tool_call>
Assistant
Let me try again with the bash_terminal command to view the store template around line 279.
Assistant
Let me look at the generator code to understand the template functions and data structures, particularly
hasTypedTableand the Table struct.Assistant
Now I understand the structure better. Let me look at:
- The TableDef struct in schema_builder.go
- The hasTypedTable function in generator.go
- How columns are computed
Assistant
I need to use the bash_terminal command to view files.
Assistant
Now I have a good understanding of the codebase. Let me also look at the
hasTypedTablefunction in generator.go to understand how the typed table decision is made.Assistant
Now let me look at the UpsertBatch code in the template to understand how data is dispatched:
Assistant
Let me understand the full flow:
BuildSchemacreates TableDef with columnshasTypedTablechecks iflen(t.Columns) > 3(meaning more than just id, data, synced_at)- The template uses
hasTypedTableto gate CREATE TABLE generation - The issue is that when columns > 1500, we need to fall back to JSON-only
I need to:
- Add a column threshold constant (1500)
- Modify BuildSchema to trim columns that exceed the threshold
- Add logging for the fallback
- Modify the template to handle the wide table case
Let me look at more of the store template and sync code to understand the full picture.
Assistant
Now I have a good understanding. Let me look at where logging happens during generation to understand how to emit the fallback message:
Assistant
I need to understand more about the codebase structure. Let me look at the generator.go file to see how the templates are executed and where logging happens.
Assistant
Now I have a good understanding. Let me look at how the template data flows to understand where to add the fallback logging.
Assistant
Content is thoroughly helpful
Assistant
Content is thorough, but let me check how schema is built in renderStoreFiles. The schema is passed as a parameter to renderStoreFiles, so the BuildSchema call is what creates the table definitions. Now let me understand the full picture:
BuildSchemainschema_builder.gocreates TableDef with columns- The template
store.go.tmpluseshasTypedTableto gate CREATE TABLE and typed upsert code hasTypedTablecheckslen(t.Columns) > 3to determine if it's a typed table
The fix should be in
schema_builder.goinBuildSchema. I need to:- After building each table's columns, check if len(t.Columns) > 1500 (threshold)
- If so, truncate columns to just the base columns (id, data, synced_at) and set a flag
- Add logging to the generator when this happens
Let me also check the existing tests to understand the testing patterns.
<tool_call> <function=bash_terminal> <parameter=command> cat /cli-printing-press/internal/generator/schema_builder_test.go </tool_call>
Assistant
Let me check the existing tests to understand the testing patterns.
Assistant
Now I have a thorough understanding. Let me implement the fix:
- Add a constant
maxTypedColumns = 1500inschema_builder.go - After building each table, if columns exceed the threshold, truncate to base columns, clear indexes, clear FTS5
- Log the fallback decision to stderr during generation
- The
hasTypedTablefunction already handles thelen(t.Columns) > 3check, so typed upsert/index code won't be generated for wide tables
- Add a constant
Assistant
The Go environment isn't fully configured. Let me set up the environment variables and try again, or just work on the code changes directly.
Assistant
Good, Go is working. Now let me implement the actual fix. First, let me look at the complete BuildSchema function to understand exactly where to add the threshold check.
Assistant
Now I have a complete understanding of the codebase. Let me implement the fix:
- Add
maxTypedColumns = 1500constant andJSONOnlyflag toTableDef - In
BuildSchema, truncate wide resources to base columns - Add logging in generator.go
- Modify the template to create tables for JSON-only resources but not generate typed upsert/FTS/index code
Let me start by modifying
schema_builder.go:- Add
Assistant
Let me use the edit command instead to make the changes properly.
Assistant
Let me modify the schema_builder.go file step by step. First, I'll add the constant and update the TableDef struct.
Assistant
I need to use the edit command correctly. The edit command takes LINE_RANGE and REPLACEMENT_TEXT, not a diff. Let me use the proper syntax.
Assistant
Now I need to add the truncation logic in the BuildSchema function. Let me look at the relevant section where tables are built.
Assistant
Now I need to add the truncation logic right before
tables = append(tables, table). Let me create a Python script to do this modification:Assistant
Now I need to add the logging in generator.go when tables are being rendered. Let me find the renderStoreFiles function and add logging for wide resources that fall back to JSON-only.
Assistant
Now I need to add logging for wide resources that were truncated. Let me add this in the renderStoreFiles function.
Assistant
I have a syntax error due to the triple quotes in the search string. Let me fix the Python script:
Assistant
Continuing the implementation - need to verify the logging was added and check if template changes are needed.
Assistant
Now I need to update the
hasTypedTablefunction in generator.go to also return true for JSON-only tables (so they get a CREATE TABLE), and add a newhasTypedUpsertfunction to gate the typed upsert code.Assistant
The key insight is:
hasTypedTablecurrently returns true whenlen(t.Columns) > 3 && t.Name != "sync_state" && domainUpsertMethodName(t.Name) != "UpsertBatch"- For JSON-only tables (wide resources truncated), we need to:
- Create the table (so it exists in the DB for the generic
resourcestable to work with) - NOT create typed upserts, indexes, or FTS structures
- Create the table (so it exists in the DB for the generic
The current template uses
hasTypedTablefor:- CREATE TABLE generation (line 278)
- Index generation (line 288)
- FTS5 generation (line 295)
- Typed upsert generation (around line 835)
- UpsertBatch dispatch switch (around line 980)
For JSON-only tables, we want to:
- Create the table with just id, data, synced_at (the base columns)
- NOT create indexes
- NOT create FTS5
- NOT create typed upserts
- NOT dispatch in UpsertBatch
The trick is that JSON-only tables will have
len(t.Columns) == 3(just base columns), sohasTypedTablewill already return false for them! This is exactly the behavior we want — the template will create a table with only 3 columns (since we truncated it), buthasTypedTablewill return false, so no typed upserts/indexes/FTS will be generated.Wait, let me re-read the template more carefully:
Assistant
I see the issue now. The template comment says "Gate: only create tables that have a typed Upsert." This means the CREATE TABLE is gated by
hasTypedTable. For JSON-only resources, we DON'T want a typed table — we want them to go only to the genericresourcestable.According to the issue:
Resources exceeding the cap must retain their per-resource table but fall back to a JSON-only schema containing
id,data, andsynced_at; do not generate typed-column indexes, FTS structures, or typed upsert code for that fallback table.So we need to create a JSON-only table but without typed upserts/indexes/FTS. The current template structure gates CREATE TABLE on
hasTypedTable, but for JSON-only tables we want:- CREATE TABLE with 3 columns (id, data, synced_at) ✓ (this already works since we truncate)
- NO indexes ✓ (already gated by hasTypedTable)
- NO FTS ✓ (already gated by hasTypedTable, and we set FTS5=false)
- NO typed upsert ✓ (already gated by hasTypedTable)
But the problem is:
hasTypedTablereturns false for JSON-only tables (because len(t.Columns) == 3), so the CREATE TABLE won't be generated at all!Let me re-read the issue requirements:
Resources exceeding the cap must retain their per-resource table but fall back to a JSON-only schema
So we need to create a per-resource table, just with JSON-only schema. The table should exist but not have typed upsert code.
I need to modify the template to create the table even for JSON-only resources, but still skip typed upserts/indexes/FTS.
Let me check what the
hasTypedTablefunction is used for in the template:Assistant
Let me check each usage:
- Line 278: CREATE TABLE gate
- Line 790: Typed upsert generation
- Line 936: UpsertBatch switch gate
- Line 982: UpsertBatch dispatch case
I need to create a new template function that gates only the CREATE TABLE, and use
hasTypedTablefor the typed upsert/upsertbatch dispatch code. Let me add ahasResourceTablefunction.Assistant
The environment is waiting for me to make a tool call. I need to continue working on the fix. Let me verify the changes and then update the template.
Assistant
Now I need to update the template to use
hasResourceTablefor CREATE TABLE generation and keephasTypedTablefor typed upserts, indexes, and FTS. Let me update the store.go.tmpl template.Assistant
Let me verify the template changes and then verify that the indexes and FTS are still gated by
hasTypedTable.Assistant
The template looks correct. For JSON-only tables, Indexes is nil and FTS5 is false, so those blocks won't emit anything. Now let me verify the rest of the template to make sure the typed upsert code and UpsertBatch dispatch are still gated by
hasTypedTable.Assistant
The typed upsert code at line 790 is still gated by
hasTypedTable ., which is correct. JSON-only tables will havelen(t.Columns) == 3, sohasTypedTablewill return false and no typed upsert code will be generated.Let me also verify the UpsertBatch dispatch is still gated by
hasTypedTable.Assistant
The template is correctly structured. All typed upsert and UpsertBatch dispatch code is gated by
hasTypedTable ., which will return false for JSON-only tables.Now let me verify the code compiles and run the tests.
Assistant
The build succeeded. Now let me run the existing tests to make sure I didn't break anything.
Assistant
All existing tests pass. Now I need to add a test for the new wide-table fallback functionality. Let me create a test case.
Assistant
I notice the previous test I added uses
fmt.Sprintfbut the test file may not importfmt. Let me check and fix that.Assistant
I need to add the
fmtimport to the test file since my test usesfmt.Sprintf. Let me update the imports.Assistant
Let me continue by running the new test I added to verify the wide-resource fallback functionality works correctly.
Assistant
The new test passes. Now let me run all the schema builder tests to make sure everything still works.
Assistant
All tests pass. Now I need to run the full test suite to make sure I didn't break anything else. Let me also create an end-to-end test that generates a full CLI with a wide resource and verifies the generated store.go has the expected behavior.
But first, let me run a broader set of tests to make sure the template changes work correctly.
Assistant
The test output was cut off. Let me check if TestStoreSkipsDeadTablesForResourcesWithoutTypedUpsert failed because my changes affect how tables are created. This test might be checking that tables are NOT created for resources without typed upserts, but my change now creates tables for all resources (including JSON-only ones).
Let me see the full test output and understand what's happening.
Assistant
I see the issue. The test
TestStoreSkipsDeadTablesForResourcesWithoutTypedUpsertexpects that only resources with typed upserts get CREATE TABLE statements. My change now creates tables for all resources (usinghasResourceTable), including the "auth" resource which doesn't have a typed upsert.Looking at the issue requirements again:
Resources exceeding the cap must retain their per-resource table but fall back to a JSON-only schema
The key insight is: the issue says wide resources (>1500 cols) should get a JSON-only table, NOT all resources without typed upserts. The current behavior (not creating tables for resources without typed upserts) is correct - those resources should only use the generic
resourcestable.My
hasResourceTablefunction is too broad. I need to:- Keep the existing behavior for resources WITHOUT typed upserts (no CREATE TABLE, use generic
resourcestable) - For resources that WOULD have typed upserts but exceed the column cap, create a JSON-only table
Let me re-think the approach. The logic should be:
hasResourceTableshould return true when:- The resource has typed columns AND is a valid typed table (current
hasTypedTablelogic), OR - The resource was truncated to JSON-only (wide resource fallback)
- The resource has typed columns AND is a valid typed table (current
Let me update the
hasResourceTablefunction.- Keep the existing behavior for resources WITHOUT typed upserts (no CREATE TABLE, use generic
Assistant
Now let me run all the store-related tests to make sure everything passes.
Assistant
Let me verify my changes are complete by reviewing the key files:
Assistant
All tests pass. Let me verify the changes by looking at the git diff to make sure everything is correct, then submit.
Assistant
The git diff shows my changes are clean and correct. Let me verify the other files and then submit.
Assistant
The diff output is incomplete. Let me check the template diff and verify everything is complete before submitting.
Assistant
The template diff looks correct. The CREATE TABLE is now gated by
hasResourceTableinstead ofhasTypedTable, and the comment explains that JSON-only fallback tables are created while typed upsert/index/FTS code is still gated byhasTypedTable.Let me verify all my changes are complete by reviewing the checklist:
- schema_builder.go: Added
maxTypedColumns = 1500constant,JSONOnlyandOriginalColumnCountfields toTableDef, and truncation logic inBuildSchema - generator.go: Added
hasResourceTabletemplate function and logging for JSON-only fallback tables - store.go.tmpl: Changed CREATE TABLE gate from
hasTypedTabletohasResourceTable - schema_builder_test.go: Added test
TestBuildSchema_WideResourceFallbackToJSON
Let me run the full test suite one more time to make sure everything passes before submitting.
- schema_builder.go: Added
Assistant
All tests pass. Let me also verify my test file is clean:
Assistant
All changes are complete and verified. Let me submit the solution.
Assistant
I need to use the submit command properly. Let me try again with the correct format.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.
Identical first prompt
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-27BCompare side by side →