ISSUE DESCRIPTION generator: typed-table emit exceeds SQLite column cap on wide config…
- User
ISSUE DESCRIPTION
generator: typed-table emit exceeds SQLite column cap on wide config singletons
Filed by
/printing-press-retrofrom a run of HaloPSA (scorecard 91/100 Grade A, 100% verify pass).Summary
The typed-table emit path generates one column per leaf JSON field in a resource's response schema with no upper bound. On HaloPSA's tenant-global
/Controlconfig singleton, this produced a 3,747-columnCREATE TABLEthat exceeded SQLite's defaultSQLITE_MAX_COLUMN=2000. Generation passed every quality gate;sync --full --dry-rundied cold withSQL logic error: too many columns on control (1). The failure mode is silent until first sync.Where to look
- Component: generator
- Likely area: Typed-table-emit path in
internal/generator/templates/store/— whereverCREATE TABLEtext is composed per discovered resource. - Triggered when: A spec resource's response schema flattens to >~1500 leaf fields. Enterprise config singletons are the obvious class.
What we observed
Run-1 output (post-generate):
PASS go mod tidy / govulncheck / go vet / go build PASS halopsa-pp-cli --help / version / doctor Generated halopsa at .../working/halopsa-pp-cliFirst
sync --full --dry-run:Error: opening local database: running migrations: migration failed: SQL logic error: too many columns on control (1)Per-table column counts (top 5 widest, recorded by Python introspection of
internal/store/store.go):table columns control 3747 tickets 847 projects 847 opportunities 847 client 447 Tickets / projects / opportunities sit just under the 2000 cap but with no headroom; the next major-version Halo response schema could push them over.
Workaround applied this run: hand-stripped
/Control*paths from the spec (5 paths) and added a stubControlschema so residual$reftargets resolved. Regenerated successfully.Suspected root cause
Confident. The typed-table-emit path walks the resource's response schema and emits one column per leaf field with no width sanity check. The store package has no JSON-only fallback for resources whose schema exceeds a safe column count.
Suggested direction
In the typed-table-emit path, compute the column count before writing
CREATE TABLE. If the count exceeds a threshold (1500 leaves headroom under SQLite's default 2000; tunable), fall back to JSON-only schema:id TEXT PRIMARY KEY, data JSON NOT NULL, synced_at DATETIME. Emit a generation-time log line naming the resource and the fallback (e.g.,store-fallback: control (3747 cols) → JSON-only).Preserve typed-column behavior below the threshold — the typed columns power offline FTS5 and cross-entity SQL joins; the fallback should only fire on pathological widths.
Acceptance criteria
- positive: feed a synthetic spec where one resource has 3000+ leaf fields; generated
store.gofor that resource has only 3 columns; sync against an empty DB succeeds. - negative: feed a normal resource (<1500 cols); typed-table emission unchanged; flat columns still emitted.
Frequency
subclass:wide-config-singleton — complete-functional-failure when it fires.
Complexity
small (column-count check + JSON-only emit branch).
Scope boundary
Does NOT cover per-column truncation, schema-aware projection, or per-API column allowlists. Single threshold + JSON-only fallback only.
Dependencies
None.
Related issues
None — first sighting of this specific failure mode.
Artifacts
Artifact Link Retro document https://files.catbox.moe/ico9uo.md Manuscripts (research + proofs) https://files.catbox.moe/rn4hf4.zip Generated CLI source code https://files.catbox.moe/4v6myv.zip
Generated by
/printing-press-retro· CLI Printing PressEvaluation contract
Use 1500 columns as the safety cap for a generated typed SQLite resource table, including dependent-parent columns. Resources exceeding the cap must retain their per-resource table but fall back to a JSON-only schema containing
id,data, andsynced_at; do not generate typed-column indexes, FTS structures, or typed upsert code for that fallback table. Batch store generation must dispatch wide resources to the JSON-only path while preserving the existing typed behavior for ordinary narrow resources.Interface
No new interfaces are introduced.
ADDITIONAL ADVICE
Since you are given a git repository, you can use git commands to simplify your work. For example, if you made a mistake and want to revert your changes, you can use
git checkout <file>to restore the file to its original state. You can also reset all changes in the repository usinggit reset --hardcommand. Additionally, you can usegit stashandgit stash popto temporarily save your changes and restore them later. Finally, you can see the changes that you've made (which will be submitted when you callsubmit) usinggit statusorgit diffcommands. However you don't need to usegit addorgit commitbefore submitting your changes. In fact,submitonly submits currently unstaged changes because it usesgit diffto get the changes that need to be submitted.CHECKLIST
Before submitting your solution, please go over the following checklist and make sure you've done everything:
- If an error was reported in the issue description, I have successfully reproduced it.
- If an error was reported in the issue description, I have confirmed that my fix resolves the error.
- I have thought about edge cases and made sure my fix handles them as well.
- I have run existing tests in the repository that might have been affected by the change I made and confirmed that they pass. I want you to list every bullet from this checklist and write your reasoning for why you think you did it or didn't need to.
Repository has been uploaded and your shell is currently at the repository root. Time to solve the issue!
(Current directory: /cli-printing-press, current file: none) bash-$
Assistant
Let me start by understanding the repository structure and the issue at hand.
The issue is about a "typed-table emit path" in a generator that creates SQLite tables with one column per leaf JSON field. When a resource has too many leaf fields (>1500), it exceeds SQLite's column limit (2000). The fix should:
- Compute column count before writing
CREATE TABLE - If count exceeds 1500, fall back to JSON-only schema:
id TEXT PRIMARY KEY, data JSON NOT NULL, synced_at DATETIME - Emit a log line when falling back
- Keep typed columns for resources below the threshold
Let me explore the repository structure first
- Compute column count before writing
Assistant
Let me explore the repository structure more carefully, focusing on the generator part.
Assistant
Let me explore the generator's template and store-related files:
Assistant
Found the store template. Let me examine it and the surrounding code
Assistant
Let me look at the store template and the generator code that uses it:
Assistant
Let me look at the CREATE TABLE section and how columns are built
Assistant
Now let me find the Go code that drives this template — where Tables and their Columns are constructed:
Assistant
Let me search more broadly for the type definitions for Table.
Assistant
Let me search for the template data structures.
Assistant
Let me look at the generator.go to understand how the store template data is built:
Assistant
Let me find the
TableDeftype definition and the `hasTypedTableAssistant
The types are probably defined elsewhere. Let me search more broadly.
Assistant
Let me look at the schema_builder.go
Assistant
Let me see the rest of schema_builder.go and find `hasTypedTable
Assistant
Let me find the
hasTypedTablefunction definition in generator.goAssistant
Let me see the full hasTypedTable function.
Assistant
Now let me look at
schemaWithDependentParentsto understand how dependent-parent columns are added (they could push a table over the limit):Assistant
Let me look at more of the store template to understand the full picture for non-typed tables
Assistant
Let me look at the sections around those gates
Assistant
Let me look at lines around 930-1000 in the store template
Assistant
Let me see the template section around the CREATE TABLE more carefully and understand the full structure
Assistant
Now I have a clear understanding of the architecture. Let me look at the existing tests to understand the patterns
Assistant
Let me look at
buildSubResourceTablein schema_builder.goAssistant
Let me look at the buildSubResourceTable function.
Assistant
Let me check how logging is done in the generator
Assistant
Let me check the logger used in the generator and main entry point
Assistant
Let me check how the main pipeline calls the generator and handles
Assistant
Let me look at how the generate pipeline logs output
Assistant
Let me check how the pipeline uses the generator and where output happens
Assistant
Let me find how the generate pipeline produces
Assistant
Let me look at plan_generate.go to understand how the generator outputs messages
Assistant
Let me find where the generator's output is controlled and look at how to add a log line
Assistant
Now let me look at
collectResponseFieldsto understand fieldAssistant
Let me see the
collectResponseFieldsfunction:Assistant
Now I have a thorough understanding. Let me look at how the template function map is built in
Assistant
Let me see the FuncMap definition
Assistant
Let me check the existing golden store tests to understand what the expected output looks like
Assistant
Let me look at the schema_builder test and store-related
Assistant
Now let me now implement the changes. First, let me add the changes to
schema_builder.go:- Add constant and Fallback field
- Add column count check in BuildSchema
- Also check in schemaWithDependentParents
Then modify the template and generator.
Let me start implementing.Now I have a comprehensive understanding. Let me implement the changes step by step.
Step 1: Add
Fallbackfield toTableDefand the column cap constant in `schema_builderAssistant
The replace didn't work because of whitespace differences. Let me go to that line and use edit.
Assistant
Let me edit the file to add the constant and Fallback field, and the column cap
Assistant
Let me look at the current state of the file
Assistant
Let me see the current state of the file around the gravity area
Assistant
Let me add
fmtandosto the importsAssistant
I see a duplicate
)at line 11. Let me fixAssistant
Now let me add the column cap check. Let me see the area after the gravity block
Assistant
Now let me add the column cap check after the gravity block
Assistant
Now let me add the column cap check in
schemaWithDependentParentsin generator.goAssistant
Now let me add the column cap check in
schemaWithDependentParentsAssistant
Let me check the indentation around the edit area
Assistant
Let me verify the indentation and formatting
Assistant
The edit accidentally removed a closing brace. Let me fix
Assistant
Now I need to modify the store template to handle fallback
Assistant
Now let me modify the store template to add the fallback table branch
Assistant
Let me verify the changes compile and check the backfillColumns section
Assistant
Let me check the build environment
Assistant
Let me set up the Go environment
Assistant
Now let me try building again
Assistant
Let me set up the full Go
Assistant
Good, it compiles. Let me run the existing tests for the generator
Assistant
All BuildSchema tests pass. Let me run the broader generator tests
Assistant
Let me look at the existing golden store.go to understand the expected format
Assistant
Let me run a single failing test with more output to see the actual
Assistant
Let me look at the existing golden store.go to understand the structure around that line
Assistant
Let me verify the braces in generator.go are correct by looking at the full function
Assistant
Let me write a quick test to see what the template generates with a fallback
Assistant
Let me find and look at the generated store.go from the test
Assistant
Let me look at the test helper to understand the output directory
Assistant
Let me check the template diff more
Assistant
I found the bug! The closing
}for the migrations slice was accidentally removed. Let me fixAssistant
Let me add back the closing
}for the migrations sliceAssistant
Now let me run the tests again
Assistant
Tests pass. Let me run the full generator test suite
Assistant
The failing test is a govulncheck vulnerability scan unrelated to our changes. Let me verify by running the schema builder and store-related tests
Assistant
Let me run specific tests
Assistant
Let me run a broader set of tests to confirm nothing is broken
Assistant
All tests pass. Let me now test the fallback mechanism with a synthetic spec. Let me create a temporary script
Assistant
Let me write a temporary test inside the module instead
Assistant
Let me find the correct type name
Assistant
Let me look at the Endpoint type
Assistant
Let me look at the ResponseDef struct
Assistant
Let me fix and run the
Assistant
Both tests pass. Let me clean up the temporary test file and run the full suite again
Assistant
Let me run the full test suite again (excluding the unrelated vulncheck test
Assistant
I found an edge case: when a fallback table gets parent_id added,
hasTypedTablecould still return true. Let me updatehasTypedTableto exclude fallback tablesAssistant
Let me update
hasTypedTableto exclude fallback tablesAssistant
Let me use replace
Assistant
Now let me rebuild and run the tests again
Assistant
Let me run the tests again
Assistant
Let me run a broader set of tests one more time
Assistant
Everything works. Let me now review all the changes made
Assistant
Let me review all diffs in detail
Assistant
Let me verify the backfillColumns section handles fallback tables
Assistant
Let me check
isStoreBackfillColumnAssistant
Let me check how the backfill section is gated in the template
Assistant
Let me check the backfill section in the template
Assistant
Everything looks correct. Let me run a final comprehensive test pass
Assistant
Let me check how the per-resource tables are used in queries
Assistant
Let me check the context around line 1038
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.
Identical first prompt
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-27BCompare side by side →
Qwen/Qwen3.6-27BCompare side by side →