ISSUE DESCRIPTION generator: typed-table emit exceeds SQLite column cap on wide config…
- User
ISSUE DESCRIPTION
generator: typed-table emit exceeds SQLite column cap on wide config singletons
Filed by
/printing-press-retrofrom a run of HaloPSA (scorecard 91/100 Grade A, 100% verify pass).Summary
The typed-table emit path generates one column per leaf JSON field in a resource's response schema with no upper bound. On HaloPSA's tenant-global
/Controlconfig singleton, this produced a 3,747-columnCREATE TABLEthat exceeded SQLite's defaultSQLITE_MAX_COLUMN=2000. Generation passed every quality gate;sync --full --dry-rundied cold withSQL logic error: too many columns on control (1). The failure mode is silent until first sync.Where to look
- Component: generator
- Likely area: Typed-table-emit path in
internal/generator/templates/store/— whereverCREATE TABLEtext is composed per discovered resource. - Triggered when: A spec resource's response schema flattens to >~1500 leaf fields. Enterprise config singletons are the obvious class.
What we observed
Run-1 output (post-generate):
PASS go mod tidy / govulncheck / go vet / go build PASS halopsa-pp-cli --help / version / doctor Generated halopsa at .../working/halopsa-pp-cliFirst
sync --full --dry-run:Error: opening local database: running migrations: migration failed: SQL logic error: too many columns on control (1)Per-table column counts (top 5 widest, recorded by Python introspection of
internal/store/store.go):table columns control 3747 tickets 847 projects 847 opportunities 847 client 447 Tickets / projects / opportunities sit just under the 2000 cap but with no headroom; the next major-version Halo response schema could push them over.
Workaround applied this run: hand-stripped
/Control*paths from the spec (5 paths) and added a stubControlschema so residual$reftargets resolved. Regenerated successfully.Suspected root cause
Confident. The typed-table-emit path walks the resource's response schema and emits one column per leaf field with no width sanity check. The store package has no JSON-only fallback for resources whose schema exceeds a safe column count.
Suggested direction
In the typed-table-emit path, compute the column count before writing
CREATE TABLE. If the count exceeds a threshold (1500 leaves headroom under SQLite's default 2000; tunable), fall back to JSON-only schema:id TEXT PRIMARY KEY, data JSON NOT NULL, synced_at DATETIME. Emit a generation-time log line naming the resource and the fallback (e.g.,store-fallback: control (3747 cols) → JSON-only).Preserve typed-column behavior below the threshold — the typed columns power offline FTS5 and cross-entity SQL joins; the fallback should only fire on pathological widths.
Acceptance criteria
- positive: feed a synthetic spec where one resource has 3000+ leaf fields; generated
store.gofor that resource has only 3 columns; sync against an empty DB succeeds. - negative: feed a normal resource (<1500 cols); typed-table emission unchanged; flat columns still emitted.
Frequency
subclass:wide-config-singleton — complete-functional-failure when it fires.
Complexity
small (column-count check + JSON-only emit branch).
Scope boundary
Does NOT cover per-column truncation, schema-aware projection, or per-API column allowlists. Single threshold + JSON-only fallback only.
Dependencies
None.
Related issues
None — first sighting of this specific failure mode.
Artifacts
Artifact Link Retro document https://files.catbox.moe/ico9uo.md Manuscripts (research + proofs) https://files.catbox.moe/rn4hf4.zip Generated CLI source code https://files.catbox.moe/4v6myv.zip
Generated by
/printing-press-retro· CLI Printing PressEvaluation contract
Use 1500 columns as the safety cap for a generated typed SQLite resource table, including dependent-parent columns. Resources exceeding the cap must retain their per-resource table but fall back to a JSON-only schema containing
id,data, andsynced_at; do not generate typed-column indexes, FTS structures, or typed upsert code for that fallback table. Batch store generation must dispatch wide resources to the JSON-only path while preserving the existing typed behavior for ordinary narrow resources.Interface
No new interfaces are introduced.
ADDITIONAL ADVICE
Since you are given a git repository, you can use git commands to simplify your work. For example, if you made a mistake and want to revert your changes, you can use
git checkout <file>to restore the file to its original state. You can also reset all changes in the repository usinggit reset --hardcommand. Additionally, you can usegit stashandgit stash popto temporarily save your changes and restore them later. Finally, you can see the changes that you've made (which will be submitted when you callsubmit) usinggit statusorgit diffcommands. However you don't need to usegit addorgit commitbefore submitting your changes. In fact,submitonly submits currently unstaged changes because it usesgit diffto get the changes that need to be submitted.CHECKLIST
Before submitting your solution, please go over the following checklist and make sure you've done everything:
- If an error was reported in the issue description, I have successfully reproduced it.
- If an error was reported in the issue description, I have confirmed that my fix resolves the error.
- I have thought about edge cases and made sure my fix handles them as well.
- I have run existing tests in the repository that might have been affected by the change I made and confirmed that they pass. I want you to list every bullet from this checklist and write your reasoning for why you think you did it or didn't need to.
Repository has been uploaded and your shell is currently at the repository root. Time to solve the issue!
(Current directory: /cli-printing-press, current file: none) bash-$
Assistant
Let me start by exploring the repository structure to understand the
Assistant
Now let me look at the table structure generator:
Assistant
Now let me find where the Tables/Columns are computed for the generator:
Assistant
Now let me look at the rest of schema_builder.go:
Assistant
Now let me look at the schemaWithDependentParents function and where store_schema_version_test is referenced:
Assistant
Now let me look at the upsert generation:
Assistant
Now let me see the dispatch in UpsertBatch:
Assistant
Now let me look at the test templates to understand the full structure:
Assistant
Now let me find where the schema data is passed to the templates:
Assistant
Now let me understand the schema_builder tests to know what they look like:
Assistant
Now I have a good understanding. Let me look at the existing test patterns:
Assistant
Now let me run a quick check to understand the test pattern (the relevant one — let me look at what
isBackfillColumndoes):Assistant
Now I have enough understanding. Let me look at the whole
store.go.tmpltemplate to understand all the places I need to handle the JSON-only fallback:Assistant
Now I have a clear picture. Let me also check the existing tests:
Assistant
Let me look at the structure of where I might want to make the changes. Let me first write a test to reproduce the error:
Assistant
Now let me think about the design more carefully. The issue says:
- We need a 1500-column safety cap.
- Resources exceeding the cap must retain their per-resource table but fall back to a JSON-only schema containing
id,data, andsynced_at. - Do not generate typed-column indexes, FTS structures, or typed upsert code for that fallback table.
- Batch store generation must dispatch wide resources to the JSON-only path while preserving the existing typed behavior for ordinary narrow resources.
- Emit a generation-time log line naming the resource and the fallback (e.g.,
store-fallback: control (3747 cols) → JSON-only).
Looking at the architecture, the right place to make the change is in
BuildSchemainschema_builder.go, where we already build the TableDef. We can introduce a "JSON-only" or "wide" flag, or alternatively alter the table's columns.Let me look at how the table is consumed to know if we should reset to baseTableColumns + add a flag, or replace columns:
Assistant
Let me check the schema_version_test template for the same:
Assistant
Now let me also see the sync.go.tmpl to understand the full context:
Assistant
Now I understand. Let me think about the design:
Design Plan:
-
Add a "wide" / "JSON-only" concept to the TableDef. When a resource's typed columns exceed 1500, the schema builder should:
- Reduce the columns to just
id,data,synced_at(already in baseTableColumns) - Drop FTS5/Indexes
- Set a
Wideflag
- Reduce the columns to just
-
Update
hasTypedTableto returnfalsefor wide tables (since they have only 3 columns and aren't typed). -
Update the template so that wide tables still get a
CREATE TABLEfor the JSON-only shape but no typed upsert, FTS, etc. -
Emit a generation-time log line
store-fallback: <name> (<n> cols) → JSON-only.
Let me check the gen log pattern:
-
Assistant
Now let me think about the cleanest approach.
Approach: Add a
Wide bool(orJSONOnly bool) field to TableDef. The schema_builder.go computes the column count and:- If
len(t.Columns) > 1500, setWide=true, resetColumnsto base columns, drop FTS5/Indexes, and emit a log line.
Then
hasTypedTableneeds to also check for!t.Wide. The template will need to emit a CREATE TABLE for the JSON-only shape (so we keep per-resource table but with id/data/synced_at only).Let me also check what
domainUpsertMethodNamedoes:- If
Assistant
Good. Now let me also check the templating around the columns to understand how to keep CREATE TABLE working for wide tables but not the upsert:
Assistant
Now let me design the solution. I'll add a
Widefield to TableDef. Then in the schema builder, if a table's typed columns exceed 1500 (counting from the base 3 + typed leaves), we mark it Wide, reset its Columns to just the base, drop FTS5/Indexes, and emit a log.Let me start by creating a test to reproduce:
Assistant
Now let me first run the existing tests to see what passes:
Assistant
Now I know the test environment works. Let me create a test script to reproduce the issue:
Assistant
Let me update the test to use the existing fields and then proceed with the actual fix:
Assistant
I've reproduced the bug. The control table has 3003 columns (3 base + 3000 typed), which is way above SQLite's default 2000 cap. Now let me implement the fix.
Implementation Plan:
- Add
Wide boolfield toTableDefinschema_builder.go. - Add
MaxTypedColumns = 1500constant. - In
BuildSchema(or a helper function called from it), detect when a resource's typed columns exceedMaxTypedColumns. When that happens:- Set
Wide = true - Reset
Columnsto base only (id, data, synced_at) - Drop
FTS5,Indexes,FTS5Fields,FTS5Triggers - Emit a stderr log line
- Set
- Modify
hasTypedTableto returnfalsefor Wide tables. - Modify the
store.go.tmpltemplate to emit a CREATE TABLE for Wide tables (id, data, synced_at) but not typed columns/indexes/FTS. - The
UpsertBatchdispatch should skip the typed-table dispatch for wide tables.
Let me also look at how the migrations slice is iterated so the CREATE TABLE for wide tables is still emitted:
- Add
Assistant
I see — the gate is
hasTypedTablewhich means wide tables won't even get aCREATE TABLE. We need to change the gate to handle wide tables separately. Let me think about this more carefully.The instruction says:
Resources exceeding the cap must retain their per-resource table but fall back to a JSON-only schema containing
id,data, andsynced_atSo we need to:
- Emit a
CREATE TABLEfor the wide table (with id, data, synced_at only) - NOT emit typed-column indexes
- NOT emit FTS structures
- NOT emit typed upsert code
The simplest approach:
- Modify
hasTypedTableto also returnfalsefor wide tables. - Add a new predicate
hasWideTable(orhasResourceTable) that returnstruefor tables that need aCREATE TABLE(typed or wide). - Use that predicate for the
CREATE TABLEgate.
Now let me look at the existing gate, which I see is
{{- if hasTypedTable .}}for the table creation. We need to:- Change CREATE TABLE gate to a new predicate that includes wide tables.
- Keep typed-upsert and FTS5 gated on
hasTypedTableonly.
Let me implement this step-by-step. First, let me make changes to
schema_builder.go:- Emit a
Assistant
Now I have a clear picture. Let me think about the order of operations carefully:
- Schema builder change: After the existing column-extraction loop, check if
len(table.Columns) > MaxTypedColumns. If so, reset to base only, setWide = true, drop FTS5/Indexes, log a line.
But the question is: do we want to keep FTS5 info if the resource has few fields? The issue says "preserve typed-column behavior below the threshold". So below the threshold, the resource gets the full typed table with FTS etc. Above the threshold, it gets the JSON-only fallback.
Let me also look at the parent_id column that gets added later. The
schemaWithDependentParentsfunction adds aparent_idcolumn AFTERBuildSchema. This means dependent wide tables will need to be handled too. But for now, let me check if dependent resources are even common in wide configs (the issue is about a config singleton, so likely top-level).Let me check:
- Schema builder change: After the existing column-extraction loop, check if
Assistant
OK so the
parent_idcolumn is added to dependent resources. For wide resources, this would push them over the cap. But after we setWide = truein BuildSchema, parent_id would be added (since depSet only depends on table name, not on whether it's wide). We should also skip this for wide tables — or at least re-evaluate the threshold after parent_id is added.Actually, looking at the issue: "Use 1500 columns as the safety cap for a generated typed SQLite resource table, including dependent-parent columns." So the threshold should include parent_id columns.
Let me think about the cleanest design:
- Apply the wide check after both BuildSchema and
schemaWithDependentParents, not just BuildSchema. - Or: Apply wide check in BuildSchema, and then
schemaWithDependentParentsshould skip adding parent_id to wide tables (since their columns are already minimal).
Actually, the simpler approach is to apply the wide check at the end of BuildSchema but make
schemaWithDependentParentsaware of Wide tables. Let me re-think.Actually I think the cleanest approach is to handle this AFTER all column additions are done. Let me wrap it in a helper function
applyWideResourceFallbackthat takes the schema and applies the threshold check, and modifyschemaWithDependentParents(and the call sites) to call it after all column additions.Let me trace the order more carefully:
schemaWithDependentParentscallsBuildSchema(g.Spec), then adds parent_id to dependent tables.- This is called from
renderVisionAndRootFilesandrenderMCPToolFiles.
So I'll add a helper function
applyJSONOnlyFallback(schema) []TableDefand call it after all column additions.Let me start the implementation:
- Apply the wide check after both BuildSchema and
Assistant
Oops, I accidentally duplicated
baseTableColumns. Let me remove the duplicate:Assistant
There's a duplicate line. Let me clean it up:
Assistant
Still duplicated. Let me see the file again:
Assistant
Good. Now I need to add the wide-resource check. The right approach is to add a helper function
applyJSONOnlyFallbackthat takes a[]TableDefand mutates wide ones. Let me also think about where it should be called from.Looking at the data flow:
BuildSchemais called.schemaWithDependentParentscallsBuildSchemathen addsparent_idcolumns.- The result is passed to template rendering.
The wide check should be applied AFTER all column additions (including dependent parent_id). Let me put the helper in
schema_builder.goand call it fromschemaWithDependentParents.Let me find the end of BuildSchema:
Assistant
Now let me find a good place to add the helper function. Let me put it at the end of the file:
Assistant
Now let me append the helper function:
Assistant
Now I need to update
schemaWithDependentParentsto callapplyJSONOnlyFallbackafter adding parent_id columns:Assistant
Hmm. The issue is probably the comment with parentheses — shell thinks
(is a subshell. Let me try with the file content via Python:Assistant
Good. Now let me also update
hasTypedTableto returnfalsefor wide tables. Let me look:Assistant
The ! was eaten. Let me fix that:
Assistant
Now let me update the store.go.tmpl to use the new
hasResourceTablepredicate for the CREATE TABLE gate:Assistant
Now I need to update the CREATE TABLE gate. The current structure is:
{{- range .Tables}} {{- if hasTypedTable .}} CREATE TABLE ... {{- range .Indexes}} ... {{- end}} {{- if .FTS5}} ... {{- end}} {{- end}} {{- end}}For wide tables, I want to:
- Emit
CREATE TABLEwith the JSON-only shape (which is just base columns since wide tables have base columns only). - Skip the Indexes loop (since wide tables have no indexes).
- Skip the FTS5 block.
So the new structure should be:
{{- range .Tables}} {{- if hasResourceTable .}} CREATE TABLE ... {{- if hasTypedTable .}} {{- range .Indexes}} ... {{- end}} {{- if .FTS5}} ... {{- end}} {{- end}} {{- end}} {{- end}}Let me make this change:
- Emit
Assistant
Let me verify:
Assistant
Good. Now let me also check the backfillColumns block, which iterates over all
Columns. For wide tables, the columns would just be id/data/synced_at, and the isBackfillColumn check would filter those out, so nothing would be emitted. That's fine. Let me confirm:Assistant
isStoreBackfillColumnreturnsfalsefor "id", "data", "synced_at" — so for wide tables, the backfill loop will emit nothing. Good.Now let me also check the test templates to make sure they handle wide tables correctly:
Assistant
The test
TestMigrate_AddsColumnsOnUpgradeonly fires for tables with backfill columns (typed columns beyond id/data/synced_at). Wide tables have no backfill columns, so no test will be generated. Good.Now let me look at the upsert batch test:
Assistant
Good. The upsert batch test gates on
hasTypedTable, so wide tables won't have a typed-populate test. That's the correct behavior.Now let me also look at the SearchX functions:
Assistant
This gates on
FTS5. Since wide tables haveFTS5 = false, no SearchX function will be emitted for them. Good.Now let me run the test to see if my fix works:
Assistant
Good, the build works. Now let me run my repro test:
Assistant
My test only uses
BuildSchemadirectly, notschemaWithDependentParents. Let me update the test to also check the wide-resource handling:Assistant
The fix works at the schema_builder level. Now let me verify the full pipeline by running a test that renders templates:
Assistant
Let me look at
minimalSpecto see how to set up a test:Assistant
Let me write an end-to-end integration test:
Assistant
My extraction logic is off — it's grabbing too much. Let me fix it:
Assistant
The end-to-end test passes. The CREATE TABLE for control only has 3 columns (id, data, synced_at) and there's no typed UpsertControl function. Let me also verify that the generated code compiles and can sync:
Assistant
Let me move the test inside the package:
Assistant
Now let me also test that the generated code compiles. Let me extract and verify:
Assistant
The generated code compiles. Now let me also test that the generated test file compiles and passes:
Assistant
Excellent! The generated store tests pass. Now let me also run the negative test - a normal resource (<1500 cols) should still produce typed output:
Assistant
The negative test passes. The normal resource retains its typed columns. Now let me also clean up these temporary tests since the instructions say I should NOT add tests to the repository. Let me re-read:
However UNDER NO CIRCUMSTANCES should you modify existing tests or add new tests to the repository. This will lead to an error in the system that evaluates your performance. Instead, you can just create a temporary script, use it to test changes and remove it before submitting.
Let me clean up the temporary test files. But first, let me also verify that I'm not breaking any existing tests by running the broader test suite:
Assistant
Good, all temporary tests removed. Now let me run the existing tests to make sure I didn't break anything:
Assistant
I see two failures. Let me look at these:
TestGeneratedSyncHonorsHTMLExtraction- this could be related to my change. Let me investigate.TestGenerateWithNoAuth- this is a known Go vulnerability check failing (GO-2026-5856in crypto/tls), not related to my change.
Let me investigate the first failure:
Assistant
Let me check this test to understand what's failing:
Assistant
Let me check if this test was passing before my change:
Assistant
It passes without my change. So my change broke something. Let me restore my changes:
Assistant
OK so my change broke
TestGeneratedSyncHonorsHTMLExtraction. Let me investigate what's going wrong. The test sets up an HTML extraction test. Let me look at what's happening:Assistant
Let me look at the failing test:
Assistant
Let me actually compile and run the failing test to see what's happening:
Assistant
Let me check what's in the generated
internal/cli/sync.goto see how the dependent messages resource is handled:Assistant
Hmm interesting, it passes when run alone. Let me see if there's some race issue:
Assistant
Wait, the same code passes when I use my simplified test but fails when using the original test. Let me look at the difference more carefully. Maybe my check_sync_temp_test doesn't reproduce the failure because of timing. Let me try to reproduce it without my temp tests:
Assistant
Let me look at the generated test to see what messages table looks like:
Assistant
Let me search for the messages-related code in the generated store:
Assistant
I see the issue. The messages table has columns
[id, data, synced_at, parent_id]— but the test expectschannels_id(the sub-resource FK by parent table name).Let me look at the original test expectations:
Assistant
Now let me check the sync.go.tmpl and see how it uses dependentResourceDef to call the typed table:
Assistant
The original table has
parent_idcolumn. The test expectschannels_idto be in the JSON output. Let me see where the test gets the JSON:The test calls
db.List("messages", 10). Let me check what List does:Assistant
Listreturns from theresources(generic) table. So the test expectationchannels_idis on the JSON data in thedatacolumn. Now let me look at the sync dependent resource code that populates the messages JSON:Assistant
So when syncDependentResource runs, it injects
obj[parentFKKey] = parentIDJSON(whereparentFKKeyischannels_id). So the JSON stored should havechannels_id.But the test is failing. Let me look at the test more carefully:
Assistant
So the test calls
syncDependentResourcewithdependentResourceDef{Name: "messages", ParentTable: "channels", ...}. Looking at the failure,synced count = 0, want 1. So nothing is synced.Let me run my dump test in verbose mode and look for "syncDependentResource" code:
Assistant
OK so messages has the typed table. Now let me look at the sync.go (not store.go) in the generated output. Let me re-dump the sync.go instead:
Assistant
Let me look at syncDependentResource's content area to find how parentFKKey is set:
Assistant
OK so
parentFKKey = "channels_id". Now let me look at how the items get processed and check if my changes broke something:Assistant
OK so the dependent resource logic is unchanged. Now let me think about why my change broke this. Let me compare the messages table before and after my change. The issue is that messages is a dependent resource, and the table has
parent_idcolumn. With my change, it should still have this column.Let me also see what the parent table schema looks like:
Assistant
OK so messages is a dependent resource of channels. Let me look at the test setup:
if err := db.Upsert("channels", "bad", []byte(`{"id":"bad"}`)); err != nil { t.Fatalf("insert bad parent: %v", err) } if err := db.Upsert("channels", "good", []byte(`{"id":"good"}`)); err != nil { t.Fatalf("insert good parent: %v", err) }It inserts parents into
channelstable, then callssyncDependentResourcewithParentTable: "channels". The test wants the messages to be synced.Let me look at
dependentParentRows:Assistant
OK so it calls
db.ListIDs(parentTable). Let me find that:Assistant
OK so it looks for a typed table first. The test inserted into the generic resources table only via
db.Upsert("channels", ...). So ListIDs should fall through to the resources table.Let me see what db.Upsert does:
Assistant
OK so
db.Upsertonly inserts into the generic resources table. SoListIDs("channels")should fall back to the resources table.Let me check if my change has broken the schema_builder such that the channels table is no longer being created. Actually let me think...
channelshas no typed columns (because there's notypes: channelsdefinition). SoBuildSchemagives channels a base table (id/data/synced_at, 3 columns).len(columns) > 1500is false, so it stays typed... wait, but thenhasTypedTableislen(t.Columns) > 3which is 3 > 3 = false. So channels doesn't have a typed table.So ListIDs should hit the fallback (generic resources table). OK so this should still work.
Let me actually run my full dump test to see the dump more clearly:
Assistant
Let me actually compare the original store.go before my changes and after my changes:
Assistant
OK, the messages resource is the same. Let me run the failing test in isolation and add debug output:
Assistant
OK so messages has 3 columns in BuildSchema (no types declared). Let me find the test that runs to see what changed:
Assistant
Interesting! When I run the same test in isolation, it passes! But when I run
TestGeneratedSyncHonorsHTMLExtractiondirectly, it fails. Let me try running both at once:Assistant
So when
TestGeneratedSyncHonorsHTMLExtractionruns first, it fails, but when my reproduction runs first, it passes. The difference is that the test usesrunGoCommandRequired(t, outputDir, "test", ...)- let me see what that does and check what's different:Assistant
Hmm. So the test does
go test -mod=mod -run ... ./internal/cliwith a separate GOCACHE. Let me look more carefully:Assistant
Let me see if my reproduction uses the same approach - yes, I'm running
go test -v ...but without-mod=mod. Let me try with-mod=mod:Actually wait, the difference is that my repro uses
go testwithoutmod tidy. The test doesgo mod tidyfirst. Let me try that:Assistant
I reproduced the failure! Now the question is why. Let me also test by reverting my change:
Assistant
Confirmed: my change is causing the regression. Let me restore:
Assistant
Let me investigate. With my change, the test reports
synced count = 0. That meanssyncDependentResourcereturned a result with Count=0. So either:dependentParentRowsreturned 0 rows (can't iterate parents).- The sync was done but the items couldn't be extracted.
Let me add more logging to my repro:
Assistant
Now I see something interesting! In my output, I see this:
`CREATE TABLE IF NOT EXISTS "channels" ( "id" TEXT PRIMARY KEY, "data" JSON NOT NULL, "synced_at" DATETIME DEFAULT CURRENT_TIMESTAMP )`, `CREATE TABLE IF NOT EXISTS "json_pages" ( "id" TEXT PRIMARY KEY, "data" JSON NOT NULL, "synced_at" DATETIME DEFAULT CURRENT_TIMESTAMP )`, `CREATE TABLE IF NOT EXISTS "messages" ( "id" TEXT PRIMARY KEY, "data" JSON NOT NULL, "synced_at" DATETIME DEFAULT CURRENT_TIMESTAMP, "parent_id" TEXT )`,This looks like an array literal where each element is a SQL statement. So the
channelstable is being created with just id/data/synced_at, andmessagesis created with id/data/synced_at/parent_id.Wait — but this is good. Let me see what's actually going wrong. Let me dump just the channels table CREATE statement (without the trailing
,):Assistant
Got it! The issue is
ListIDsis returning 0 IDs, even though the resources table has rows. Let me look atListIDsagain:func (s *Store) ListIDs(resourceType string) ([]string, error) { var table string err := s.db.QueryRow( `SELECT name FROM sqlite_master WHERE type='table' AND name=?`, resourceType, ).Scan(&table) var rows *sql.Rows if err == nil && table != "" { rows, err = s.db.Query(fmt.Sprintf(`SELECT id FROM "%s"`, strings.ReplaceAll(table, `"`, `""`))) } if err != nil || table == "" { // Fall back to generic resources table rows, err = s.db.Query("SELECT id FROM resources WHERE resource_type = ?", resourceType) if err != nil { return nil, err } } defer rows.Close()The
channelstable exists (per the migration), so it tries to querySELECT id FROM "channels"directly. Butdb.Upsertonly writes to theresourcestable, NOT to thechannelstable. So the channels table is empty, and ListIDs returns 0.This is the regression: Before my change, the migration didn't create a
channelstable (because it had no typed projection —len(columns) > 3was false), soListIDswould fall through to the genericresourcestable. After my change, thechannelstable IS created (becausehasResourceTablereturns true), soListIDsqueries the empty channels table.This is a bug in the existing test setup! The test relies on the
channelstable NOT being created so thatListIDsfalls back toresources. With my change, the channels table is created.The issue says: "Resources exceeding the cap must retain their per-resource table". So we need to keep the per-resource table. But we need to make sure
db.Upsert(which writes only to the genericresourcestable) keeps working with parent iteration.The fix is to update
ListIDs(or similar) to handle this case — but the issue is more fundamental. Let me re-read the issue:Resources exceeding the cap must retain their per-resource table but fall back to a JSON-only schema containing
id,data, andsynced_at; do not generate typed-column indexes, FTS structures, or typed upsert code for that fallback table.So the fallback is ONLY for resources exceeding the cap. But my change is now creating per-resource tables for ALL resources, including ones that previously had no per-resource table. This breaks tests that relied on
ListIDsfalling back.The original gate
hasTypedTablewas stricter (only tables with >3 columns AND not sync_state AND not "UpsertBatch"). Tables with exactly 3 columns (like channels) didn't get a per-resource table at all.The fix should be more surgical: Only wide resources should get a per-resource table with id/data/synced_at. The current behavior for narrow resources (no table) should be preserved.
Let me re-think. The key insight is:
- For narrow resources with no typed projection (channels), no CREATE TABLE is needed.
- For narrow resources with typed projection (e.g., pages with 100 fields), CREATE TABLE is needed.
- For wide resources (3000+ fields), CREATE TABLE with id/data/synced_at only is needed.
So my
hasResourceTablepredicate needs to be:- Returns true if
hasTypedTable(t)is true (typed table exists). - Returns true if
t.Wideis true (wide fallback table needed). - Returns false otherwise.
Let me fix this:
Assistant
Wait, but I have a reference to
hasTypedTablewhich doesn't exist as a function. Let me change this approach — uset.Wide || (len(t.Columns) > 3 && ...)directly:Assistant
Let me verify:
Assistant
Now let me run the test again:
Assistant
The dependent resource test passes now. Let me run the original failing test:
Assistant
Let me also remove my repro_temp_test.go and run a broader set of tests:
Assistant
Only
TestGenerateWithNoAuthis failing now, and it's a pre-existing govulncheck failure unrelated to my change. Let me verify it's pre-existing:Assistant
Confirmed pre-existing. Let me restore my changes:
Assistant
Now let me run all tests except the govulncheck one to verify my changes don't break anything else:
Assistant
All other tests pass. Let me also test more thoroughly:
Assistant
The test regex doesn't match the right pattern. Let me try again:
Assistant
All non-G tests pass. Now let me re-run my earlier verification tests:
Assistant
Wide resource works. The generated store tests pass. Now let me also verify the negative case (resource below 1500 cols still works):
Assistant
Both positive and negative cases work. Now let me also verify edge cases:
- Resource with exactly 1500 columns (boundary).
- Resource with 1501 columns (just over).
Assistant
Interesting. With 1500 fields, we get 1503 columns (1500 + id + data + synced_at). So the column count is
len(fields) + 3. Let me re-check the threshold:The issue says: "Use 1500 columns as the safety cap for a generated typed SQLite resource table, including dependent-parent columns."
So the cap is 1500 columns total. With 1500 fields, we get 1503 columns total. So 1500 fields is OVER the cap (1503 > 1500). The correct test is:
- 1497 fields = 1500 columns = exactly at cap (keep typed).
- 1498 fields = 1501 columns = just over (JSON-only fallback).
But really the question is what the issue means by "1500 columns":
- Strictly: 1500 typed columns + id + data + synced_at = 1503 total.
- Or: Total = 1500 (so 1497 fields).
The original issue says: "This produced a 3,747-column
CREATE TABLEthat exceeded SQLite's defaultSQLITE_MAX_COLUMN=2000." and "Use 1500 columns as the safety cap for a generated typed SQLite resource table, including dependent-parent columns."The phrasing "1500 columns as the safety cap" suggests the cap is on the total number of columns (including id, data, synced_at). And "including dependent-parent columns" suggests we should also include the dependent parent column.
So the cap should be
len(t.Columns) <= 1500. Let me check: 1497 fields = 1500 columns total → keep typed. 1498 fields = 1501 columns total → fall back.Let me check my logic again.
applyJSONOnlyFallbacktriggers whenlen(table.Columns) > MaxTypedColumns. With 1503 columns (1500 fields + 3 base), that's 1503 > 1500, so it falls back. Good.For 1500 fields: 1500 + 3 = 1503 columns. With my current threshold
> 1500, this triggers fallback.Actually wait, let me re-read: "Use 1500 columns as the safety cap for a generated typed SQLite resource table, including dependent-parent columns."
I think the cap is on total columns. So
len(columns) > 1500triggers fallback. With 1500 fields (1503 columns), this falls back. With 1497 fields (1500 columns), this keeps typed. That's what my current code does.The test I wrote was wrong — I was checking
UpsertItemsexists when fields count is 1500, but that's actually over the cap (1503 > 1500). Let me fix the test:Assistant
The boundary works correctly:
- 1497 fields = 1500 cols total → keep typed
- 1498 fields = 1501 cols total → fall back to JSON-only
- 1499 fields = 1502 cols total → fall back to JSON-only
Now let me also check the edge case where dependent parent_id adds a column. This should still trigger fallback if total > 1500:
Assistant
Let me look at the right way to set up:
Assistant
OK so the
profilefield is*profiler.APIProfile. Let me simplify my test - just verify the schemaWithDependentParents logic works:Assistant
Hmm. The children table only has 4 columns (id, parent_id, data, synced_at) which means the children resource didn't get any typed columns. Let me think about this...
The issue is that
childrenis a sub-resource, and sub-resources don't get typed columns. The buildSubResourceTable only adds id, parent_id, data, synced_at. So we don't get 1498 fields in children.Let me check what's happening:
Assistant
So the children sub-resource (
list) only gets base columns frombuildSubResourceTable, which is 4 columns (id, parent_id, data, synced_at). The 1498 fields I defined don't get added because sub-resources don't add typed columns in the schema_builder. So this case won't trigger the wide fallback.OK so dependent sub-resources don't have typed columns. My fix is sufficient for the existing data flow. Let me think about whether there are other edge cases.
Wait - what about top-level resources with many fields that ARE also dependents? Let me check the docs:
Looking at
schemaWithDependentParents:for _, dep := range g.profile.DependentSyncResources { depSet[dep.Name] = true }It uses
dep.Namewhich is the resource name. So if a top-level resource is also a dependent, parent_id gets added.But actually, in practice, dependents are typically sub-resources that aren't top-level. The case where a top-level resource is also a dependent is unusual.
Let me think about another edge case: what if a resource has gravity < 2 (no typed columns) but is also flagged as dependent? It gets parent_id added, going from 3 to 4 columns. That's still well under 1500. Not an issue.
Now let me clean up my temporary tests:
Assistant
Good, all temp tests removed. Now let me run the broader tests to make sure nothing else is broken:
Assistant
All non-G tests pass. Now let me also run other package tests:
Assistant
Two failures:
pipeline- test failedpressauth- test failed
Let me check if these are pre-existing:
Assistant
This is unrelated to my change - it's a $HOME issue. Let me verify:
Assistant
The HOME environment variable is empty. Let me set it:
Assistant
Yeah, the pressauth tests just need HOME set. Not related to my change. Now let me check pipeline:
Assistant
Let me see which test fails:
Assistant
This test uses chmod 0 which fails when running as root. It's unrelated to my change. Let me confirm:
Assistant
Confirmed: pre-existing failure (because we're running as root, chmod 0 doesn't restrict access). Let me restore:
Assistant
Good. Now let me also do a final check to make sure all the relevant generator tests pass:
Assistant
Let me look at the diff of my changes to verify they are all correct:
Assistant
Wait, my test added chromedp/cdproto as a direct dependency! That was from running the wide resource test which used
go mod tidy. Let me revert that:Assistant
Good. Now let me look at the actual diff for the schema_builder.go:
Assistant
The diffs look clean and minimal. Let me also verify the templates still parse properly with a check on the end-to-end pipeline once more:
Assistant
Builds clean. Let me also write one final verification to make sure my fix produces the expected output. Let me create a test that captures the exact assertions from the issue:
Assistant
Both acceptance criteria pass:
- Positive: 3000+ fields → control table has 3 columns, generated store tests pass.
- Negative: 100 fields (under cap) → typed Upsert emitted, all 100 columns in CREATE TABLE.
Now let me also clean up and remove my temp tests:
Assistant
All temp tests removed. Now let me check if there are any other places that might be affected. Let me look at the
domainUpsertMethodNamecalls to ensure my changes don't break anything:Assistant
domainUpsertMethodNamereturns "UpsertBatch" for reserved table names. This is a separate code path. Let me check:Assistant
Looking at
domainUpsertMethodNameit just returnsUpsert+ Pascal name. The checkdomainUpsertMethodName(t.Name) != "UpsertBatch"only matters if the table name happens to Pascalize to "Batch" (giving "UpsertBatch"). That's basically never (you'd need a resource named "batch"). So this check is essentially always true and doesn't matter.Now let me also verify that all generated code with wide resources can run a sync. Let me write one final test that does a full integration test:
Assistant
Good. Let me also do a comprehensive final test by running the generator tests except the govulncheck ones (which are unrelated to my change):
Assistant
All non-govulncheck tests pass. The only failure is the pre-existing govulncheck issue.
Now let me also check that my fix handles the case where a resource's columns go from 4 to 1501 (e.g., a 1497-field resource gets parent_id appended):
Assistant
Excellent! The pivot case works:
- 1497 fields = 1500 cols total
- After parent_id added: 1501 cols (over cap)
- Falls back to JSON-only with 3 columns
This shows the cap is correctly evaluated AFTER all column injection (BuildSchema + parent_id).
Let me also check that without parent_id, 1497 fields stays typed (the column count is 1500 = at cap, not over):
Assistant
The at-cap case works:
- 1497 fields = 1500 cols total (exactly at cap)
- Stays typed (no JSON-only fallback)
- The threshold is
> MaxTypedColumns, so 1500 is OK, 1501 is not.
Let me clean up:
Assistant
Now let me also do one more comprehensive run to make sure my final state is good:
Assistant
Let me also verify the change works without the
Widefield having any unintended effects. Let me run the whole package's tests one more time:Assistant
Only the pre-existing govulncheck failure. My change is clean. Let me also look at the final state of my changes:
Assistant
Good. Let me show the final state:
Assistant
The changes look clean and minimal. Let me also double-check that the test files I have left are the original ones, not my temp tests:
Assistant
That's a build artifact from running tests. Not something I created. Let me check git history:
Assistant
That
.localdirectory is not in gitignore but is also untracked. It's not something I introduced — let me clean it up anyway:Assistant
Now my changes are clean. Let me also do a final verification with all my tests passing:
Assistant
All tests pass. Let me also quickly verify my key files still compile:
Assistant
Let me also do a final sanity check on the generator output:
Assistant
The test says cols=4. Let me investigate - maybe my code counted wrong. Wait, I see - my test is counting ALL non-empty lines including the backtick, "comma", etc. Let me fix the test:
Assistant
Let me write a more robust test:
Assistant
The regex didn't match. Let me look at the actual format:
Assistant
The fix works. The 3747-column control resource is reduced to a 3-column JSON-only table. The
store-fallback: control (3747 cols) → JSON-onlylog line is emitted.Let me clean up:
Assistant
All temp tests cleaned up. Let me also clean up the .local directory:
Assistant
The go.mod is dirty from running my test (mod tidy). Let me revert:
Assistant
Final state. Let me also re-run the build:
Assistant
Let me also check if any tests that I broke earlier need re-checking:
Assistant
All tests pass. My changes are complete.
Now let me do a final review of my checklist:
-
If an error was reported in the issue description, I have successfully reproduced it.
- YES. I reproduced the issue: a control resource with 3000 fields produces 3003 columns in CREATE TABLE, which would exceed SQLite's 2000 cap.
-
If an error was reported in the issue description, I have confirmed that my fix resolves the error.
- YES. After my fix, the 3000-field control resource produces a 3-column JSON-only table. The generated store code compiles and the generated store tests pass.
-
I have thought about edge cases and made sure my fix handles them as well.
- YES. Edge cases I verified:
- Boundary: 1497 fields (1500 cols total) keeps typed table.
- Just over: 1498 fields (1501 cols total) falls back to JSON-only.
- Wide with parent_id: A 1497-field resource with parent_id (1501 cols total) also falls back.
- Negative case: 100 fields (103 cols total) keeps typed projection with full Upsert and FTS5.
- Dependent resource: messages table (with 4 columns from base + parent_id) is unchanged.
- sync_state: never demoted to wide (special-cased).
- YES. Edge cases I verified:
-
I have run existing tests in the repository that might have been affected by the change I made and confirmed that they pass.
- YES. I ran:
./internal/generator/test suite (excluding pre-existing govulncheck failure)./internal/...test suite (excluding pre-existing HOME-related and chmod-related failures unrelated to my change)
- YES. I ran:
Let me submit:
-
Assistant
Wait, go.mod is still dirty. Let me check:
Assistant
Wait, that
go build ./...triggered a re-modification of go.mod. Let me check the .local dir is gone:Assistant
The build modifies go.mod. Let me investigate what's happening. Actually,
go buildshouldn't modify go.mod. Let me check:Assistant
The build is promoting chromedp/cdproto to a direct dependency. This is happening because the build is running with the chromedp-related code somewhere. Let me check if my code uses chromedp:
Assistant
So the generated code uses chromedp. When the test builds the wide-resource spec (which doesn't use auth_browser), but the generator package itself imports generated code with chromedp... let me check:
Assistant
The generator package itself doesn't import chromedp. But maybe
go buildis doing a full resolution that includes indirect deps. Let me try a more targeted build:Assistant
So even
go build ./internal/generator/triggers the change. This must be because the test files reference it... let me check:Assistant
So the template uses chromedp. The test build is somehow resolving this. Let me revert go.mod and confirm the test still passes:
Assistant
Tests pass without modifying go.mod. The chromedp change must be from a previous test run that did
go mod tidyon a generated project. Let me check git status:Assistant
Final state is clean. Let me submit:
Assistant
My changes are:
-
internal/generator/schema_builder.go:- Added
Wide boolfield toTableDef. - Added
MaxTypedColumns = 1500constant. - Added
applyJSONOnlyFallbackhelper function that demotes wide resources to JSON-only tables (id/data/synced_at only) and emits a stderr log line.
- Added
-
internal/generator/generator.go:- Added
hasResourceTabletemplate predicate (broader thanhasTypedTable). - Modified
hasTypedTableto returnfalsefor wide tables. - Updated
schemaWithDependentParentsto apply the JSON-only fallback after all column additions.
- Added
-
internal/generator/templates/store.go.tmpl:- Changed CREATE TABLE gate from
hasTypedTabletohasResourceTable. - Moved the
IndexesandFTS5blocks inside ahasTypedTablegate so wide tables don't emit them.
- Changed CREATE TABLE gate from
Let me run my final check:
-
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.
Identical first prompt
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-27BCompare side by side →
Qwen/Qwen3.6-27BCompare side by side →