Skip to content
TheChatArchiveThe Chat Archive
composer-2.5· unknownHugging Face ibragim-bad/swe_rebench_07_2026_trajectories

generator: typed-table emit exceeds SQLite column cap on wide config singletons Filed by…

  1. User

    generator: typed-table emit exceeds SQLite column cap on wide config singletons

    Filed by /printing-press-retro from a run of HaloPSA (scorecard 91/100 Grade A, 100% verify pass).

    Summary

    The typed-table emit path generates one column per leaf JSON field in a resource's response schema with no upper bound. On HaloPSA's tenant-global /Control config singleton, this produced a 3,747-column CREATE TABLE that exceeded SQLite's default SQLITE_MAX_COLUMN=2000. Generation passed every quality gate; sync --full --dry-run died cold with SQL logic error: too many columns on control (1). The failure mode is silent until first sync.

    Where to look

    • Component: generator
    • Likely area: Typed-table-emit path in internal/generator/templates/store/ — wherever CREATE TABLE text is composed per discovered resource.
    • Triggered when: A spec resource's response schema flattens to >~1500 leaf fields. Enterprise config singletons are the obvious class.

    What we observed

    Run-1 output (post-generate):

    PASS go mod tidy / govulncheck / go vet / go build
    PASS halopsa-pp-cli --help / version / doctor
    Generated halopsa at .../working/halopsa-pp-cli
    

    First sync --full --dry-run:

    Error: opening local database: running migrations:
    migration failed: SQL logic error: too many columns on control (1)
    

    Per-table column counts (top 5 widest, recorded by Python introspection of internal/store/store.go):

    tablecolumns
    control3747
    tickets847
    projects847
    opportunities847
    client447

    Tickets / projects / opportunities sit just under the 2000 cap but with no headroom; the next major-version Halo response schema could push them over.

    Workaround applied this run: hand-stripped /Control* paths from the spec (5 paths) and added a stub Control schema so residual $ref targets resolved. Regenerated successfully.

    Suspected root cause

    Confident. The typed-table-emit path walks the resource's response schema and emits one column per leaf field with no width sanity check. The store package has no JSON-only fallback for resources whose schema exceeds a safe column count.

    Suggested direction

    In the typed-table-emit path, compute the column count before writing CREATE TABLE. If the count exceeds a threshold (1500 leaves headroom under SQLite's default 2000; tunable), fall back to JSON-only schema: id TEXT PRIMARY KEY, data JSON NOT NULL, synced_at DATETIME. Emit a generation-time log line naming the resource and the fallback (e.g., store-fallback: control (3747 cols) → JSON-only).

    Preserve typed-column behavior below the threshold — the typed columns power offline FTS5 and cross-entity SQL joins; the fallback should only fire on pathological widths.

    Acceptance criteria

    • positive: feed a synthetic spec where one resource has 3000+ leaf fields; generated store.go for that resource has only 3 columns; sync against an empty DB succeeds.
    • negative: feed a normal resource (<1500 cols); typed-table emission unchanged; flat columns still emitted.

    Frequency

    subclass:wide-config-singleton — complete-functional-failure when it fires.

    Complexity

    small (column-count check + JSON-only emit branch).

    Scope boundary

    Does NOT cover per-column truncation, schema-aware projection, or per-API column allowlists. Single threshold + JSON-only fallback only.

    Dependencies

    None.

    Related issues

    None — first sighting of this specific failure mode.

    Artifacts

    ArtifactLink
    Retro documenthttps://files.catbox.moe/ico9uo.md
    Manuscripts (research + proofs)https://files.catbox.moe/rn4hf4.zip
    Generated CLI source codehttps://files.catbox.moe/4v6myv.zip

    Generated by /printing-press-retro · CLI Printing Press

    Evaluation contract

    Use 1500 columns as the safety cap for a generated typed SQLite resource table, including dependent-parent columns. Resources exceeding the cap must retain their per-resource table but fall back to a JSON-only schema containing id, data, and synced_at; do not generate typed-column indexes, FTS structures, or typed upsert code for that fallback table. Batch store generation must dispatch wide resources to the JSON-only path while preserving the existing typed behavior for ordinary narrow resources.

    Interface

    No new interfaces are introduced.

  2. Assistant

    I'll implement the 1500-column SQLite cap with a JSON-only fallback in the typed-table emit path. Exploring the store generator templates and related code first.

  3. Assistant

    Implementing the column-cap fallback: adding JSONOnly to TableDef, enforcing the 1500-column limit in the schema builder, and updating store templates for JSON-only domain tables.

  4. Assistant

    Implemented the SQLite column-cap guard in the typed-table emit path.

    Behavior

    • maxSQLiteResourceColumns = 1500 — safety cap with headroom under SQLite’s default 2000-column limit.
    • When a resource table exceeds the cap (including after parent_id injection for dependent sync), it falls back to JSON-only: id, data, synced_at.
    • Fallback clears typed columns, indexes, and FTS5 structures.
    • Emits store-fallback: <table> (<N> cols) → JSON-only to stderr at generation time.
    • Narrow resources are unchanged — typed columns, indexes, and FTS still emit when under the cap.

    Code changes

    internal/generator/schema_builder.go

    • Added JSONOnly on TableDef.
    • Added applyJSONOnlyFallbackIfWide() called from BuildSchema and again from schemaWithDependentParents (post-parent_id).

    internal/generator/generator.go

    • Added hasDomainTable template func (typed tables + JSON-only wide fallbacks).
    • hasTypedTable now excludes JSONOnly tables.

    internal/generator/templates/store.go.tmpl

    • CREATE TABLE and UpsertBatch dispatch use hasDomainTable.
    • JSON-only tables get a simple 3-column upsert (no field extraction, no FTS).

    internal/generator/templates/sync.go.tmpl

    • upsertSingleObject routes JSON-only domain tables through their Upsert* methods.

    Tests

    • TestBuildSchema_WideResourceFallsBackToJSONOnly — 3000+ fields → 3 columns + JSONOnly.
    • TestBuildSchema_NarrowResourceKeepsTypedColumns — normal resources keep typed columns.
    • TestGenerateStoreWideResourceJSONOnlyFallback — full generate, stderr log, 3-column CREATE TABLE, compiles, and internal/store tests pass (migration succeeds).

    HaloPSA’s control table (3747 columns) would now generate as a 3-column JSON-only table instead of failing at first sync.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.