Skip to content
TheChatArchiveThe Chat Archive
composer-2.5· unknownHugging Face ibragim-bad/swe_rebench_07_2026_trajectories

Time series: concurrent single-row SQL INSERTs silently lose samples (sealed-slot lost…

  1. User

    Time series: concurrent single-row SQL INSERTs silently lose samples (sealed-slot lost update)

    Summary

    Under many concurrent single-row INSERT INTO <timeseriesType> SET ts=…, <tag>=…, <field>=… statements (each in its own transaction, as happens over the HTTP/wire protocols), the time series engine silently drops samples. The client receives success on every insert (HTTP 200, no exception, no ConcurrentModificationException), yet the stored row count comes up short. Confirmed real data loss: a full table scan returns the same short count as count(*), and a GROUP BY sensor_id shows specific series missing rows.

    The batched InfluxDB line-protocol path (POST /api/v1/ts/{db}/write, which appends many samples in one engine call/transaction) is not affected. Single-threaded inserts are not affected.

    Impact

    Silent, unacknowledged data loss on the time series write path under concurrency. Severity is high because there is no error surfaced to the client — an application doing concurrent per-row ingestion over SQL (HTTP or gRPC) loses samples without any signal.

    Environment

    • Observed on 26.6.1-SNAPSHOT (== the arcadedata/arcadedb:latest image at time of writing).
    • Reproduced both embedded (in-process engine) and over HTTP against the Docker image.

    Reproduction

    A. Embedded (fast, deterministic at high concurrency)

    A disabled regression test is included in PR #4450: engine/src/test/java/com/arcadedb/engine/timeseries/TimeSeriesConcurrentInsertTest.java

    It creates a TIMESERIES TYPE … SHARDS 1, runs 48 threads × 5000 single-row INSERTs (each in its own database.transaction(...)), then asserts count(*) == 48*5000. Result: e.g. 239911 / 240000 — 89 samples lost, with no exception and no failed transaction.

    Enable it by removing the @Disabled annotation.

    B. Over HTTP (against the published image)

    CREATE TIMESERIES TYPE sensor TIMESTAMP ts TAGS (sid STRING) FIELDS (v DOUBLE) SHARDS 1
    

    Then 16 threads, each POSTing 5000 distinct single-row inserts to POST /api/v1/command/{db}:

    {"language":"sql","command":"INSERT INTO sensor SET ts=<unique>, sid='s<thread>', v=<i>"}
    

    SELECT count(*) FROM sensor returns short by 10–22 out of 80000 across runs.

    Observations on the trigger

    • Concurrency level (thread count) is the driver, not total row count. 5 threads often look clean; 16–48 threads reliably lose.
    • SHARDS 1 maximizes it (all appends contend on one shard).
    • Single-threaded never loses.
    • The unfiltered SELECT count(*) reflects the true (short) count; a full row scan agrees with it (so it is real loss, not a count(*) artifact).

    Root cause analysis

    The write position for a sample is a read-modify-write of the per-data-page sample count. In TimeSeriesBucket.appendSamples (engine/.../engine/timeseries/TimeSeriesBucket.java):

    final int sampleCountInPage = dataPage.readShort(DATA_SAMPLE_COUNT_OFFSET) & 0xFFFF; // read slot
    final int rowOffset = DATA_ROWS_OFFSET + sampleCountInPage * rowSize;
    dataPage.writeLong(rowOffset, timestamps[i]);                                         // write row at slot
    ...
    dataPage.writeShort(DATA_SAMPLE_COUNT_OFFSET, (short) (sampleCountInPage + 1));        // advance slot
    

    TimeSeriesShard.appendSamples serializes appends per shard with a appendLock around a nested db.begin()/append/db.commit() on getWrappedDatabaseInstance() (and the comment there notes the lock exists specifically to avoid a commit-time ConcurrentModificationException on page 0). The append commit is nested inside the enclosing request transaction (LocalDatabase.begin() pushes a nested TransactionContext when a transaction is already active), and the durable publish is effectively deferred to the outer transaction commit (intentionally, so the bucket page writes ship to followers via the parent Raft WAL TX_ENTRY — the code references issue #4382). That outer commit happens outside appendLock.

    A temporary collision detector added to appendSamples (recording fileId:pageNumber:slot per append) confirmed the mechanism: a following serialized append reads a stale sampleCountInPage — the prior append's count → N+1 increment is not visible — so two appends compute the same rowOffset and write the same slot N; the later commit overwrites the earlier row. The page version advances normally, and PageManager.updatePageVersion's version check does not raise a conflict for these commits, so the existing outer-transaction retries never kick in.

    Sample collision-detector output (48-thread embedded run):

    DIAG COLLISION slot=1:1:0 pageVer=239 prevThread=74 curThread=88
    DIAG COLLISION slot=1:1:1 pageVer=240 prevThread=68 curThread=87
    DIAG COLLISION slot=1:1:2 pageVer=241 prevThread=42 curThread=68
    ...
    

    (slot reused across threads; page version high and advancing while the slot/count read is stale.)

    Why the batched path is safe

    TimeSeriesEngine.appendBatch groups all of a shard's samples into a single transaction per shard, so there is never a second concurrent transaction reading a stale slot on the same page.

    Relevant code

    • engine/.../query/sql/executor/SaveElementStep.java — saveToTimeSeries routes a SQL INSERT into engine.appendSamples.
    • engine/.../engine/timeseries/TimeSeriesEngine.java — appendSamples (per-row) vs appendBatch (line protocol).
    • engine/.../engine/timeseries/TimeSeriesShard.java — appendSamples (appendLock + nested begin/commit on the wrapped instance).
    • engine/.../engine/timeseries/TimeSeriesBucket.java — getOrCreateActiveDataPage + appendSamples (the DATA_SAMPLE_COUNT_OFFSET slot RMW).
    • engine/.../engine/PageManager.java — updatePageVersion (the version conflict check that is not catching this).

    Suggested direction (for discussion)

    The core tension is that appendLock serializes the in-memory append but the durable publish is deferred to the enclosing transaction's commit (kept that way so writes ship via the parent Raft WAL). Any fix needs to make the slot read-modify-write atomic with respect to the actual publish without breaking HA replication — e.g. ensuring the per-shard append publishes/serializes the slot under the lock, or ensuring the concurrent same-page commits raise a conflict so the existing outer-transaction retry path heals them. This is a delicate change to the transaction/HA path and warrants careful design plus HA + performance verification.

    Notes

    • A reliable embedded regression test is included (disabled) in PR #4450; enable once fixed.
    • The time series load tests in the same PR catch this at scale on the SQL protocols.

    Interface

    No new interfaces are introduced.

  2. Assistant

    Investigating the time-series append path and implementing a fix for concurrent single-row INSERTs losing samples.

  3. Assistant

    Implementing append in the enclosing transaction and holding appendLock until that transaction commits or rolls back, so slot updates aren't visible to other threads until publish.

  4. Assistant

    Trying nested commit for immediate persistence while holding the lock until the enclosing transaction commits.

  5. Assistant

    Implementing synchronized slot allocation in TimeSeriesBucket to make the per-page sample-count RMW atomic across threads.

  6. Assistant

    Re-implementing the enclosing-transaction path: append in the outer transaction and hold appendLock until commit/rollback.

  7. Assistant

    Implementing the root fix: after nested append commit, clear bucket pages from the enclosing transaction so its commit cannot overwrite persisted data.

  8. Assistant

    Summary

    Work targets silent sample loss on concurrent single-row time series INSERTs when a nested append publishes bucket pages while the enclosing SQL transaction can still commit stale bucket pages and overwrite newer data.

    Production changes (3 files)

    1. TimeSeriesShard.java

      • When an enclosing transaction is active: keep appendLock until that transaction commits or rolls back (via afterCommit / afterRollback callbacks).
      • Still use a nested HA-wrapped transaction for the actual write (issue #4382).
      • After nested commit, removeFile() on the enclosing transaction so it does not commit stale bucket pages over nested publishes.
    2. TimeSeriesBucket.java

      • synchronized slot allocation for the per-page sample-count RMW.
      • max(transaction, committed) sample counts for data pages, header totals, and data-page count.
      • Merge committed row bytes into the TX page when the committed page is ahead of the in-TX view.
      • PageId uses database.getWrappedDatabaseInstance() for cache consistency.
      • getSampleCount() reads the header from the active TX when present, otherwise from the page manager.
    3. TransactionContext.java

      • addAfterRollbackCallback / addAfterRollbackCallbackIfAbsent so the shard can release appendLock on rollback.

    Verification

    • TimeSeriesBucketTest, TimeSeriesConcurrentWriteNoDuplicatesTest, TimeSeriesEngineTest pass.
    • TimeSeriesConcurrentInsertTest remains @Disabled (unchanged from HEAD). With @Disabled removed, it still fails (~80–130 missing rows out of 240000 in repeated runs); the regression is not fully resolved yet.
    • git diff --check is clean.

    Next steps (if you continue)

    The remaining gap is likely outer-transaction page caching beyond removeFile, or count/slot visibility under extreme concurrency; worth profiling with @Disabled removed and iterating on enclosing-TX page eviction and/or commit-order invariants on the bucket file. I can keep pushing on that if you want this test green before enablement.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.