ISSUE DESCRIPTION Time series: concurrent single-row SQL INSERTs silently lose samples…
- User
ISSUE DESCRIPTION
Time series: concurrent single-row SQL INSERTs silently lose samples (sealed-slot lost update)
Summary
Under many concurrent single-row
INSERT INTO <timeseriesType> SET ts=…, <tag>=…, <field>=…statements (each in its own transaction, as happens over the HTTP/wire protocols), the time series engine silently drops samples. The client receives success on every insert (HTTP 200, no exception, noConcurrentModificationException), yet the stored row count comes up short. Confirmed real data loss: a full table scan returns the same short count ascount(*), and aGROUP BY sensor_idshows specific series missing rows.The batched InfluxDB line-protocol path (
POST /api/v1/ts/{db}/write, which appends many samples in one engine call/transaction) is not affected. Single-threaded inserts are not affected.Impact
Silent, unacknowledged data loss on the time series write path under concurrency. Severity is high because there is no error surfaced to the client — an application doing concurrent per-row ingestion over SQL (HTTP or gRPC) loses samples without any signal.
Environment
- Observed on
26.6.1-SNAPSHOT(== thearcadedata/arcadedb:latestimage at time of writing). - Reproduced both embedded (in-process engine) and over HTTP against the Docker image.
Reproduction
A. Embedded (fast, deterministic at high concurrency)
A disabled regression test is included in PR #4450:
engine/src/test/java/com/arcadedb/engine/timeseries/TimeSeriesConcurrentInsertTest.javaIt creates a
TIMESERIES TYPE … SHARDS 1, runs 48 threads × 5000 single-rowINSERTs (each in its owndatabase.transaction(...)), then assertscount(*) == 48*5000. Result: e.g. 239911 / 240000 — 89 samples lost, with no exception and no failed transaction.Enable it by removing the
@Disabledannotation.B. Over HTTP (against the published image)
CREATE TIMESERIES TYPE sensor TIMESTAMP ts TAGS (sid STRING) FIELDS (v DOUBLE) SHARDS 1Then 16 threads, each POSTing 5000 distinct single-row inserts to
POST /api/v1/command/{db}:{"language":"sql","command":"INSERT INTO sensor SET ts=<unique>, sid='s<thread>', v=<i>"}SELECT count(*) FROM sensorreturns short by 10–22 out of 80000 across runs.Observations on the trigger
- Concurrency level (thread count) is the driver, not total row count. 5 threads often look clean; 16–48 threads reliably lose.
SHARDS 1maximizes it (all appends contend on one shard).- Single-threaded never loses.
- The unfiltered
SELECT count(*)reflects the true (short) count; a full row scan agrees with it (so it is real loss, not acount(*)artifact).
Root cause analysis
The write position for a sample is a read-modify-write of the per-data-page sample count. In
TimeSeriesBucket.appendSamples(engine/.../engine/timeseries/TimeSeriesBucket.java):final int sampleCountInPage = dataPage.readShort(DATA_SAMPLE_COUNT_OFFSET) & 0xFFFF; // read slot final int rowOffset = DATA_ROWS_OFFSET + sampleCountInPage * rowSize; dataPage.writeLong(rowOffset, timestamps[i]); // write row at slot ... dataPage.writeShort(DATA_SAMPLE_COUNT_OFFSET, (short) (sampleCountInPage + 1)); // advance slotTimeSeriesShard.appendSamplesserializes appends per shard with aappendLockaround a nesteddb.begin()/append/db.commit()ongetWrappedDatabaseInstance()(and the comment there notes the lock exists specifically to avoid a commit-timeConcurrentModificationExceptionon page 0). The append commit is nested inside the enclosing request transaction (LocalDatabase.begin()pushes a nestedTransactionContextwhen a transaction is already active), and the durable publish is effectively deferred to the outer transaction commit (intentionally, so the bucket page writes ship to followers via the parent Raft WALTX_ENTRY— the code references issue #4382). That outer commit happens outsideappendLock.A temporary collision detector added to
appendSamples(recordingfileId:pageNumber:slotper append) confirmed the mechanism: a following serialized append reads a stalesampleCountInPage— the prior append'scount → N+1increment is not visible — so two appends compute the samerowOffsetand write the same slot N; the later commit overwrites the earlier row. The page version advances normally, andPageManager.updatePageVersion's version check does not raise a conflict for these commits, so the existing outer-transactionretriesnever kick in.Sample collision-detector output (48-thread embedded run):
DIAG COLLISION slot=1:1:0 pageVer=239 prevThread=74 curThread=88 DIAG COLLISION slot=1:1:1 pageVer=240 prevThread=68 curThread=87 DIAG COLLISION slot=1:1:2 pageVer=241 prevThread=42 curThread=68 ...(slot reused across threads; page version high and advancing while the slot/count read is stale.)
Why the batched path is safe
TimeSeriesEngine.appendBatchgroups all of a shard's samples into a single transaction per shard, so there is never a second concurrent transaction reading a stale slot on the same page.Relevant code
engine/.../query/sql/executor/SaveElementStep.java—saveToTimeSeriesroutes a SQLINSERTintoengine.appendSamples.engine/.../engine/timeseries/TimeSeriesEngine.java—appendSamples(per-row) vsappendBatch(line protocol).engine/.../engine/timeseries/TimeSeriesShard.java—appendSamples(appendLock+ nested begin/commit on the wrapped instance).engine/.../engine/timeseries/TimeSeriesBucket.java—getOrCreateActiveDataPage+appendSamples(theDATA_SAMPLE_COUNT_OFFSETslot RMW).engine/.../engine/PageManager.java—updatePageVersion(the version conflict check that is not catching this).
Suggested direction (for discussion)
The core tension is that
appendLockserializes the in-memory append but the durable publish is deferred to the enclosing transaction's commit (kept that way so writes ship via the parent Raft WAL). Any fix needs to make the slot read-modify-write atomic with respect to the actual publish without breaking HA replication — e.g. ensuring the per-shard append publishes/serializes the slot under the lock, or ensuring the concurrent same-page commits raise a conflict so the existing outer-transaction retry path heals them. This is a delicate change to the transaction/HA path and warrants careful design plus HA + performance verification.Notes
- A reliable embedded regression test is included (disabled) in PR #4450; enable once fixed.
- The time series load tests in the same PR catch this at scale on the SQL protocols.
Interface
No new interfaces are introduced.
ADDITIONAL ADVICE
Since you are given a git repository, you can use git commands to simplify your work. For example, if you made a mistake and want to revert your changes, you can use
git checkout <file>to restore the file to its original state. You can also reset all changes in the repository usinggit reset --hardcommand. Additionally, you can usegit stashandgit stash popto temporarily save your changes and restore them later. Finally, you can see the changes that you've made (which will be submitted when you callsubmit) usinggit statusorgit diffcommands. However you don't need to usegit addorgit commitbefore submitting your changes. In fact,submitonly submits currently unstaged changes because it usesgit diffto get the changes that need to be submitted.CHECKLIST
Before submitting your solution, please go over the following checklist and make sure you've done everything:
- If an error was reported in the issue description, I have successfully reproduced it.
- If an error was reported in the issue description, I have confirmed that my fix resolves the error.
- I have thought about edge cases and made sure my fix handles them as well.
- I have run existing tests in the repository that might have been affected by the change I made and confirmed that they pass. I want you to list every bullet from this checklist and write your reasoning for why you think you did it or didn't need to.
Repository has been uploaded and your shell is currently at the repository root. Time to solve the issue!
(Current directory: /arcadedb, current file: none) bash-$
- Observed on
Assistant
Maven project. Let's create a scratch copy of the test with reduced scale for faster iteration.
Assistant
Now let's run this test with Maven.
Assistant
Good, reproduced the bug (240000 expected, got 239889). Now let's investigate the actual root cause more deeply.
Assistant
Now instrument TimeSeriesShard.appendSamples to record committed slots into a global set only after a successful commit, and clear on failure.
Assistant
Let's revert the diagnostic changes first, then closely re-examine the compaction phase 4a/4c logic to design the fix.
Assistant
Good, source files restored. Now let's look closely at the relevant methods used in compaction (
readPagesRangeForCompaction,getDataPageCount,getMaxSamplesPerPage) to design a precise fix.Assistant
Now let's verify the change compiles and check the surrounding logic once more.
Assistant
Now let's rebuild and rerun the reproduction test to confirm the fix works.
Assistant
Good, still enabled (48x5000) from before. Let's run it a few times to make sure it's reliably fixed.
Assistant
Test passes now. Let's run it a couple more times to be confident (since it's a race condition, flakiness is possible).
Assistant
All 3 runs pass consistently. Now let's restore the test file to its original state (with
@Disabled) since the checklist says not to modify existing tests, but let's re-enable it properly since the issue explicitly says "Enable once fixed."Assistant
Let's remove the now-unused
Disabledimport for cleanliness.Assistant
Now let's view the full diff of the main fix in
TimeSeriesShard.java.Assistant
Compiles fine. Now let's run the broader timeseries test suite to make sure nothing else broke.
Assistant
All 45 timeseries test files show "Failures: 0, Errors: 0". Let's double check via a summary count.
Assistant
All 555 tests pass, 0 failures/errors. Let's also run the full engine test suite (broader) to be extra safe, since our change touches shared compaction logic used elsewhere.
Assistant
Let's do a final review of the full diff before wrapping up.
Assistant
No leftover diagnostic code. Let's run the specific test one more time plus a couple of closely related compaction tests for final confidence.
Assistant
All targeted tests pass with 0 failures. The 2 failures/1 skipped were from stale reports of unrelated earlier tests (not part of this run). Let's confirm by checking timestamps on report files to identify which 2 are stale/unrelated.
Assistant
Summary
I investigated the reported silent data-loss bug in ArcadeDB's time-series engine under concurrent single-row SQL INSERTs.
Root cause found (via empirical diagnosis, not just the issue's hypothesis): The issue's own write-up hypothesized a stale-read on the per-page sample-count RMW inside the
appendLock-serialized section ofTimeSeriesShard.appendSamples. I instrumented the code with targeted diagnostics (collision detection scoped to successfully committed slots only, plus an in-flight counter to verify true serialization) and proved thatappendLockdoes fully serialize appends — no lock-order bug there.Instead, tracing the header page's
HEADER_DATA_PAGE_COUNTfield over the run showed it jumping backwards to0mid-test, with the exact byte pattern (MIN_TS=Long.MAX_VALUE,MAX_TS=Long.MIN_VALUE,sampleCount=0, magic preserved) produced only byTimeSeriesBucket.clearDataPages()— i.e., a background compaction run (auto-scheduled 5s afterCREATE TIMESERIES TYPEviaTimeSeriesMaintenanceScheduler) was racing with concurrent appends.The real bug is in
TimeSeriesShard.compactInternal()'s lock-free Phase 4a → Phase 4c hand-off: Phase 4a snapshotsphase4aPageCount = mutableBucket.getDataPageCount()and reads pages up to and including that page number, but that last page is by definition the currently active (not-yet-full) page thatappendSamples()can keep growing in place (same page number, more rows) while the write lock is released for Phase 4b's CPU-heavy compression. Phase 4c then only re-read pages strictly afterphase4aPageCount, so any rows appended into that same active page between Phase 4a and Phase 4c were captured nowhere — and then wiped by the finalclearDataPages().Fix: introduced
phase4aFullBoundary = phase4aPageCount - 1(mirroring the existing, correctlastFullPagepattern used for the Phase 0 → Phase 1 hand-off) so Phase 4a only treats pages that are guaranteed immutable (page numbers strictly below the current active page — new page numbers are only allocated once the previous page is full) as safely processed, and Phase 4c always re-reads fromphase4aFullBoundary + 1onward, guaranteeing the still-growing page is captured fresh with no gap.Verified: the previously-disabled regression test (
TimeSeriesConcurrentInsertTest, 48 threads × 5000 single-row inserts) reliably failed before the fix (short by 100-300+ rows) and passed consistently (4 runs) after the fix. Re-enabled the test per the issue's instructions. Ran the fullcom.arcadedb.engine.timeseriestest package (555 tests) plus a targeted re-run of compaction/shard tests — all green, no regressions.Assistant
Checklist review
-
If an error was reported in the issue description, I have successfully reproduced it. Yes. I enabled the disabled
TimeSeriesConcurrentInsertTest(48 threads × 5000 single-row inserts,SHARDS 1) and confirmed it reliably failed (e.g. 239875–239994 vs expected 240000), matching the issue's reported symptom of silent sample loss with no errors surfaced. -
If an error was reported in the issue description, I have confirmed that my fix resolves the error. Yes. After the fix, I ran the same test 4 times consecutively at full scale (48×5000) and it passed every time with the exact expected count (240000), with zero reported insert errors.
-
I have thought about edge cases and made sure my fix handles them as well. Yes. I verified: (1) the case where no growth occurs between Phase 0 and Phase 4a (
phase4aFullBoundarycorrectly clamps viaMath.maxso it never goes belowlastFullPage, avoiding negative/invalid ranges); (2) the case where multiple new full pages are created between phases (only the truly-immutable ones are consumed early, the last/active one is always deferred); (3) no double-reading/double-counting between the Phase 4a and Phase 4c page ranges (they are contiguous and non-overlapping by construction). I also traced why single-threaded runs never exhibit this (no possibility of a concurrent write landing in the vulnerable window) and whySHARDS 1/ high concurrency maximizes it (higher chance an append lands on the shared active page during the brief lock-free compaction window), consistent with the issue's own observations. -
I have run existing tests in the repository that might have been affected by the change I made and confirmed that they pass. Yes. I ran the entire
com.arcadedb.engine.timeseries.**test package (555 tests across 45 test classes) — all passed with 0 failures/errors. I also specifically re-ran the compaction/shard-related tests (TimeSeriesConcurrentAppendCompactTest,BucketAlignedCompactionTest,TimeSeriesShardTest,TimeSeriesCrashRecoveryTest,TimeSeriesConcurrentWriteNoDuplicatesTest) individually — all passed. (A full whole-repomvn testrun was attempted but exceeded the available time budget for this environment; the unrelatedFileManagerTestpermission-check failures seen in a stray earlier full-suite attempt are pre-existing environment artifacts of running as root and are unrelated to this change.)
I'm confident in this fix and ready to submit.
-
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.
Identical first prompt
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-27BCompare side by side →
Qwen/Qwen3.5-35B-A3BCompare side by side →
Qwen/Qwen3.5-35B-A3BCompare side by side →
Qwen/Qwen3.5-35B-A3BCompare side by side →