Concurrency, processes and crashes
This page covers what is specific to the Python bindings: which calls work from several threads, from asyncio or from several processes, what a signal or a crash does to the process and the data, and how to write an update that concurrent writers cannot lose. What the engine guarantees at each isolation level is in the main ArcadeDB docs: see Isolation.
How to read the numbers
Every number on this page, except the ratios in Reads with fewer calls into the JVM, comes from probe scripts that were run on 2026-10-24 on one Linux x86_64 machine. The setup of each probe is next to its numbers. The builds in the table below are older than the current release, 26.11.1.post4, and the probes were not run again on it, so read each number as a measurement of the build that produced it. The scripts and their raw logs are not part of this repository yet.
| Label | Package | Python |
|---|---|---|
JVM |
|
3.12.13, GIL on |
Native |
The native wheel (formerly |
3.12.13, GIL on |
Native, Python 3.14t |
The same native wheel |
3.14.5, free-threaded, GIL off |
-
A cell is one run of a probe, unless the run count is given. A count shows that a failure can happen, or that it did not happen in that run. It is not a rate: another run gives another count.
-
A recommendation names the number of runs behind it. Where a probe did not run, or ran once, the page says so.
-
The Python 3.14t runs are shorter than the others: 3 minutes of load instead of 10, 500 operations per writer instead of 1,500, 10 instead of 20 iterations per task. The probe on several processes was not run there.
Threads
Share one Database object between the threads of a process.
The probe: one writer thread and 1, 4 or 16 reader threads in one process, all using one Database, for 10 minutes (3 minutes on Python 3.14t). The writer commits batches of 5 rows and rolls back every seventh batch. On every read, a reader checks that the row count is not below the rows the writer had already committed, that the count is a multiple of 5 (no batch half applied), that no row of a rolled-back batch is visible, and that a random batch reads back whole with matching checksums. At the end the probe reopens the database and counts again.
| Build | Reader threads | Rows committed | Rows found after reopen | Reads | Errors | Bad checksums |
|---|---|---|---|---|---|---|
JVM, Python 3.12 |
1 |
6,533,625 |
6,533,625 |
2,453,267 |
0 |
0 |
JVM, Python 3.12 |
4 |
1,856,195 |
1,856,195 |
2,817,025 |
0 |
0 |
JVM, Python 3.12 |
16 |
506,540 |
506,540 |
3,988,149 |
0 |
0 |
Native, Python 3.12 |
1 |
3,527,420 |
3,527,420 |
2,121,381 |
0 |
0 |
Native, Python 3.12 |
4 |
1,414,500 |
1,414,500 |
3,441,639 |
0 |
0 |
Native, Python 3.12 |
16 |
552,005 |
552,005 |
5,198,903 |
0 |
0 |
Native, Python 3.14t (GIL off) |
4 |
1,780,025 |
1,780,025 |
3,967,064 |
0 |
0 |
All 7 runs ended with no error, no bad checksum and the same number of rows before and after the reopen. This probe has one writer. For several writers, read the next section.
Reads with fewer calls into the JVM, two threads
Some read calls of the JVM build cross into the JVM fewer times than before: lookup_by_key without Java arrays, ResultSet.one() and count(), Document.get, and the transaction-stack mark. This was timed on the arcadedb-jvm package, built with the change and without it, in a mixed window of two Python threads on one Database: one reader thread and one writer thread. The reader rotates lookup_by_key with get, .one(), .count() and .first() on a graph of 20,000 vertices and 40,000 edges. The writer commits transactions of 10 vertex inserts, each with a CREATE EDGE. Each window ran 20 seconds after a 5 second warm-up of the same window.
| Figure (time per operation) | Clean pairs | Time with the change, as a ratio of the time without it (median, 95% interval) |
|---|---|---|
Mixed window, 2 Python threads (reads and writer transactions together) |
10 |
0.489 [0.475, 0.499] |
The same reads on 1 thread, no writer |
10 |
0.510 [0.507, 0.514] |
2 threads, per reader operation |
10 |
0.476 [0.461, 0.487] |
2 threads, per writer transaction |
10 |
0.992 [0.945, 1.007] |
Control: Cypher 2-hop count and |
10 |
1.016 [0.978, 1.037] |
A ratio below 1 means the build with the change needs less time per operation. The mixed window is the figure that was set before the run as the claim: its interval lies below 1. The one-thread row shows the single-thread reads got cheaper by a similar ratio, so these figures do not show a gain that is specific to concurrency. The gain is in the reader’s share of the window, and the writer’s share and the Cypher control did not move beyond their noise. The ratios come from one Linux x86_64 laptop with the two timed cores, Python 3.12, alternating order of the two builds, the first round dropped, and an A/A control (the same build twice) whose intervals contain 1. They are not measured at 4 or more threads, on the native build, or on other hardware.
Several writers
Two transactions that change the same record conflict at commit. The one that loses raises ArcadeDBError, and the name ConcurrentModificationException is in the message or in the cause chain. The engine treats it as retryable: see Isolation for when it is raised. db.run_in_transaction(fn, retries=…) runs fn in a transaction again after such a conflict. The plain calls, begin, commit and rollback, and what an error does to a transaction are on Transactions.
The probe: 8 threads, one shared Database, 4 counter records. Each operation is one transaction that changes one counter, chosen by a fixed pseudo-random sequence, and inserts one row with a unique key into a ledger type. Each thread makes 1,500 operations (500 on Python 3.14t), so 12,000 in total (4,000 on Python 3.14t). The probe counts a lost update as an operation whose transaction committed but whose increment is not in the final counters. Each cell below is the number of lost updates out of the number of operations, from one run.
| Update | Retry | Isolation | JVM, Python 3.12 | Native, Python 3.12 | Native, Python 3.14t (GIL off) |
|---|---|---|---|---|---|
Atomic SQL: |
|
default |
0 of 12,000 |
0 of 12,000 |
0 of 4,000 |
Atomic SQL: |
own retry loop |
default |
0 of 12,000 |
0 of 12,000 |
0 of 4,000 |
Read with |
|
default |
1,137 of 12,000 |
201 of 12,000 |
76 of 4,000 |
Read with |
own retry loop |
default |
2,883 of 12,000 |
331 of 12,000 |
108 of 4,000 |
Read with |
own retry loop |
|
0 of 12,000 |
0 of 12,000 |
0 of 4,000 |
The first two rows change the counter in the statement itself: UPDATE Counter SET n = n + 1 WHERE id = ?. The last three read the counter with SELECT n, compute n + 1 in Python, and write it back with UPDATE Counter SET n = ? WHERE id = ?. "Own retry loop" is the loop in A retry loop for REPEATABLE_READ: it retries only on the three Java exception names ConcurrentModificationException, NeedRetryException and LockTimeoutException, up to 60 attempts, with a random backoff of up to 100 ms. The last row starts each transaction with BEGIN ISOLATION REPEATABLE_READ.
Without any retry, the atomic update raised ConcurrentModificationException for 6,602 of 12,000 operations on the JVM build, 7,613 of 12,000 on the native build and 2,552 of 4,000 on Python 3.14t. The other operations committed (5,398, 4,387 and 1,448), and the counters added up to exactly those numbers. No write was lost without an error.
What the runs show:
-
An atomic update in the statement, with a retry, lost nothing: 0 lost updates in each of the 6 runs (2 retry styles, 3 builds).
-
Reading the value, computing in Python and writing it back lost updates under the default isolation level, also when every conflict was retried: with
run_in_transaction1,137 of 12,000 on the JVM build and 201 of 12,000 on the native build, with the own loop 2,883 and 331. In this probe the read is aSELECTand the write is a separateUPDATEwith the computed value. -
The same read-compute-write under
REPEATABLE_READwith the own retry loop lost nothing: 0 lost updates in each of the 3 runs, one per build.
Based on that, do the arithmetic in the statement and retry the transaction. If the new value needs a computation that SQL cannot do, start the transaction with BEGIN ISOLATION REPEATABLE_READ and retry as in A retry loop for REPEATABLE_READ. The evidence for each is one run per build, as in the table. Do not read a value, compute in Python and write it back under the default isolation level. The table does not cover the record object’s save() on the native build: see save() of a record object refuses a lost update.
# Atomic in the engine: the retry only has to repeat the whole statement.
db.run_in_transaction(
lambda: db.command("sql", "UPDATE Counter SET n = n + 1 WHERE id = ?", key),
retries=60,
)
save() of a record object refuses a lost update
record.modify().set(…).save() on a record that exists is a read-compute-write: the object holds the values it was read with, and save() writes the changed properties back. If another transaction changed the stored record after the object was read, that write used to overwrite the change without an error. The JVM build already refused it. The arcadedb (native) build now refuses it too: save() raises ArcadeDBError with ConcurrentModificationException in the message, and writes nothing.
The engine keeps no record version, so the native build compares values. Inside the transaction of the update, save() reads the stored record and compares it with the values the object held as read. A record that differs is a lost update. A record that another transaction changed and then changed back is not detected. A save() of a new record has nothing to compare and is not checked. A second save() of the same object does not conflict with the first, because after a successful save the object holds the values it wrote.
save() of an existing record is a write, so it needs an open transaction on both builds (with db.transaction():, or begin()). Outside one it raises ArcadeDBError ("Transaction not begun"). To retry after a conflict, read the record again and repeat the change in a new transaction: db.run_in_transaction(fn, retries=…) does this when fn does the lookup, modify(), set() and save() itself. A retry that reuses the old object fails again, because the object still holds the old values.
def rename(rid, name):
rec = db.lookup_by_rid(rid).modify() # read inside the retried function
rec.set("name", name).save()
db.run_in_transaction(lambda: rename(rid, "new name"), retries=60)
The check costs on the write side only. A laptop A/B on the native wheel (10 clean interleaved pairs for each figure, with a clean A/A control, reported as the median of the pair ratios of the new build against main, where above 1 means slower) measured one lookup_by_rid, modify(), set() and save() of a record in a transaction: 1.241 (95% interval 1.239 to 1.244) for a record with 5 properties and 1.426 (95% interval 1.412 to 1.437) for a record with 20 properties and a 768-float vector. Each save() now runs one extra SELECT, and when a changed value is not one the engine stores exactly as sent, the UPDATE also returns the stored record. All 14 read figures (lookups by RID and by key, first(), a 10,000-row SELECT, a two-hop expand(), a two-hop Cypher query, each on both record sizes, and two projection controls) stayed between 0.991 and 1.004 of main, so reads are unchanged. If you need atomic arithmetic, an UPDATE … SET n = n + 1 in the statement does not use this path.
A retry loop for REPEATABLE_READ
db.transaction() cannot retry, because a with block cannot run twice. This loop is the one the probe used for the last row of the table.
import random
import time
RETRYABLE = ("ConcurrentModificationException", "NeedRetryException", "LockTimeoutException")
def is_retryable(error):
# The Java exception name can be in the message of any error in the chain.
while error is not None:
if any(name in f"{type(error).__name__}: {error}" for name in RETRYABLE):
return True
error = error.__cause__ or error.__context__
return False
def run_repeatable_read(db, fn, attempts=60):
for attempt in range(attempts):
try:
db.command("sql", "BEGIN ISOLATION REPEATABLE_READ")
try:
result = fn()
db.commit()
return result
except BaseException:
if db.is_transaction_active():
db.rollback()
raise
except Exception as error:
if not is_retryable(error):
raise
time.sleep(random.random() * min(0.1, 0.002 * 2 ** min(attempt, 6)))
raise RuntimeError("retry budget exhausted")
def add_one(key):
n = db.query("sql", "SELECT n FROM Counter WHERE id = ?", key).first().get("n")
db.command("sql", "UPDATE Counter SET n = ? WHERE id = ?", n + 1, key)
run_repeatable_read(db, lambda: add_one(3))
asyncio
Call the bindings from asyncio through asyncio.to_thread or loop.run_in_executor. A call made directly on the event loop blocks the loop for as long as the call takes.
arcadedb.aio.AsyncDatabase does this wrapping for query, command, run_in_transaction and find_nearest, on a bounded thread pool of its own: see asyncio: AsyncDatabase. The probe below calls the bindings through asyncio.to_thread and run_in_executor directly.
The probe: 100 tasks, each making 20 operations (10 on Python 3.14t), so 2,000 operations (1,000 on Python 3.14t) per call style. One operation is db.run_in_transaction(fn, retries=60) with an atomic counter update and one unique ledger insert, then a count. A heartbeat task sleeps 10 ms in a loop and records how late it wakes up. Each cell below is that lateness in milliseconds, as the 99th percentile and the maximum. In the short phases, where the 99th percentile equals the maximum, there were too few heartbeats for them to differ. The two async_executor() rows use the bindings' AsyncExecutor with set_parallel_level(1) and set_parallel_level(4): each task inserts one row with command(…, callback=…, error_callback=…), and the callbacks hand the result to the loop with loop.call_soon_threadsafe.
| Call style | JVM, Python 3.12 | Native, Python 3.12 | Native, Python 3.14t (GIL off) |
|---|---|---|---|
|
63.2 / 89.8 |
7.4 / 10.0 |
11.4 / 11.4 |
|
108.7 / 108.7 |
2.2 / 9.7 |
3.8 / 3.8 |
Blocking call made directly on the event loop |
653.5 / 653.5 |
928.9 / 928.9 |
382.3 / 382.3 |
|
22.9 / 22.9 |
11.1 / 11.1 |
3.5 / 3.5 |
|
118.1 / 118.1 |
25.3 / 25.3 |
2.9 / 2.9 |
In all 15 phases (5 call styles, 3 builds) every operation landed (2,000 of 2,000 or 1,000 of 1,000), with no deadlock and no error. That is one run per cell, so no recommendation beyond "this works" follows from it. The phase that called the bindings directly on the loop stalled it for up to 653.5 ms on the JVM build, 928.9 ms on the native build and 382.3 ms on Python 3.14t. The lateness in the other phases differs between the builds. Measure it on your own workload if a bound matters to you.
Processes
One process owns an embedded database at a time. Start worker processes with spawn and give each its own database. If several processes must share data, run a server and use a driver: see Packages that talk to a server.
The probe has these cases. Each ran once per build.
| Case | JVM | Native |
|---|---|---|
|
All 4 workers finished with 2,000 rows |
All 4 workers finished with 2,000 rows |
A second process opens a database that another process holds open, read-write |
Failed with |
The same error. The first process kept its 130 rows |
|
3 children each, every child made its own database of 2,000 rows |
The same |
|
The child hung: the probe ended it after 60 s |
The child exited 0 and counted 100 rows |
|
The child hung: the probe ended it after 60 s |
The child exited 0 and counted 100 rows |
|
The child exited 0 |
The child exited 0 |
|
The child exited 0 |
The child exited 0 |
Based on that:
-
Use
spawnwith one database per process. It worked in 1 run on each build, with 4 workers. -
Do not fork a process that already created or opened a database. On the JVM build a child that used the inherited
Databasehung in 2 of 2 variants. On the native build the same two variants worked in 1 run each. That is too few observations to rely on, so treat it as unsupported on both builds. -
A fork before any database is created or opened worked on both builds, in 1 run per variant with 3 children each.
Signals and crashes
A write that has returned from commit() survived every kill in the probe. The probe cannot say anything about a power loss or an operating system crash, which a process kill does not simulate. For how the engine flushes its write-ahead log, see WAL flush and durability.
The kill probe: a child process writes in a loop, in transactions of 5 or 200 rows, and prints the number of rows after every commit that returned. The parent sends SIGKILL (kill -9) at a random time between 0.2 and 2.5 s after the first commit, then reopens the same database. The reopen must recover, CHECK DATABASE must report nothing, every acknowledged row must be present with a matching checksum, no transaction may be half applied, and the database must accept a write. The same database was used for all runs of a build.
| Build | Runs | Runs that passed every check | Commits acknowledged before the kill (per run) | Acknowledged commits lost | CHECK DATABASE findings |
|---|---|---|---|---|---|
JVM, Python 3.12 |
20 |
20 |
12 to 613 |
0 |
0 |
Native, Python 3.12 |
20 |
20 |
151 to 1517 |
0 |
0 |
Native, Python 3.14t (GIL off) |
5 |
5 |
700 to 1463 |
0 |
0 |
A killed process lost no acknowledged commit in any of the 45 runs, and every check passed.
The signal probe: a child opens a transaction and inserts rows without committing. The parent sends the signal 0.3 s after the first insert. After the child ended, or after 60 s if it did not, the parent reopens the database and runs the same checks. For the rows with a handler, the child must also report that it caught KeyboardInterrupt. Each build ran every row 5 times (2 times on Python 3.14t).
| Build | Signal | Main thread of the child, transaction open | Runs | Runs where the child hung | Exit status | Uncommitted rows visible after reopen | CHECK DATABASE findings |
|---|---|---|---|---|---|---|---|
JVM, Python 3.12 |
|
blocked in |
5 |
0 |
-6 |
0 |
0 |
JVM, Python 3.12 |
|
inserting rows in a loop |
5 |
0 |
-6 |
0 |
0 |
JVM, Python 3.12 |
|
blocked in |
5 |
5 |
none within 60 s |
0 |
0 |
JVM, Python 3.12 |
|
inserting rows in a loop, |
5 |
0 |
0 |
0 |
0 |
JVM, Python 3.12 |
|
inserting rows in a loop, no handler |
5 |
0 |
-2 |
0 |
0 |
Native, Python 3.12 |
|
blocked in |
5 |
0 |
-15 |
0 |
0 |
Native, Python 3.12 |
|
inserting rows in a loop |
5 |
0 |
-15 |
0 |
0 |
Native, Python 3.12 |
|
blocked in |
5 |
0 |
0 |
0 |
0 |
Native, Python 3.12 |
|
inserting rows in a loop, |
5 |
0 |
0 |
0 |
0 |
Native, Python 3.12 |
|
inserting rows in a loop, no handler |
5 |
0 |
-2 |
0 |
0 |
Native, Python 3.14t (GIL off) |
|
blocked in |
2 |
0 |
-15 |
0 |
0 |
Native, Python 3.14t (GIL off) |
|
inserting rows in a loop |
2 |
0 |
-15 |
0 |
0 |
Native, Python 3.14t (GIL off) |
|
blocked in |
2 |
0 |
0 |
0 |
0 |
Native, Python 3.14t (GIL off) |
|
inserting rows in a loop, |
2 |
0 |
0 |
0 |
0 |
Native, Python 3.14t (GIL off) |
|
inserting rows in a loop, no handler |
2 |
0 |
-2 |
0 |
0 |
-
Data. No uncommitted row was visible after any reopen,
CHECK DATABASEreported nothing and the database was writable, in all 60 runs. -
JVM build,
SIGINT. With the main thread blocked in one longtime.sleep(120)and a transaction open, the child did not react to Ctrl-C within 60 s in 5 of 5 runs. A child that was busy inserting rows in the same state reacted in 5 of 5 runs. The JVM installs its own handlers forSIGINTandSIGTERMwhen it starts, and Python learns of the signal only when the blocked call returns. A fix is open as arcadedb-python#26, which gives both signals back to Python after the JVM starts. Until it ships, expect a script that is blocked in one long call with a transaction open not to stop on Ctrl-C. -
JVM build,
SIGTERM. The process aborted with exit status -6 in 10 of 10 runs, where a plain Python process and the native build end with -15. The committed data was intact. #26 changes this as well. -
Native build. The child reacted to
SIGINTin every run: it caughtKeyboardInterruptwhere it had a handler and ended with status -2 where it had none.SIGTERMended it with -15 in every run. Both held on Python 3.12 and on Python 3.14t.
If a process must stop and it does not react to a signal, SIGKILL cost no acknowledged commit in 45 runs (20 on the JVM build, 20 on the native build, 5 on Python 3.14t).
Durability: surviving power loss
By default the engine does not flush its write-ahead log at commit: arcadedb.txWalFlush is 0. A commit that returned then survives a crash of the process, as the kill probe above shows, but not a power loss or an operating system crash. Setting arcadedb.txWalFlush to 1 flushes the log data to stable storage at every commit, and 2 flushes the file metadata as well. What each level guarantees is in the main ArcadeDB docs: see WAL flush and durability. The setting is described in Settings.
The call db.set_wal_flush(…), the group commit and file sync settings, and what a bulk load does with them are on Durability settings.
The setting belongs to the database. The same SQL turns it on for arcadedb (native) and arcadedb-jvm, and the value is stored with the database, so it is still in force after a reopen:
import arcadedb
def wal_flush(db):
row = db.query("sql", "SELECT settings FROM schema:database").first()
return next(s["value"] for s in row.get("settings") if s["key"] == "arcadedb.txWalFlush")
with arcadedb.create_database("./durable") as db:
print(wal_flush(db))
db.command("sql", "ALTER DATABASE `arcadedb.txWalFlush` 1")
print(wal_flush(db))
db.command("sql", "CREATE DOCUMENT TYPE Note")
with db.transaction():
db.command("sql", "INSERT INTO Note SET text = 'durable'")
0
1
The snippet ran on one build of each package (26.11.1.dev0, Python 3.12, Linux), and the value read 1 after a reopen on both. A system property sets the same value for every database in the process, as in Where the logs go: -Darcadedb.txWalFlush=1 in jvm_kwargs={"jvm_args": […]} or in ARCADEDB_JVM_ARGS for arcadedb-jvm, and in ARCADEDB_NATIVE_ISOLATE_ARGS for arcadedb. A database created in such a process reported 1 on both packages.
The flush has a cost on every commit. On one Linux laptop, a commit of one row took about 0.07 ms at level 0 and about 5 ms at level 1. That is one machine and one run, so read it as an order of magnitude. The cost per row falls when a transaction holds many rows: see Bulk loads and commit size.
On Linux, macOS and Windows the JDK asks the operating system for a flush that reaches the device, and the native image uses the same JDK code. How far a device keeps its promise is up to the hardware.
Free-threaded Python
The native wheel is a py3-none wheel, and it installed and ran on free-threaded Python 3.14.5 with the GIL off. The Python 3.14t rows in the tables above are those runs: threads, several writers, asyncio and the signal and kill probes. The probe on several processes was not run there. The runs are shorter than those on Python 3.12, as listed at the top of this page.
The JVM package was not run on Python 3.14t. Its wheels up to 26.11.1.dev0 were tagged for 3.11 to 3.14 (cp314), so uv found no wheel for 3.14t (it reported no wheel with a matching Python version tag for arcadedb-jvm 26.10.2.post1). The newer wheels are py3-none, and JPype has a 3.14t wheel on Linux, so uv can now install the package on 3.14t. No JVM test run has been made there, so treat it as untested.