High Availability

To start an ArcadeDB server with HA enabled, at minimum you need to:

Example starting a 3-node cluster on a single host (different ports):

The Raft gRPC port is 2434 by default and is configurable via arcadedb.ha.raftPort. The HTTP port in each entry is required so that replicas can forward non-idempotent requests to the leader over HTTP.

Each comma-separated entry in arcadedb.ha.serverList can be written in either of two interchangeable forms. The two forms may be freely mixed in the same list, and both accept the optional name@ prefix (see Named Peers).

The following two entries are equivalent:

A client protocol’s routing view tells a driver which node to send writes to and which nodes it may read from, so it deals in the ports clients dial - never the Raft port the cluster uses to talk to itself, and not necessarily the HTTP port either.

Declare bolt and grpc whenever your nodes do not all listen on the same port for that protocol. With the field omitted, a peer’s address for that protocol is derived as that peer’s host plus this node’s own port. That is correct only for a homogeneous deployment where every node listens on the same port - a Kubernetes StatefulSet, for example - and a one-time WARNING per protocol is logged whenever the fallback is used.

On a cluster like that one, leaving the ports undeclared makes every peer derive to the same address - and an address two peers both claim identifies neither of them, since two listening sockets cannot share one host:port. Rather than advertise a leader address that is really the address of the node answering (which would send a self-redirecting client straight back to the node that just refused it), ArcadeDB advertises nothing for that protocol: BOLT then offers this node as READ and ROUTE but never as writer, and a refused gRPC call falls back to naming the leader’s HTTP address. A WARNING naming the field to declare is logged once per protocol. (Since v26.9.1)

A peer that joined the Raft group at runtime (a Kubernetes scale-up beyond arcadedb.ha.serverList, connect cluster, POST /api/v1/cluster/peer) is part of the routing view like a listed one: the Bolt ROUTE table offers it as a reader, and as the writer while it leads, a gRPC call refused on a follower names it when it leads, and its address is derived the same way from its Raft host. (Since v26.11.1: such a peer was replicated to but never routed to, and the workaround was to pre-size arcadedb.ha.serverList to the largest node count.)

The same derive-from-this-node’s-port fallback applies to the endpoints the nodes use to reach each other: a follower downloading a snapshot from the leader, a cluster verify comparing checksums with a peer, and the database presence matrix (GET /api/v1/cluster?presence=true). When two peers resolve to one host:port, that address identifies at most one of them and nothing can say which, so ArcadeDB withholds it rather than guessing: the operation is refused and reported as refused - a peer comes back ERROR / "not verified" from a verify, and unreachable from a presence matrix - instead of quietly answering for the wrong node. (Since v26.9.1)

The presence matrix lists every server the node knows: the server list plus any server that joined at runtime (a Kubernetes scale-up, connect cluster). Each one is either a column of the matrix or listed in unreachable, and a declared server the cluster no longer counts as a member is also listed in notInConfiguration, since a database it lacks is not a replication gap. (Since v26.11.1: before, a pod that joined at runtime was missing from the matrix.)

The refusal is visible in three places:

One request is exempt: the capability check, in which the leader asks each peer which replication formats its build understands so it can use a compact one only when every node can read it. That call changes nothing on the peer and its reply says who answered, so ArcadeDB dials the shared address anyway and credits the answer to the node that identified itself. A cluster that has never declared its http ports therefore still gets the compact formats, instead of falling back to the verbose ones forever with nothing to explain why. (Since v26.10.1)

On the common single-host layout where every node shares an address and the Raft and HTTP ports move in step (2434/2480, 2435/2481, …​), the capability check derives each peer’s HTTP port from that same offset, so it reaches the peers the shared-address fallback cannot name. The derived port is used for the capability check only - never for a snapshot download, a cluster verify or a forwarded write, which keep refusing a shared address. (Since v26.10.1)

A peer that still did not answer carries capabilitiesUnknownReason next to its entry in GET /api/v1/cluster, saying whether it is unreachable, running an older build, or not identified by any address of its own - the last being the one you fix by declaring its http port. (Since v26.10.1)

Every node also learns the HTTP address of a peer it cannot reach where it resolves it, the same way on every node. (Since v26.10.2) Each node states its own HTTP port and address in the capability check it sends to the others, and each answer carries the addresses the answering node holds for the other members. Neither is trusted on its own: an address is used only after the peer it names answers there under its own id. From then on that address serves every peer-to-peer call (snapshot downloads, verifies, forwarded writes) and is passed on to the other nodes. In practice, a peer admitted without an http port (connect cluster host:raftPort, POST /api/v1/cluster/peer), or declared on one node only, becomes reachable from every node within a capability round or two, whatever the port layout. An address this node’s own arcadedb.ha.serverList declares is never replaced. Only the port is learnt: an offered address is accepted only on the peer’s own Raft host. On a cluster with arcadedb.ssl.enabled, offered addresses are not probed at all, because the probe carries the cluster token and would send it over plain HTTP.

By default, each node identifies its slot in arcadedb.ha.serverList from the numeric suffix of arcadedb.server.name (for example, ArcadeDB_0 maps to the first entry, ArcadeDB_1 to the second, and so on). Display names shown in logs and Studio are then synthesized from the local node’s prefix. This positional convention works well for Kubernetes StatefulSets but is awkward when nodes have human-readable names such as frankfurt, london, nyc.

You can add an optional name@ prefix to any entry in arcadedb.ha.serverList to give each peer a stable, human-readable name:

When peer names are configured, each node identifies its slot by matching arcadedb.server.name against the configured peer names first; if no match is found, it falls back to the legacy prefix_N / prefix-N suffix resolution. This means:

The Raft peer ID is still derived from the address (e.g. 10.0.0.1_2434), so changing or omitting peer names does not affect Raft identity stability.

ArcadeDB uses a leader/replica model with Raft consensus. At any time one server holds leadership and accepts writes; the others are replicas that serve reads and stand by for failover.

Each server persists its own Raft log segments under <rootPath>/raft-storage/. The log is used for recovery after restart and to replicate state to peers that fall behind.

Any read (query) can execute on any server in the cluster. All writes must go through the leader: a replica that receives a write transparently forwards it to the current leader via HTTP.

Internally, every committed write follows a replicate-first, commit-after three-phase commit:

If Phase 2 times out or fails, Phase 3 never runs: no local writes, no divergence. Multiple concurrent transactions are batched into fewer Raft round-trips via group commit (HA_RAFT_GROUP_COMMIT_BATCH_SIZE).

Phase 3 runs on the Raft apply thread of every node, the leader included, when the entry reaches its position in the log: the leader publishes the pages its transaction prepared, a replica applies the entry’s WAL bytes. Every node therefore writes its pages in log order, and the leader’s own commit can never overtake or trail the apply of a neighbouring entry.

Two transactions validated on different nodes against the same version of a page (for example two documents that share a page, each written by a different node) are both shipped to the leader with the same next version. The leader validates every entry, page by page, against the versions the log has already assigned, at the point that decides the log order, and refuses the second one before appending it: the originator receives a retryable ConcurrentModificationException, exactly as it would for a conflict on a single node, re-reads the page and retries. An entry is never merged onto another one region by region, so a node can neither lose an acknowledged write nor drift from its peers. While such an entry is in flight, a transaction on the leader that touches one of its pages fails its own Phase 1 with the same retryable error, without a round trip.

Schema DDL does not travel through the WAL pages described above. A schema change is recorded in a schema recording session, and when the outermost session on the thread closes, the leader replicates it as its own Raft entry type (SCHEMA_ENTRY), carrying the schema document and any bucket or index files the change created. One session is one entry.

Normally each DDL statement is one session. Since v26.10.1 a SQL script made only of schema definition statements runs as a single session when arcadedb.schemaBulkDDLScript is on (the default), and so does a Java Schema.bulkChange() block: the whole batch is one Raft entry (see reference/sql/sql-script.adoc#sql-script-ddl-batching). The trade-off is that the database write lock is held for the whole script.

That matters on the leader. Before appending an ordinary transaction, a commit waits for an active schema session to finish, so the transaction cannot reach the log ahead of the schema entry that creates the files it writes to. The wait lasts at most arcadedb.ha.quorumTimeout (10 s by default); after that the commit goes ahead anyway. A DDL script that runs longer than that leaves concurrent writers racing the schema entry. A long index build already opens the same window, so this is not new, but batching makes long sessions more common. Keep DDL scripts to schema definition, run data loads separately, or set arcadedb.schemaBulkDDLScript=false to go back to one entry per statement.

On the follower, since v26.10.1, a DDL entry is applied incrementally: only the components for the files the entry created are instantiated, instead of every component in the database being rebuilt (arcadedb.ha.schemaIncrementalApply, on by default). Building a large schema used to cost time quadratic in the number of types.

Since v26.10.2 the schema changes that used to rewrite schema.json on the leader alone travel in a SCHEMA_ENTRY too, so every follower, and the node that becomes leader after a failover, enforces the same rules:

In 26.10.1 and earlier these changed the leader’s schema only: a MANDATORY constraint set on one leader, for example, was not enforced by the next one after a failover.

Calling one of these setters through the Java API on a follower’s database now fails, as every other DDL does, with a ServerIsNotTheLeaderException (Changes to the schema must be executed on the leader server) instead of changing that follower alone. Send the statement through SQL instead, which a follower forwards to the leader. Schema.setZoneId() now also requires the updateDatabaseSettings permission, like the two format setters.

An entry the incremental path cannot express makes the follower rebuild every component of the database on its apply thread: an entry that retires a file, adds a dictionary, a compacted index or a bloom filter, writes pages into a compacted index or a bloom filter, or renames component files, and any entry while arcadedb.ha.schemaIncrementalApply is false. On a large schema that is what shows up as an apply-lag spike after a DDL statement, and it re-creates every index instance cold (see the note on vector indexes below).

It is no longer silent. Each fallback is logged at INFO with the database, the entry’s Raft index, the reason, what triggered it and how long the full load took:

and published as two metrics, tagged by reason, that are present from the first scrape and restart from 0 with the process:

The reason tag is one of incremental_disabled, file_renamed, not_loaded, file_retired, non_incremental_new_file, non_incremental_touched_file and touched_file_missing.

ALTER TYPE <name> NAME <new-name> (and the Java DocumentType.rename()) renames the type’s files as well as the type: its buckets, the files of its indexes and, for a vertex type, its edge buckets. In 26.10.1 and earlier only the leader renamed them. Since v26.10.2 the rename replicates with the schema change: the SCHEMA_ENTRY names every file that was renamed, and each follower renames the same bucket, index and edge files before it loads the new schema. That load is a full one (reason file_renamed, see When a follower falls back to a full schema load (from v26.10.2)). Sent to a follower, the statement is forwarded to the leader like any other DDL.

A node running an older build cannot rename the files, and would drop the type’s buckets on its next schema reload. So while any peer of the Raft configuration runs an older build, or cannot be reached to say which build it runs, the rename is refused up front with HTTP 409 Conflict (Cluster is not ready for this operation, with the missing peers in exceptionArgs), and nothing is renamed on any node. Finish the rolling upgrade, or restore contact with the peer, and run the statement again.

No setting overrides this refusal: arcadedb.ha.securityEntryCapabilityGate does not apply to it. To rename a type while a peer is gone for good, remove that peer from the Raft configuration first (DELETE /api/v1/cluster/peer/{peerId}, see Cluster Management REST API).

By default each SCHEMA_ENTRY carries the whole schema document, so on a schema of several megabytes a one-property CREATE PROPERTY costs megabytes on the leader’s CPU, on the wire, in every node’s Raft log and in each follower’s rewrite of schema.json. With arcadedb.ha.schemaDelta (on by default) the leader ships a delta against the schema the followers already hold. It falls back to the whole document whenever a delta cannot be trusted: the first schema entry after it becomes leader, an entry shipped in instalments, a compaction entry, or a change too large for a delta to be worth it.

Upgrades need no ordering. A node running a version that predates the delta cannot see it: it would apply an empty schema change and its schema would diverge from the leader’s without an error. To prevent that, the leader asks every peer in the Raft configuration, through POST /api/v1/cluster/capabilities, whether it can decode a delta, and ships whole documents unless every peer has said yes. A peer that is unknown, unreachable, too old to have the endpoint, or whose answer has gone stale counts as no. During a rolling upgrade, deltas start on their own once the last node runs v26.10.1 or later.

While the setting is on, the leader keeps one parsed copy of the schema document per replicated database in heap to diff the next change against: tens of MB per database on a multi-MB schema. Turning the setting off releases it and forces whole documents, which is also useful when diagnosing a schema divergence. Turning it off is safe at any time; turning it back on costs one whole-document entry per database to rebuild the diff base.

Whether deltas are actually being shipped is not exposed through metrics yet. The counters are available only from Java, through RaftReplicatedDatabase.getSchemaDeltasShipped() and getSchemaDocumentsShipped(), and with arcadedb.ha.logVerbose=2 a follower logs Applying a <n>-char schema delta to database '<name>' …​ for each one.

ALTER DATABASE and the set database setting server command change settings that the database keeps in its configuration.json. In 26.10.1 and earlier they wrote that file on the node that ran the command only, so the followers, and a new leader after a failover, kept the previous value. For arcadedb.dateImplementation and arcadedb.dateTimeImplementation that changed what reads returned, not only how the node was tuned.

Since v26.10.2 a changed setting is replicated as its own Raft entry, and every node, the one that ran the command included, applies exactly the keys the entry names. A setting the command did not change keeps each node’s own value. The command returns once its own node has applied the change.

Since v26.11.1 an ARRAY_OF_FLOATS or ARRAY_OF_DOUBLES value is stored fixed-width (see Data Types), a form older builds cannot read. It is not a Raft entry but part of the record bytes in the pages a transaction replicates, so a node on an older build would apply them and then fail every read of the record. No node writes the new form until every peer of the Raft configuration advertises the fixed-width-float-arrays capability; until then the old form is written, which every build reads. During a rolling upgrade the new form starts on its own once the last node runs v26.11.1. Both forms are always read, so a cluster holding a mix of them needs no action.

A quorum is the number of peers that must acknowledge a write before it commits. Configure it via arcadedb.ha.quorum:

If the configured quorum is not met within arcadedb.ha.quorumTimeout milliseconds, the transaction is rolled back and an error is returned to the client.

When a read runs on a replica, ArcadeDB offers three consistency levels, configurable per-server via arcadedb.ha.readConsistency:

A node that is no longer a member of the Raft configuration (it left with leaveCluster or was removed with DELETE /api/v1/cluster/peer/{id}) receives no more entries. On such a node, linearizable reads and read_your_writes reads whose bookmark it has not applied are refused with a retryable error naming the reason, so the client can retry on a member. This includes reads that were already waiting when the node was removed: they are refused within a second instead of after the quorum timeout. eventual reads, and read_your_writes reads whose bookmark the node already applied, are still served. (Since v26.10.2: such reads waited the whole quorum timeout, and a read_your_writes read then served data missing the write its bookmark named.)

The same node refuses at once, with a retryable NeedRetryException, everything that would submit an entry to the cluster: a commit that carries writes (a read-only transaction still commits), a database drop, a database create or install, an ALTER DATABASE configuration change, and the replication of a security user, group or API token. The message names the reason and the way out, for a database operation Database '<name>' cannot <operation> on this server: it is not a member of the Raft cluster’s configuration any more (it left or was removed), so it receives no entries and could not apply this one. Send the request to a server of the cluster, or add this server back to it and for a security document Cannot replicate the security <document> from this server: it is not a member of the Raft cluster’s configuration any more (it left or was removed), so it receives no entries and the change would apply on every other server but not here. Send the request to a server of the cluster, or add this server back to it. (Since v26.10.2: the leader still accepted the entry and committed it on every member, while the removed node waited arcadedb.ha.quorumTimeout for an apply it never received and then released the transaction unapplied; a drop dropped the database on every member and then failed on the removed node, and a security change was in force everywhere except on the node that answered the request.)

Clients communicate the consistency contract through three HTTP headers:

The bookmark is a lower bound: it says the server had applied at least that index when it answered. That holds on a refused request too, so a client can keep its read-your-writes barrier after an error instead of falling back to a stale one.

The one endpoint where it is not a header is the streaming bulk load, POST /api/v1/batch/{database} with Accept: application/x-ndjson. There the response has already started by the time the commit index is known, and a header set at that point would be dropped silently, so the value travels in the body instead, as commitIndex on the terminal summary or error line. The Java driver handles this transparently: RemoteDatabase.getLastCommitIndex() advances the same way whichever encoding was used.

Since v26.10.2 a replica relays such a streamed load to the leader in both directions at once, so a client that sends it to a replica receives the leader’s progress lines while it is still uploading, as it does on the leader. A body that grows past arcadedb.server.httpBodyContentMaxSize after the leader started answering ends the stream with an error line carrying status 413 during the upload too. In 26.10.1 and earlier the replica handed back the leader’s answer only once the whole upload had been sent.

The Raft group-commit queue is bounded (arcadedb.ha.groupCommitQueueSize, default 10000 pending transactions). When a write arrives but the queue is full, the server waits up to arcadedb.ha.groupCommitOfferTimeout ms (default 100) for a slot, and if still full throws ReplicationQueueFullException — a NeedRetryException that clients automatically retry with backoff. This protects the server from OOM under sustained write overload and lets clients back off gracefully.

Set arcadedb.ha.serverRole=replica to pin a node as a permanent replica. The node’s Raft priority is set to 0 at startup so it is never elected leader, while still receiving all writes and serving reads. Useful for read-scale deployments or for cross-DC witnesses that should not take over as leader.

If the leader becomes unreachable, replicas start a new Raft election. A replica with an up-to-date log is elected as the new leader and the cluster resumes serving writes. Pre-vote prevents partitioned nodes from triggering disruptive elections.

Common causes of leader unavailability include:

When a replica rejoins after being offline, it catches up via the Raft log automatically; if it has fallen behind past the log purge boundary, the leader streams a full database snapshot over HTTP and the replica installs it atomically. Since v26.10.2 the snapshot is compressed at DEFLATE level 1 by default (arcadedb.ha.snapshotCompressionLevel), and two followers are served at the same time; see Performance Tuning for when to change either.

A vector index’s proximity graph is built by every node on its own, so it is not part of the data the nodes compare or share. Since v26.10.2 a snapshot ships the graph file (.vecgraph) empty, without its side files, and the follower builds and verifies a graph of its own on the first search of each vector index after the install, the way every node does after CREATE INDEX. That first search pays for one graph build. In 26.10.1 and earlier the follower received the leader’s graph pages without the manifest that vouches for them and reused them unverified: each such reuse is counted in unverifiedGraphReuses in the index’s getStats() and logged at WARNING (Vector index <name> is reusing a persisted graph that carries no manifest …​). A follower that installed a snapshot before upgrading keeps that graph until its next graph persist, a REBUILD INDEX, or its next snapshot install.

The leader also auto-recovers a follower’s replication channel that is wedged on a stale peer address — for example after a Kubernetes pod is rescheduled onto a new IP: it re-resolves the peer and re-establishes the channel on its own, instead of leaving the follower silently disconnected and the cluster at bare quorum until an operator forces a leadership transfer. See Replication-Channel Self-Healing.

A follower can be up, healthy, and reachable by every other node, and still receive nothing from the leader. The leader keeps one long-lived gRPC channel per follower. If that channel is wedged on a peer address that is no longer valid (the classic case is a Kubernetes pod restarting onto a new IP inside the JVM’s positive DNS cache window), the leader’s appender for that follower keeps writing into a socket that goes nowhere. The follower’s matchIndex stops advancing, or never advances at all (matchIndex=-1), while the rest of the cluster commits normally. Quorum is unaffected as long as a majority is still healthy, so the cluster keeps serving writes at reduced redundancy and nothing fails loudly.

ArcadeDB detects and repairs this on its own, in three escalating steps.

1. Detect: the follower is reported unreachable. The leader’s cluster monitor tracks the time since the last successful RPC to each follower. Once that exceeds arcadedb.ha.peerUnreachableThreshold (default 10s), the follower is reported unreachable. This is a diagnostic signal only: it does not change Raft membership, quorum, or the write path. It is, however, the input every step below depends on, so setting it to 0 disables the whole self-healing sequence.

2. Reset: rebuild that one follower’s channel. Once a follower has stayed continuously unreachable for arcadedb.ha.peerChannelResetDuration (default 60s), the leader closes that follower’s replication channel so the next send re-resolves DNS and reconnects. Only the unreachable peer’s channel is touched; the other followers, and leadership itself, are left alone, so there is no flapping risk. A follower that reconnects inside the interval clears the streak and is never reset, so an in-progress reconnection is not disrupted.

A single reset does not always stick, because the rebuilt channel may still resolve the stale address if the JVM DNS cache has not expired. The reset is therefore retried once per interval, up to 5 attempts. The counter re-arms as soon as the follower becomes reachable again. Set the duration to 0 to disable the automatic reset and keep only the manual leadership transfer.

3. Escalate: transfer leadership to rebuild the appender. The retry budget alone leaves a gap: the streak only re-arms when the follower becomes reachable, so a permanently wedged channel would sit in the given-up state until an operator restarted the leader process. arcadedb.ha.peerChannelResetEscalation (default true) closes it. When the budget is exhausted and the channel is still dead, the leader transfers leadership to a healthy peer, so the new leader builds a brand-new appender to the stuck follower.

The target is selected with the same rules as a manual step-down and is never the wedged follower itself. If no healthy peer is eligible, the leader logs at SEVERE and leaves the follower for the operator.

The escalation is bounded, not perpetual. Leadership transfer is exactly what makes a new leader inherit the problem, so each healthy peer escalates a given follower at most once per 30-minute cooldown. If the follower is unreachable for a reason a fresh appender cannot fix (it is genuinely down, or partitioned away), the cluster passes leadership around at most once per peer and then settles on the operator-intervention path instead of churning. Set the flag to false to keep the detection and logging but never move leadership automatically.

With ephemeral Raft storage, wiping the log on restart could leave the cluster in a broken state:

Durable storage lets a restarted node rejoin by replaying its own persisted log rather than always requiring a full snapshot resync from the leader.

Migration: the per-node directory is named raft-storage-<nodeName> dynamically, which container runtimes and Kubernetes cannot mount as a volume (a wildcard like raft-storage- is created as a literal directory, not expanded). Set arcadedb.ha.raftStorageDirectory to a *static path (e.g. /home/arcadedb/raft-storage) and mount that path on durable storage before upgrading. On Kubernetes, back it with a PersistentVolumeClaim — see Kubernetes. On a plain filesystem you can instead copy the existing raft-storage-<nodeName> directory across the upgrade. A throwaway or test cluster that intentionally wants ephemeral behavior can opt out with arcadedb.ha.raftPersistStorage=false.

When a write is forwarded from a replica to the leader (or issued directly to the leader), it is possible for the transaction to be committed cluster-wide by quorum and then fail to apply on the node that is answering the client. This is a distinct outcome from an ordinary failure: the data is durably committed, so retrying would create a duplicate.

This case has its own contract. The server raises a TransactionCommittedRemotelyException, mapped over HTTP to status 409 Conflict. A client that receives it must not retry the transaction — the write already succeeded cluster-wide; it should instead re-read to observe the committed state. This is different from a transient conflict (a genuine write-write conflict or DeadlockException), which remains safe to retry.

A replica forwards the writes it receives to the leader (see High Availability Concepts for the writes it runs itself, inside a transaction the client opened). Some writes cannot be forwarded and are refused instead: a schema change sent over a protocol with no forward, or any write when the forward itself cannot be made. Such a refusal names the leader, and since v26.10.2 every wire protocol answers it as a redirect, in that protocol’s own terms, with the leader’s address in the text or in a field of its own:

In 26.10.1 and earlier Bolt, MongoDB, PostgreSQL and Redis answered such a refusal as an ordinary write conflict (Neo.TransientError.Transaction.DeadlockDetected, 112 WriteConflict, 40001, TRYAGAIN) without the leader’s address, so a client retried the same replica, which refused again. Gremlin Server and MCP answered it without the leader’s address too, and in production mode with the bare placeholder.

The address is added after the message is concealed in production mode, so it reaches the client in every mode. A refusal raised during an election, when no leader is known yet, names nobody and keeps its retryable answer (503 over HTTP): retry shortly.

Each server keeps its own list of running statements: the HTTP list queries and terminate query server commands and Cypher SHOW TRANSACTIONS and TERMINATE TRANSACTIONS run on the server that receives them, a replica included, and are never forwarded to the leader. A statement id names a statement on the server that issued it.

Since v26.10.2 a write that a replica forwards to the leader can be followed and stopped from either side:

The forward carries the statement’s id under the cluster token, which an HA cluster always has configured (see Security). The usual visibility rule applies on both servers: root sees and stops every statement, any other user only their own.

By default, when an HA cluster forms for the first time and one or more peers already hold a non-empty database on local disk, the peers do not download the database over HTTP from a single source. Instead, every peer reports a (fingerprint, lastTxId) tuple per database, the cluster elects the peer with the highest lastTxId as the source via leadership transfer, and any peer whose fingerprint matches the source bootstraps locally in seconds with zero bytes transferred.

This is the recommended path for scaling a single-instance ArcadeDB deployment up to a multi-node HA cluster after a large dataset has already been imported. Operator workflow:

Time-to-cluster goes from "minutes-to-hours of HTTP snapshot transfer per replica" to "seconds, because everyone already has the bytes". For the Kubernetes recipe, see Pre-staging the database on every pod.

The bootstrap protocol handles the following cases automatically:

While the cluster is deciding which copy to keep, every node - the leader included - answers 503 on /api/v1/ready and refuses client requests on the databases being decided, then serves normally as soon as the decision is applied. This usually takes seconds; if a peer is unreachable it can last up to arcadedb.ha.bootstrapTimeoutMs. (Since v26.10.1 the leader is held too: before, it served its own copy while collecting, and a write accepted there could stop the cluster from ever agreeing on a baseline.)

The committed bootstrap baseline (fingerprint and lastTxId per database) is visible in the response to GET /api/v1/cluster and in the Studio cluster dashboard.

Most problems a node reports are temporary: it is catching up, or downloading a snapshot, and it rejoins on its own. Two are not, and since v26.10.1 GET /api/v1/cluster says so directly:

Both are null on a healthy node, and both raise a matching entry in alerts. A third field, crashLoopEscalated, is true once the node has given up restarting itself and needs to be restarted by hand.

alerts also distinguishes two situations that used to look the same. bootstrap-diverged-databases means this node kept a copy that the cluster never adopted - the data is here and intact. bootstrap-database-missing means the opposite: this node is missing a database the cluster has. Do not copy a node’s database directory to the other nodes when it is the one missing the database; run POST /api/v1/cluster/resync/{database} on it instead, transferring leadership first if that node is the leader.

A node in either state answers 503 on /api/v1/ready, so Kubernetes takes it out of the Service. (Since v26.10.1 this no longer depends on arcadedb.server.readinessRequiresHA: a node whose replication has stopped is never ready, whatever that setting says.)

Cluster membership and leadership can be changed at runtime through dedicated REST endpoints. All endpoints require authentication as the root user.

Example: add a new peer at runtime.

Adding a peer with a human-readable name:

Adding a peer as a non-voting witness, which will never become leader:

POST /api/v1/cluster/verify/{database} reports one status per peer and rolls them up into a single result.overallStatus:

ALL_CONSISTENT therefore means what it says: it is not reachable while any peer is unverified. An unverified peer is deliberately not rolled up as an observed divergence, which would send an operator hunting for something nobody saw.

ArcadeDB Studio exposes a Cluster tab (visible when HA is enabled) that displays the current leader, peer roles, Raft term and commit index, per-follower replication lag, and provides buttons for leadership transfer, peer add/remove, and database verification. See Studio.

When arcadedb.ssl.enabled=true, the server binds an HTTPS listener on arcadedb.server.httpsIncomingPort (default 2490) in addition to the plain HTTP listener on arcadedb.server.httpIncomingPort (default 2480). The two listeners use different ports.

The side channels covered here are the snapshot download, the cross-node verify, the follower-to-leader command forward, the peer capability and auth-session probes, the stalled-replica resync, the bootstrap-state probe, the security-document seed and the remote shutdown command. Every one of them carries the cluster token, so on a cluster with arcadedb.ssl.enabled they should all travel encrypted. To send them encrypted, declare each peer’s HTTPS port in arcadedb.ha.serverList using the https field (object form) or the fifth positional field:

The peer certificate is validated against the configured trust store (arcadedb.ssl.trustStore), and its name is verified against the name the peer was dialled by.

If the https port is omitted on a homogeneous cluster (every node listens on the same HTTPS port, e.g. a Kubernetes StatefulSet), each peer’s HTTPS endpoint is derived automatically from the peer host plus this node’s local HTTPS port, so no extra configuration is needed. Only when no HTTPS endpoint can be resolved does the transfer fall back to plain HTTP. The plain listener is the one that is always bound, so falling back keeps a cluster that never declared its https ports able to bootstrap, resync and be administered — but it does put the cluster token on the wire in clear text, so it is announced once per JVM:

One such line is logged per JVM for the whole family of side channels, naming whichever happened to fall back first; the remedy is the same for all of them. Seeing it on a cluster you believe is encrypted end to end means at least one peer has no resolvable HTTPS endpoint — check that the https field is declared for every entry, and see Peer-to-Peer Endpoints for the case where it is declared but shared between two nodes.

The previous section covers the HTTP side channels. This one covers the Raft gRPC port (arcadedb.ha.raftPort, default 2434), which carries AppendEntries, RequestVote, InstallSnapshot and the Ratis admin/client calls.

The two are independent axes. arcadedb.ssl.* does nothing for the gRPC port, and arcadedb.ha.tls.* does nothing for the HTTP listeners; a hardened cluster wants both. Nor does gRPC TLS replace the X-ArcadeDB-Cluster-Token check described in Security — that check still authenticates the HTTP forwarding and snapshot-download traffic.

TLS on the Raft port is off by default, so a development or test cluster still starts with no configuration at all. Turn it on with four settings:

One setting covers all three gRPC endpoints (admin, client, and server-to-server), so there is nothing further to switch on per message type. Mutual authentication is on by default; see Server-only TLS to relax it.

When arcadedb.ha.tls.enabled=true, all three file settings must name a readable PEM file or the node refuses to start, with a ConfigurationException naming the offending setting:

The paths are validated at startup rather than left to the TLS layer, so a typo or an unmounted Secret volume fails once and clearly, instead of surfacing later as an opaque handshake error on every peer connection.

The private key must be PKCS#8 — the PEM whose armor labels it simply PRIVATE KEY. A PKCS#1 key, which the armor labels RSA PRIVATE KEY instead and which several toolchains emit by default, is not accepted. Convert one with openssl pkcs8 -topk8 -nocrypt.

For Kubernetes deployments using the official Helm chart — including the StatefulSet + headless Service pattern, auto-join on scale-up, and the preStop auto-leave hook — see High Availability on Kubernetes in the Kubernetes guide.

The following settings control HA behavior. A complete list of all HA parameters is available in Server Settings.