Cloud Observability

Two always-on timers record request/query latency with a percentile histogram (server-side p50/p95/p99). Query text is never used as a tag, keeping cardinality bounded.

EngineMetricsBinder bridges the internal engine Profiler into Micrometer gauges (engine-wide aggregates, refreshed from a one-second-memoized snapshot):

The two async.* metrics describe the asynchronous executor and are worth reading together with the throughput of your ingestion.

async.boundary.commits counts how often a change to setTransactionUseWAL() / setTransactionSync() forced a worker to close its open batch transaction early. Occasional increments are normal - a bulk load flips those flags at its start and end. A rate that stays high while asynchronous throughput drops means two callers are flipping them against each other on the same database, so batching is degrading towards one commit per task.

async.workers.retiring is a gauge, not a counter: it goes back to zero as the retired workers finish. Zero is the normal reading, and a brief spike during a setParallelLevel() call is expected. A reading that stays above zero means a retired worker is stuck inside user code and will never finish draining - a resize deliberately never interrupts one - so the thread stays alive until the database is closed.

graph.light.edge.census.fallbacks is a counter of how often counting the lightweight edges of one vertex’s edge list failed, so that every lightweight edge read off that list resolves its occurrence by walking the list itself - the quadratic cost the census exists to avoid, which on a super-node is the difference between milliseconds and minutes. Zero is the normal reading. The first failure of each list of a database is logged at WARNING (at most 64 distinct lists per database; the rest at FINE), so the log shows which list could not be read and this counter shows how often it keeps happening. (Since v26.11.1: one WARNING per JVM and no counter.)

bucket.recount.publishes.refused (Prometheus arcadedb_engine_bucket_recount_publishes_refused_total; in the Profiler JSON, recountPublishesRefused: {count: N}) counts the times a bucket recomputed its record count but could not cache the result, because a replicated apply wrote to the bucket while it was being scanned. One or two increments are the normal cost of an HA follower catching up. A rate that never returns to zero means a follower on which every count(*) is a full scan. (Since v26.10.2)

To push metrics to an OpenTelemetry Collector (in addition to the unchanged /prometheus scrape), enable the OTLP registry:

The metrics exporter speaks OTLP/HTTP (protobuf), not gRPC, so the endpoint is the collector’s HTTP receiver: port 4318, path /v1/metrics (this is also the default). Port 4317 is the gRPC receiver and does not work for metrics; the server logs a warning if it is configured. A URL without a path gets /v1/metrics appended. Tracing is different: it uses OTLP/gRPC on port 4317 (see below).

The metrics and tracing plugins report the OpenTelemetry service.name resource attribute. The standard OTEL_SERVICE_NAME environment variable is honored and takes precedence over OTEL_RESOURCE_ATTRIBUTES and over the arcadedb.serverMetrics.serviceName setting (default arcadedb), so in a container set it with -e OTEL_SERVICE_NAME=my-arcadedb.

This registers an OTLP meter registry alongside Prometheus; the scrape path is unaffected whether or not OTLP is enabled. The OTLP exporter lives in the optional metrics module bundled with the standard distributions.

Every HTTP request is tagged with a requestId: ArcadeDB reuses the inbound X-Request-Id header when present (sanitized, capped at 128 characters) or generates a UUID, and echoes it on the response. The db field is taken from the request’s database path parameter. When the tracing plugin is active, traceId/spanId are added too — but requestId works for correlation even with tracing disabled.

For operators who keep text logs, arcadedb.server.logIncludeTrace=true appends ` [traceId=…​]` to each line while a trace is active (default off, preserving current output):

On a High Availability cluster you can make readiness reflect Raft membership so traffic is withheld until a node has joined the group:

When this flag is true and HA is active, /api/v1/ready returns 503 (Node has not yet joined the Raft group) until a leader has been elected. Once a leader is known, a second gate checks that the node is a member of the current Raft configuration and, for a follower, that it has caught up to within arcadedb.server.readinessHAMaxLag log entries (default 100) of the commit index; until then it keeps returning 503 (Node is not yet in the Raft configuration or has not caught up). Only then does it return 204. This prevents a (re)joined follower with a wiped or lagging log from being marked ready mid-catch-up, which would otherwise let a rolling restart drop the write quorum. The default (false for readinessRequiresHA) preserves current readiness behavior.

With readinessRequiresHA on, a third reason can hold a node at 503 (since v26.10.1): its users, groups and API tokens may not be the cluster’s. These live in server-users.jsonl, server-groups.json and server-api-tokens.json, outside the databases, so no snapshot carries them (see Cluster security documents). Two kinds of node are held:

Until then the node enforces its own copies, so a user dropped, a group narrowed or a token revoked while it was away would still work there. /api/v1/ready answers 503 with Cluster security documents have not reached this node yet: <documents>. A statically configured member that simply restarts is not held, and neither is a node while it leads, because nobody else can confirm its copies.

The hold is bounded by arcadedb.ha.securityConvergenceReadinessTimeout (default 30000 ms; 0 turns it off). When the window expires the node reports ready anyway, so a scale-up or rolling restart cannot stall behind a seed that never arrives, and logs a single SEVERE line naming the documents:

When you see it, re-run connect cluster or re-POST the node to /api/v1/cluster/peer to send the seed again, or reissue the security change. The gRPC Ready RPC uses the same check.

The replication-log-writer check described in the table above is deliberately not behind this flag. This flag gates a node that is merely behind and which you may reasonably choose to serve from; a node whose log writer has failed cannot catch up at all, so it is withheld from traffic whether or not the flag is set. See Kubernetes deployment for the full StatefulSet example.