Graph Batch

The GraphBatch is a high-performance bulk graph loader designed for importing large volumes of edges into an ArcadeDB graph. It achieves 8-12x speedup over the standard edge creation API by buffering edges in memory, sorting them by vertex for sequential I/O, parallel edge connection, and deferring incoming edge connection to a single optimized pass.

Quick Start

Database db = new DatabaseFactory("mydb").create();
db.transaction(() -> {
  db.getSchema().createVertexType("Person");
  db.getSchema().createEdgeType("KNOWS");
});

try (GraphBatch batch = db.batch()
    .withExpectedEdgeCount(1_000_000)
    .build()) {

  // Phase 1: Create vertices in bulk
  RID[] vertices = batch.createVertices("Person", 100_000);

  // Phase 2: Buffer edges (auto-flushes when batch is full)
  for (int i = 0; i < edgeCount; i++)
    batch.newEdge(vertices[src[i]], "KNOWS", vertices[dst[i]]);

} // close() flushes remaining edges + connects all incoming edges

All edges (both outgoing and incoming) are fully connected when close() returns.

How It Works

The importer operates in phases for maximum throughput:

  1. Buffer: Edges are stored in flat primitive arrays (minimal GC pressure)

  2. Sort: On flush, outgoing edges are sorted by source vertex, converting random I/O into sequential page access

  3. Connect OUT: Outgoing edges are connected to source vertices in sorted order, using vectorized segment writes

  4. Accumulate IN: Incoming edges are deferred to an in-memory buffer across flushes

  5. Connect IN at close(): All deferred incoming edges are sorted by destination vertex and connected in a single pass

  6. Batch vertex update: All vertex head-chunk pointers are updated in one final sorted pass

Edges with Properties

The importer supports edges with properties. Pass them as key-value pairs:

try (GraphBatch batch = db.batch()
    .withExpectedEdgeCount(500_000)
    .withLightEdges(false)  // edges have properties
    .build()) {

  RID[] vertices = batch.createVertices("Person", 50_000);

  for (int i = 0; i < edgeCount; i++)
    batch.newEdge(vertices[src[i]], "KNOWS", vertices[dst[i]],
        "weight", weights[i], "timestamp", timestamps[i]);
}

Property edges use a bulk record creation path that bypasses per-record overhead, achieving nearly the same throughput as light edges. Edges of a type with declared properties still get what Vertex.newEdge() gives them: values converted to the declared type, defaults applied and constraints (mandatory, min/max, regexp…​) validated. A violation is raised when the buffered edges are flushed, not at the newEdge() call. Properties can be passed as name/value pairs or as a single Map. With withLightEdges(true), an edge without properties is still stored as a regular edge if its type declares a property with a default or a mandatory property.

(Since v26.10.2) An edge type with an EXTERNAL property, declared on the type or inherited from a super type, does not use the bulk record creation path: its values are stored in the paired external bucket, so those edges are saved one at a time and the import is slower than for other edge types. Before 26.10.2 the bulk path kept those values inside the edge record instead of the external bucket.

Builder Options

Method Default Description

withExpectedEdgeCount(int)

 — 

Auto-tunes the batch size to edgeCount (clamped to 100K-5M). A single flush is optimal for most workloads.

withBatchSize(int)

100,000

Maximum edges buffered before auto-flush. Overrides auto-tuning if both are set.

withLightEdges(boolean)

false

If true, edges without properties are created as light edges (no record stored, only connectivity pointers). Deprecated: the edge type must declare LIGHTWEIGHT, otherwise an IllegalArgumentException is raised.

withBidirectional(boolean)

true

If true, incoming edges are connected for the edge types declared bidirectional; an edge type declared UNIDIRECTIONAL never gets them, whatever this flag says. Set to false to load only unidirectional edge types: newEdge() then refuses an edge of a bidirectional type with an IllegalArgumentException (since 26.10.1; before, the batch silently skipped the incoming side, and a query walking those edges from their target found nothing).

withWAL(boolean)

false

Write-Ahead Logging during import. Disable for maximum speed; enable if crash recovery is needed.

withCommitEvery(int)

0 (WAL off) / 50,000 (WAL on)

Edges per transaction commit within a flush. Set to 0 to commit once per flush. Auto-set to 0 when WAL is disabled.

withEdgeListInitialSize(int)

2048

Initial size in bytes for edge segments. Larger values reduce segment splits for high-degree vertices.

withPreAllocateEdgeChunks(boolean)

true

Pre-allocate empty edge segments at vertex creation time, eliminating lazy allocation cost. Each segment takes withEdgeListInitialSize bytes on disk even if the vertex never gets an edge, so turn it off when loading many vertices whose edges come later: the edges then get segments of exactly the size they need.

withMaxVertices(long)

0 (no bound)

Upper bound on the distinct vertices the batch’s edges can touch. The batch keeps a small entry per touched vertex until close(); the bound stops that table from growing past what it needs. Nothing is allocated up front (since 26.10.2).

withParallelFlush(boolean)

true

Parallel edge connection partitioned by bucket. Uses the database async executor to connect edges across buckets concurrently.

Auto-Tuning Batch Size

The withExpectedEdgeCount() method automatically selects the optimal batch size based on benchmarks. A single flush (batch size = edge count) delivers the best throughput, so the batch size is set to edgeCount, clamped between 100K and 5M:

Expected Edges Auto-tuned Batch Approx. RAM

< 100K

100K (minimum)

~200MB

100K - 5M

= edge count (single flush)

~300MB-1GB

> 5M

5M (maximum)

~1GB

You can always override with withBatchSize() if you know your workload characteristics.

Key Methods

Method Description

createVertices(typeName, count)

Creates vertices in bulk within a single transaction. Returns an array of RIDs for use with newEdge().

newEdge(srcRID, edgeType, dstRID, properties…​)

Buffers an edge for batch processing. Automatically triggers a flush when the batch size is reached.

newEdgeWithHints(srcRID, edgeType, dstRID, hints, properties…​)

Same as newEdge(), for a caller that knows an endpoint has no edges yet in that direction: GraphBatch.SOURCE_HAS_NO_OUT_EDGES and/or GraphBatch.DESTINATION_HAS_NO_IN_EDGES, or 0. The batch then starts the vertex’s edge list without reading the vertex, a saving of one random disk read per distinct vertex and flush when the graph is larger than the memory. Pass the hint with the first edge of the vertex you hand to a batch, never later: a wrong hint is not detected in production and leaves the edges the vertex already had unreachable from it (with assertions enabled, -ea, the batch checks every hint and fails instead). GraphImporter uses it for the vertices it creates. Since v26.11.1

getVertexHeadReads()

How many vertices the flushes read to find the edge list they append to: one per distinct vertex and flush that the batch neither created nor was given a hint for. Since v26.11.1

getSegmentHeadReads()

How many edge-list segments the flushes read back to append to. A segment the batch wrote itself is never read again: the batch sizes it to its edges, so the next edges of that vertex go to a new segment linked to it. Only segments the batch found on disk are counted. Since v26.11.1

flush()

Manually flushes all buffered edges. Outgoing edges are sorted and connected; incoming edges are deferred.

close()

Flushes remaining edges, connects all deferred incoming edges in a single sorted pass, and batch-updates vertex head-chunk pointers.

Transactions

The batch manages its own transactions: every flush is several durable steps, committed by the batch itself. For this reason createVertices(), flush(), close() and any newEdge() call that fills the buffer refuse to run inside a transaction you opened with begin() or transaction(). They throw an IllegalStateException and leave your transaction untouched.

Commit or roll back your own transaction first. If close() is refused, the batch stays open with its pending work, and calling close() again outside the transaction writes it.

The batch’s relaxed WAL settings apply only to the batch’s own transactions, never to the ones you run between batch calls.

Recovering an Interrupted Batch

Each flush commits the edge records it created, but the vertices' head-chunk pointers are only written at close(). If the process dies before close() (kill -9, an OOM kill, a power loss), the edges committed by the earlier flushes are on disk but not reachable by traversal: CHECK DATABASE reports them as ORPHAN RECORD and counts them under unreachableEdgeRecords.

Since v26.11.1 the supported recovery is CHECK DATABASE FIX. It relinks each of those edge records into the outgoing list of its @out vertex and, for a bidirectional edge type, into the incoming list of its @in vertex, so the committed edges become traversable without reloading them (relinkedOrphanEdgeRecords in the result). Edges still buffered in memory when the process died were never written and must be loaded again. Use CHECK DATABASE FIX DELETE ORPHANS instead to delete the committed edges and reload them all. Before v26.11.1 a plain FIX left those records untouched, and deleting them was the only repair.

Performance

Benchmark results on a single machine (500K edges, 50K vertices):

Method Time (ms) Edges/sec Speedup

Standard API (tx batches of 1000)

11,933

42K

1.0x

GraphBatch (light edges)

1,028

486K

11.6x

Standard API + properties (int + long)

13,385

37K

1.0x

GraphBatch + properties

1,584

316K

8.5x

Importing into an Existing Graph

The importer can also connect edges to vertices that already exist in the database (not just vertices created by createVertices()). Simply pass existing vertex RIDs to newEdge():

try (GraphBatch batch = db.batch()
    .withExpectedEdgeCount(100_000)
    .build()) {

  // Use existing vertex RIDs from queries or lookups
  for (int i = 0; i < edgeCount; i++)
    batch.newEdge(existingVertexRIDs[src[i]], "FOLLOWS", existingVertexRIDs[dst[i]]);
}

HTTP Batch Endpoint

GraphBatch is also available over HTTP for non-Java clients. The POST /api/v1/batch endpoint accepts JSONL or CSV input and uses GraphBatch under the hood, with all builder options exposed as query parameters. See HTTP Batch Import for full details and examples in curl, Python, and JavaScript.

Relationship with GraphImporter

The GraphImporter is a declarative, JSON-driven graph importer built on top of GraphBatch. It adds file parsing (XML, CSV, JSONL), schema auto-creation, foreign key resolution, and post-import commands.

Use GraphBatch directly when you need full programmatic control over vertex and edge creation. Use GraphImporter when you want to import graph data from files using a JSON configuration.