Bulk loads and commit size

insert_many, graph_batch and GraphBatch.create_vertices load data in transactions. The commit size is the number of rows in one transaction. It decides how much work the engine holds in memory before it commits, and how many rows stay in the database when a load fails. Data that comes as a file, a DataFrame or an Arrow table has its own pages: Importing and exporting data and DataFrames, Arrow and columns.

Defaults

Call arcadedb (native) arcadedb-jvm

db.insert_many(type_name, rows)

commit_every=1000

commit_every=10000

db.graph_batch()

commit_every=1000

commit_every not set: one commit for each flush

batch.create_vertices(type_name, rows)

1000 rows for each call into the engine, one transaction for each call (the commit_every of the batch)

100000 rows for each call into the engine

db.insert_columns(type_name, columns)

commit_every=10000

commit_every=10000

db.insert_dataframe(type_name, df) and db.insert_arrow(type_name, table)

batch_size=10000, one transaction for each batch

batch_size=10000, one transaction for each batch

The JVM defaults of insert_many and graph_batch are the same as before. They stay until someone measures them on the JVM build. A vertex type loaded with insert_dataframe or insert_arrow now commits every 10,000 rows on both packages: see Vertex types in insert_dataframe and insert_arrow.

An explicit value always wins. commit_every=0 gives one transaction for insert_many and one commit for each flush for graph_batch, on both packages. In a transaction that you open yourself, insert_many commits nothing. graph_batch refuses to run in one: see GraphBatch inside a transaction. The durability that a load gets is on Durability settings, and the async executor on The async executor.

Why the native defaults are smaller

The in-flight transaction is the data that each young garbage collection must copy. A smaller transaction copies less. A laptop A/B on the columnar native wheel measured this with ab.py --noise, ABBA order, 12 clean pairs for each figure, and a clean A/A control (95 % interval of the median pair ratio, new default against old default):

Load Wall time Data copied by the young GC Peak RSS

insert_many, 1,000,000 mixed documents, commit_every=1000

0.822 to 0.831

0.55x

0.955x

GraphBatch, 200,000 vertices and 500,000 edges, parallel_flush, one client thread, commit_every=1000

0.853 to 0.864

0.63x

0.718x

commit_every=5000 for insert_many also passed (0.903 to 0.914 of the wall time). 1000 had the lower median, so it is the default. The runs used an isolate heap of 80 % of RAM. The default is now 50 %, and smaller transactions lower the heap that a load needs.

What changes for you

Rows that are already committed stay committed when a later chunk fails. Only the transaction that was open is rolled back. With a smaller commit size, a failed load keeps its rows up to the last commit, and that commit is now at most 1000 rows behind the failure instead of 10,000. If you need all-or-nothing, open your own transaction, or pass commit_every=0.

Because the two packages have different defaults, the same program can keep different rows after the same failure: each keeps its rows up to its own last commit. If the outcome of a failed load must not depend on the package, pass the same commit_every on both, or commit_every=0.

Values the bulk path carries

On both packages, insert_many keeps rows with datetime, date, Decimal, bytes, numpy scalar, or pandas NA and NaT values on the bulk path. On the arcadedb (native) package, a row with a value the bulk text cannot carry (NaN or infinity, an integer beyond 64 bits, a date-time or Decimal inside a list or dict, or a datetime.time) goes in with its own statement, in the order given.

insert_many and create_vertices check their rows while they write the JSON text, in one pass over the rows, not in a loop before it. The same rows fail the check as before, and they take the same fallback path as before. The check raises no error to your code.

Columnar loads

insert_columns takes one column for each property. insert_arrow and insert_dataframe read an Arrow table or a pandas or Polars DataFrame as columns, in batches of batch_size rows.

On the arcadedb (native) package, a batch whose columns the library’s buffers can carry crosses in one call, with no Python dict per row and no JSON text. These columns stay columnar: integers, floats, bools, text, date-times (datetime, pandas.Timestamp, and Arrow timestamps of any unit), dates, Decimal, and bytes or bytearray. The stored values are the ones the row path stores.

A column the buffers cannot carry makes the batch take the row path, which stores the same values through insert_many. These are a column that mixes types, a nested value, an integer beyond 64 bits, a string with a lone surrogate, a non-finite Decimal, and, in insert_columns, a numpy datetime64 array or a list that holds pandas NaT or NA. A DataFrame goes through Arrow first, where a NaT or NA is a null, so it stays columnar. ARCADEDB_NATIVE_BULK=0 in the environment turns the columnar path off.

On the arcadedb-jvm package, a column of only str and None crosses as one UTF-8 buffer with its row offsets, and a mask when a row is null. A column that mixes types, is empty, or holds more than 2 GiB of text crosses value by value. A column with a lone surrogate also crosses value by value, and is refused with the error it always raised, before anything is written.

On the arcadedb-jvm package, from 26.11.1.post3, a timestamp or date column of insert_arrow and insert_dataframe crosses as one long[] of epoch ticks for each batch, with one boolean[] that marks the null rows when the batch has a null. Java builds each LocalDateTime or LocalDate when it fills the document of that row. Before 26.11.1.post3 the bindings made one Java object for each cell, with a JPype call of its own. The stored values are the same: a timestamp keeps the precision of its Arrow unit (a nanosecond value is not cut to microseconds), a timestamp without a time zone is stored as the wall clock it shows, a timestamp with a time zone is stored as UTC, and a date64 value is divided into days toward negative infinity. A list of Python datetime or date values that you pass to insert_columns yourself still converts value by value. The arcadedb (native) package does not change.

The table is one laptop study of the arcadedb-jvm package: the time for each row of the 26.11.1.post3 wheel against the 26.11.1.post2 wheel, with 3 calls of 100,000 rows that carry a DATETIME and a DATE property next to four narrow ones. The ratio is the median of the pair ratios (below 1 means less time) with its 95% interval. The setup was Python 3.12, one fresh process for each run, the two builds alternated in both orders (ABBA), the first round dropped, 20 clean pairs for each figure, and an A/A control (the post2 wheel twice) whose median ratio was between 0.97 and 1.03 in every row. The figures come from one Linux x86_64 laptop with two cores for the process. The post3 wheels were the ones that CI built from the tagged commit. The wheels on PyPI were checked for the same engine as 26.11.1 and were not timed again.

What was timed Ratio, post3 over post2 (95% interval) Clean pairs

insert_arrow, an Arrow table with timestamp[ms] and date32 columns

0.295 [0.283, 0.311]

20

insert_dataframe, a pandas frame with datetime64[ms] and datetime.date columns

0.283 [0.278, 0.290]

20

insert_columns with Python lists of datetime and date values, which this change does not touch, gave 1.009 [0.983, 1.050] in the same study. The interval contains 1, so there is no claim in either direction. The study did not time memory. It read the peak resident size of the process for information only, and it was higher with the new path: 1.066 [1.014, 1.194] for insert_arrow and 1.202 [1.059, 1.299] for insert_dataframe (median of the pair ratios, 95% interval). The reason has not been examined, so the page makes no claim about memory.

Parallel and async inserts

insert_many(parallel=True) and insert_columns(parallel=True) hand the records to the async executor’s writers and wait for the writers before they return. insert_arrow and insert_dataframe do the same for a document type, one batch at a time. A vertex type refuses parallel=True there with a ValueError.

A parallel call opens no transaction, so commit_every does not apply. Each writer commits every arcadedb.asyncTxBatchSize records (10,240 by default, and async_executor().set_commit_every() changes it). On the native package, insert_columns(parallel=True) sends the columns to the library in one call, and the library hands each record to the writers. A column set the buffers cannot carry goes as one dict per row through insert_many(parallel=True).

What a failure leaves behind:

  • A record that a writer cannot store, for example a duplicate key, abandons the whole batch that writer had open, up to 10,240 records. The valid records of that batch are not stored either.

  • The call waits for all writers, then raises ArcadeDBError. The message gives the number of failed records and the first failure.

  • Nothing is rolled back. Batches of other writers, and earlier batches of the same call, may have been stored. After such an error, count the stored rows with a query. Do not assume all or none.

  • On the native package, an interrupt (KeyboardInterrupt) during a parallel call waits for the writers before it propagates, so no writer still runs when the exception reaches you.

  • In a transaction you opened, insert_arrow and insert_dataframe with parallel=True do not join it. The writers commit on their own, and a rollback of your transaction does not remove those rows.

For one record at a time, db.async_executor().create_record(document, callback=…​, error_callback=…​) queues the record (on the native package it hands the records over in batches, see Native: create_record hands records over in batches). The writers call error_callback with the exception when they reject it (a duplicate key under a UNIQUE index, or a batch abandoned at a failed commit). Without an error_callback, the failure reaches only the executor-wide on_error handler, if you set one. Call wait_completion() before you read the records back.

Vertex types in insert_dataframe and insert_arrow

On both packages, insert_dataframe and insert_arrow on a vertex type load through the same columnar insert as a document type. They no longer go through GraphBatch.create_vertices. Use graph_batch() when edges follow or when you need the RIDs of the vertices.

  • The load runs with the write-ahead log setting of the database. graph_batch() has its own use_wal argument.

  • Each batch of batch_size rows (10,000 by default) is one transaction. A failed batch is rolled back, earlier batches stay committed, and the error message says how many rows that is.

  • A batch whose save or commit raises a NeedRetryException (an MVCC conflict with another writer, a lock timeout) is rolled back and written again from its first row. It is tried up to 10 more times, after a wait that starts at 1 second and doubles up to 10 seconds. After the last try the error is raised unchanged. The columnar insert does this for every type it loads, not only for vertices, when no transaction of yours is open. A call with parallel=True does not retry, and on the native package neither does a batch that takes the row path.

  • A vertex load inside a transaction you opened is refused with ArcadeDBError before anything is written, because the load commits its own batches. A document load joins your transaction.

  • On arcadedb-jvm, a float32 vector column of such a load crosses to Java as one matrix for each batch (see Vectors). This is faster than the route of 26.11.1, and the peak memory of the load is higher than on that route for the same data. The measurement is in issue 348, which stays open for the memory side.

Change the commit size

db.insert_many("Doc", rows, commit_every=10_000)  # the old native default
db.insert_many("Doc", rows, commit_every=0)       # one transaction for all rows

with db.graph_batch(commit_every=0) as batch:     # one commit for each flush, as before
    ...

with db.graph_batch(commit_every=5000) as batch:  # also sets the create_vertices chunk
    ...

The cyclic garbage collector during insert_many (arcadedb-jvm)

On arcadedb-jvm, insert_many pauses Python’s cyclic garbage collector for the length of a call when rows is a list or a tuple, and puts it back as it was when the call ends, also when the call raises. The call allocates many objects that the collector tracks. Without the pause, the collector goes through your whole row list again and again during the load, which makes the load slower. The call itself creates no reference cycles from a list or a tuple of plain rows. A laptop A/B on the arcadedb-jvm wheel measured the pause with ab.py --noise (500,000 dict rows in slices of 100,000, ABBA order, 12 clean pairs for each figure, clean A/A control). Against the default, the call took 0.860 of the time [0.819, 0.903] (95 % interval of the median pair ratio), the whole load 0.876 [0.861, 0.923], and the peak RSS 0.976 [0.924, 1.057], which is no detectable change.

A generator or any other lazy input is not paused. Your own code runs between the chunks that the generator yields, and that code can create reference cycles that only the collector frees. If your generator builds plain dicts, you can call gc.disable() around the load yourself.

gc.disable() is process-wide. While a call with a list runs, other Python threads do not collect cyclic garbage automatically. When calls from several threads overlap, the pause does not last for the whole of each call: the collector comes back when the call that paused it ends. The collector is never left disabled by a call, and a collector that you disabled yourself stays disabled.

The arcadedb (native) package does not change the collector. In our measurements, no collection ran inside a native insert_many call, so there is nothing to pause.

Property shapes in a graph batch (arcadedb-jvm)

On the arcadedb-jvm package, GraphBatch.create_vertices and GraphBatch.new_edges load these property values through the bulk path (the same tagged JSON text as insert_many): a list, a tuple, a dict, a datetime, a date, a Decimal and bytes (also bytearray). A value is stored as the one-by-one path stores it. A list or tuple of only int and float keeps the older path, which crosses it as one double[].

A value that the text cannot carry sends work to the older path on its own. In create_vertices, a scalar that the text would change (a NaN, an integer beyond 64 bits, a lone surrogate, a subnormal float) sends the whole call to the older path, wherever in the input it sits. So does a property name that is not a str. A value inside a list or a dict that the text cannot carry (a datetime inside a list, a NaN in a list, a dict key that is not a str) sends only the chunk of at most 100,000 rows that holds it. new_edges falls back for the whole call. A value of another type (a set, a numpy or pandas value, a subclass of the types above) also takes the older path.

Nothing is lost on a fallback and the values are the same, but the stored type of a property that the type does not declare can differ: the older path stores a Python int as a Long, and the text path as an Integer. That is why a bad scalar sends the whole call to the older path. The stored type of a property then does not depend on which chunk held the bad value.

A property whose value in every row is a float32 numpy array or a Java float[] of one length crosses as one binary matrix for each property and chunk, and is stored as a float[]. A declared ARRAY_OF_FLOATS property takes the same path for lists and float64 arrays of numbers too. See Vectors.

The table is one laptop study of the arcadedb-jvm package: per-row time of this build against the build before it, 20,000 vertices and 20,000 edges for each shape in chunks of 5,000, one property on every row. Each cell is the median of the pair ratios (below 1 means faster) with its 95 % interval, from 11 to 12 clean ABBA pairs for each figure of ab.py --noise, with a clean A/A control. The arcadedb (native) package does not change.

Property shape create_vertices new_edges

List of strings

0.403 (0.378 to 0.420)

0.409 (0.389 to 0.420)

Dict

0.265 (0.243 to 0.286)

0.278 (0.269 to 0.298)

Datetime

0.515 (0.503 to 0.550)

0.482 (0.469 to 0.518)

List of floats (keeps the old path)

1.024 (0.963 to 1.109)

1.010 (0.979 to 1.027)

Native: GraphBatch.create_vertices sends columns

On the arcadedb (native) package, from 26.11.1.post4, a chunk of rows that GraphBatch.create_vertices creates crosses into the library as typed columns by address, in the layout of insert_columns, and the library writes the new record ids into one int64 buffer. Before, every chunk was one JSON text and the ids came back as a JSON list of text.

The route is taken when the rows of the chunk have the same keys in the same order and plain values. A chunk that the columns cannot carry (other keys or key order, nested lists or maps, a mixed-type column, huge integers) keeps the JSON route of 26.11.1.post3, so nothing is lost. A chunk of fewer than 64 rows keeps the JSON route as well, because the columns have a fixed cost that a handful of rows does not repay. The environment variable ARCADEDB_NATIVE_GB_COLP_MIN_ROWS sets that limit, and 0 turns it off. ARCADEDB_NATIVE_GB_COLP=0 switches the whole route off. The engine method that creates the vertices, the transaction, the retries and the commit are the same as before. Float32 array vectors are stored after the other properties of the vertex, as they were.

Reads after a load by this route are not all faster. In the L2 graph study on the mini, with the same wheel and the route switched off for comparison, a point read of one vertex took 1.230 [1.196, 1.248] of the post3 time with the route on (24.056 to 29.494 microseconds per read) and 0.983 [0.976, 0.991] with it off. A read of the neighbours of one vertex took 0.774 [0.769, 0.791] with the route on and 0.995 [0.985, 0.998] with it off, and the load itself 0.830 [0.829, 0.834] with the route on. The cause of the read difference is not known yet. Measure the read path of your own workload after a column load, and set ARCADEDB_NATIVE_GB_COLP=0 to keep the JSON route if point reads matter more than the load.

The table is one study of the arcadedb (native) package on a laptop and on a second Linux machine, the mini: the time for each operation of the post4 wheel against the 26.11.1.post3 wheel. The ratio is the median of the pair ratios (below 1 means less time) with its 95% interval. A ratio is quoted only when its interval is narrow enough to read. Setup: Python 3.12, one fresh process for each run, two cores for the process, the two builds alternated in both orders (ABBA), the first round dropped, and an A/A control (the post3 wheel twice) whose median ratio was between 0.997 and 1.006 on the laptop and between 0.981 and 1.000 on the mini. The post4 wheels were built by CI from commit e83947e, the commit of the tag, and the post3 wheels were the ones that CI built from the post3 commit. The files on PyPI were checked for the same engine as 26.11.1 (see The engine of 26.11.1.post4) and were not timed again.

What was timed Ratio on the laptop, post4 over post3 (95% interval) Ratio on the mini (95% interval) Clean pairs (laptop, mini)

GraphBatch.create_vertices of rows with the vector as a Python list, 100,000 rows per call

0.134 [0.133, 0.135]

0.118 [0.118, 0.118]

16, 17

GraphBatch.create_vertices of rows with an int and a str property, 100,000 rows per call

0.778 [0.760, 0.786]

0.735 [0.734, 0.736]

16, 17

GraphBatch.create_vertices of rows with an int and a str property, 100 rows per call

0.944 [0.911, 0.960]

0.894 [0.890, 0.901]

16, 17

GraphBatch.create_vertices of rows with an int and a str property, 1 row per call

no claim (interval too wide)

0.885 [0.872, 0.896]

16, 17

GraphBatch.create_vertices of rows with a float32 vector as a numpy array, 100,000 rows per call

0.990 [0.983, 0.995]

0.953 [0.952, 0.964]

16, 17

Memory did not improve. Across the 172 figures of the whole post4 study, the median of the peak resident size, post4 over post3, was 1.004, so there is no systematic change, and this page makes no memory claim for this route.

Time-series samples with append_samples

db.async_executor().append_samples(type_name, timestamps, *columns) loads a columnar batch into a TIMESERIES type: the timestamps, then one column for each tag and field in the order of the type’s declaration. Call wait_completion() before you read the samples back. How to declare the type is in the main ArcadeDB docs: see Time series. The call exists on both packages. The rest of this section describes the fast path of the arcadedb (native) package. On arcadedb-jvm, a NumPy array for the timestamps or for a numeric column crosses as one buffer copy, and any other sequence converts value by value.

On the arcadedb (native) package, a NumPy column crosses to the engine by its address, with no JSON text, when it is one of these:

  • The timestamps: a one-dimensional, non-empty integer or bool array (int64 is the plain case), or a datetime64[ns] array without NaT.

  • A numeric column: a one-dimensional, non-empty array of any integer or bool kind, or of a float kind up to float64 (float32 widens to float64 without a change of value). A uint64 array, as timestamps or as a column, crosses only if its largest value fits in int64.

  • Each column must have as many values as there are timestamps.

Everything else keeps the text path, which stores the same values and only takes longer: a list or a tuple, a subclass of ndarray, a two-dimensional or empty array, a float array as the timestamps, a datetime64 array of another unit or with NaT, a long double array, and a string or tag column. A call can mix both: the numeric columns cross by address and the tag column goes as text. To get the fast path, build the timestamps and the numeric columns as NumPy arrays with these dtypes and pass the arrays as they are, without .tolist(). An array that is not already contiguous int64 or float64 is converted first, which copies it in NumPy.

The measurement is a laptop A/B on the arcadedb (native) wheel, one fresh process per run, Python 3.12 and NumPy 2.5.4. The workload loaded 2,000,000 points of 100 hosts in chunks of 100,000: an int64 timestamp array, one list of host names (the tag), and three float64 arrays, with one wait_completion() at the end. The figure is the whole ingest time in seconds, including that wait, as the ratio of the build with this change to the build without it (below 1 means faster): 0.370 (95% interval 0.362 to 0.390, median of 11 clean paired runs). A document insert in the same runs, which does not use this path, took 1.020 (95% interval 1.016 to 1.022) of the time on the build with this change. The figure holds for this shape: your gain depends on how many numeric columns a call carries and how large the chunks are.

ARCADEDB_NATIVE_TS_BIN=0 in the environment turns the address path off and returns to the text path. A native library that is older than this change falls back to the text path by itself.

Graph batches next to deletes (fixed in ArcadeDB 26.11.1)

Fixed in ArcadeDB 26.11.1. Before 26.11.1, a graph batch did not reuse the space of deleted edges (ArcadeData/arcadedb#10746). The engine wrote the edge records of each graph_batch() flush onto new pages and never looked at the free space that deletes left on older pages, so the edge bucket gained one 64 KiB page per flush while the live data stayed flat. From 26.11.1 (ArcadeData/arcadedb#10836) each edge goes where an ordinary insert would put it while an existing page has room, and only the rest of a flush goes onto new pages, so a large import stays as fast as before. The 26.11.1 packages ship this fix.

On engine 26.10.2, a one-off bulk load is not affected. The workloads that are affected call graph_batch() again and again on a type that also sees deletes, for example a sliding window that adds a small batch and deletes the oldest one.

The measurement below ran on engine 26.10.2: a sliding window of 60 rounds for 240 seconds on each package. Each round wrote 30 vertices and about 90 edges and deleted the oldest round with DELETE FROM Node, so the live edge count stayed flat. The delete order does not matter: deleting the edges before the vertices gave the same growth.

How the edges were written Package Live edges at the end Edge bucket at the end (MiB)

graph_batch(), one block for each round

arcadedb (native)

5,261

329.00

graph_batch(), one block for each round

arcadedb-jvm

5,257

456.62

SQL CREATE VERTEX and CREATE EDGE, one transaction for each round

arcadedb (native)

5,252

0.25

SQL CREATE VERTEX and CREATE EDGE, one transaction for each round

arcadedb-jvm

5,269

0.25

An engine-level Java run of the same shape confirms the split: after 2,000 rounds the batch had 2,000 edge pages (125.0 MiB), and newEdge in an ordinary transaction had 4 (0.3 MiB). The cost is disk space: in the same runs a scan of the edge type and a two-hop query did not slow down while the bucket grew.

The engine-level Java run, repeated on the merge of the fix, kept the batch’s edge bucket at 4 pages after 2,000 rounds, the same as newEdge in ordinary transactions, against 2,000 pages on its parent (verification on #10746).

From 26.11.1 on, graph_batch() is fine for a workload that deletes and inserts again. On an engine older than 26.11.1, write the edges with SQL (CREATE EDGE) or with Vertex.new_edge inside db.transaction() instead. Both stayed flat in the measurement (0.25 MiB in the table).

If a graph batch is interrupted

This section is about a graph_batch() load that stops before close() returns, for example when the process is killed (kill -9, the out-of-memory killer, power loss). An exception inside a with db.graph_batch() as batch: block still runs close(). The numbers below come from a kill probe on the arcadedb (native) package: kill -9 in the middle of the edge phase, the default commit_every=1000, with and without use_wal=True. In the 40 kills of the probe, no committed vertex or edge chunk was lost, no record was torn, no edge pointed at a missing vertex, and every database reopened and accepted writes. The example below is one of these kills: 30,000 vertices and 70,000 edge records were in the database afterwards.

What is in the database

  • The vertices and edge records that a returned call committed are there. A load that is killed loses the work after the last commit and nothing before it, with one exception: create_vertices on the arcadedb-jvm package (next point).

  • On the arcadedb-jvm package, a killed create_vertices load can leave part of a call in the database. graph_batch() relaxes the write-ahead log for the load, so a transaction is not atomic after a kill. In a kill probe of 240 cycles (kill -9 at a random time during a load of 400,000 rows in calls of 50,000) on the released 26.11.1 package, 236 stored a whole number of calls and 4 stored another count, and in those 4 the stored keys were not a prefix of the input: some earlier rows were missing while later rows were present. No stored value was wrong, no row was stored twice, and the database reopened in every cycle. After an interrupted vertex load, compare the stored keys with the input and reload the type. The same probe on the 26.11.1.post1 candidate build stored another count in 2 of 240 cycles, and in both the stored keys were a prefix of the input. It has not been run on 26.11.1.post2.

  • The edges are not traversable. The engine writes the edge records first and links them to their vertices (the head pointers and the incoming lists) in close(). Until close() has run, outE(), inE(), out() and MATCH find none of them, but SELECT count(*) FROM E counts them. In the example: 70,000 edge records, 0 reachable.

  • CHECK DATABASE reports each of these edges as an ORPHAN RECORD and counts them in unreachableEdgeRecords. It also uses the words "was not connected from", "is not referenced back by" and "missing from that vertex’s OUT/IN list" for the same state. Read the statement in the CHECK DATABASE reference; this page does not repeat it.

Recovery that works

Step Edge records Reachable by traversal CHECK DATABASE afterwards

After the kill, nothing done

70,000

0

70,000 ORPHAN RECORD warnings

CHECK DATABASE (report only)

70,000

0

the same 70,000

CHECK DATABASE FIX

70,000

0

the same 70,000. FIX relinks nothing and deletes nothing here.

CHECK DATABASE FIX DELETE ORPHANS

0

0

nothing left to report

Then load the edges again over the existing vertices and call close()

90,000

90,000

0 warnings

  1. Run CHECK DATABASE. The number of ORPHAN RECORD warnings is the number of committed edges that traversal cannot reach.

  2. Do not expect CHECK DATABASE FIX to repair them. It reports the same warnings before and after, in the same process and after a restart.

  3. Run CHECK DATABASE FIX DELETE ORPHANS. It deletes every edge record that no vertex lists. The vertices stay. Use it only when the interrupted batch is the only damage, because the engine cannot tell these records from other unlisted edge records.

  4. Load the edges again. Do not call create_vertices again with the same keys: on a type with a unique index on the key it fails with DuplicatedKeyException and changes nothing, and without such an index it would create the vertices twice. Look up the RIDs of the vertices that exist, buffer the edges with new_edges, and let close() link them.

db.command("sql", "CHECK DATABASE FIX DELETE ORPHANS")

rid = {r["id"]: str(r["r"]) for r in db.query("sql", "SELECT @rid AS r, id FROM V").to_list()}
with db.graph_batch() as batch:
    batch.new_edges([rid[s] for s in sources], "E", [rid[d] for d in targets], properties)

If you load the edges again without step 3, the old records stay in the database next to the new ones. In the probe this gave 160,000 edge records for 90,000 distinct edges. Traversal is right, because the old records are unreachable, but the edge count of the type is wrong and CHECK DATABASE keeps reporting the 70,000 old records.

Edges are linked only when close() returns, and each batch links its own edges. A long edge load split over several batches, each closed before the next opens, therefore leaves only the edges of the batch that was open at the interruption unlinked.