Backup of a Database

ArcadeDB allows to execute a non-stop backup of a database while it is used without blocking writes or affecting performance.

A backup, and a CHECK DATABASE verification, always reflect every transaction committed before they started. (Since v26.10.1: under specific timing, one of these operations starting right as another one on the same database was finishing could very rarely produce a snapshot a few milliseconds out of date, missing the most recently committed transaction(s). This is now fixed.)

There are two ways to perform backups:

  • Manual backup - Execute a one-time backup using SQL or the Studio UI

  • Automatic backup - Schedule recurring backups with retention policies (see Automatic Backup Scheduler)

For manual backups, you can execute the backup of a database from SQL. Look at Backup Database SQL command for more information.

Consistency of TimeSeries data

A backup captures the database schema at a single point in time. TimeSeries types keep their compacted history in separate .ts.sealed files, which are archived alongside it.

(Since v26.10.1) the schema and the list of those files are captured together, so an archive can no longer declare a TIMESERIES type whose data it does not contain. Previously a DROP TYPE running at the same moment as a backup could produce exactly that, and the backup still reported success.

(Since v26.10.1) if one of those files cannot be read while the backup runs, the backup now fails with an error naming the file instead of skipping it and reporting success.

This is deliberate. An archive that silently omits data its own schema declares looks valid until a restore is attempted, which is the worst moment to discover it. A failed backup is visible, and simply running it again produces a consistent archive. The same applies when the database directory cannot be listed at all: that is an error rather than being read as "this database has no TimeSeries data".

Note that this costs nothing in availability. The database is locked only long enough to take the snapshot and list the files, not for the duration of the backup, so schema changes still run normally while a backup is in progress.

Vector indexes

A vector index keeps its proximity graph in a .vecgraph file, next to small side files that describe it: a manifest that vouches for the graph’s pages (.vecgraphfp), an ordinal map (.vecordmap) and, for a PRODUCT-quantized index, the quantization (.vecpq). The graph is rebuilt from the vectors in the index pages whenever it cannot be verified, so a restored database always answers searches; what changes is whether its first search has to build the graph first.

(Since v26.10.2) a full backup archives the graph together with its side files when it can prove that they belong to the same graph as the pages captured by the backup: the files are unchanged across the point at which the backup took its snapshot. The restored database then verifies the graph against its manifest and reuses it. When the proof fails, for example because the graph was being persisted at that moment, or when the backup had to freeze the data files instead of using a point-in-time window (see how-to/operations/monitoring.adoc#howto-monitoring), the graph is archived empty and without side files. Each such vector index then builds its graph once, on its first search after the restore.

An archive taken with 26.10.1 or earlier carries the graph pages without their side files. After restoring it, a vector index reuses those pages without being able to verify them: the reuse is counted in unverifiedGraphReuses in the index’s getStats() and logged at WARNING (Vector index <name> is reusing a persisted graph that carries no manifest …​). Run REBUILD INDEX on each vector index after restoring such an archive. A PRODUCT-quantized index rebuilds its graph on its own, since its quantization file is missing.

Configuration

  • -f <backup-file> (string) filename of, or path to the backup file to create.

  • -d <database-path> (string) path on local filesystem where to find the ArcadeDB database.

  • -o (boolean) true to overwrite the backup if already exists. If false and the backup-path already exists, an error is thrown. Default is false.

Cloud Backups

In a container setting it may become necessary to send backups to an S3 bucket instead of a mounted volume. Currently, ArcadeDB does not support writing a backup to S3 directly, but there are two ways of achieving this:

  1. Using an intermediary container which forwards a volume’s contents to an S3 bucket. The project docker-s3-volume makes this easy.

  2. Mounting an S3 bucket inside the ArcadeDB container directly with the S3 file system via filesystem in userspace.