Database Admin

SQL - ALIGN DATABASE

Executes a distributed alignment of the database. It must be executed on the Leader server. The alignment computes a checksum of each file and sends them to the replica nodes. Each replica node will compute the checksum on its own files. The files that are mismatching are requested by the replica to the leader. In the future single pages could be transferred instead of the entire file.

Align Database command is available only when the server is running with the HA module active.

Syntax

ALIGN DATABASE

The command returns which page have been aligned on each server.

Examples

  • Align the current database.

ArcadeDB> ALIGN DATABASE

SQL - ALTER DATABASE

Change a database setting. You can find the available settings in Settings appendix. The update is persistent.

Syntax

ALTER DATABASE <setting-name> <setting-value>
  • <setting-name> Check the available settings in Settings appendix. Since the setting name contains . characters, surround the setting name with `.

  • <setting-value> The new value to set

Examples

  • Set the date time format to support milliseconds (the default is 'yyyy-MM-dd HH:mm:ss').

ArcadeDB> ALTER DATABASE `arcadedb.dateTimeFormat` 'yyyy-MM-dd HH:mm:ss.SSS'
  • Set the default page size for buckets to 262,144 bytes. This is useful when importing database with records bigger than the default page.

ArcadeDB> ALTER DATABASE `arcadedb.bucketDefaultPageSize` 262144

SQL - BACKUP DATABASE

Executes a backup of the current database. The resulting file is a compressed archive using ZIP as algorithm. The archive contains the database directory without the transaction logs. The backup is executed taking a snapshot of the database at the time the command is executed. Any pending transaction will not be in the backup archive. ArcadeDB allows to execute a non-stop backup of a database while it is used without blocking writes or affecting performance.

Syntax

BACKUP DATABASE [ <backup-file-url> ]
  • <backup-file-url> Optional, defines the location for the backup archive. If not specified, the backup file will be backups/<db-name>/<db-name>-backup-<timestamp>.tgz, where the timestamp is expresses from the year to the millisecond. Example of backup file name backups/TheMatrix/TheMatrix-backup-20210921-172750767.zip.

Examples

  • Execute the backup of the current database with the default filename.

ArcadeDB> BACKUP DATABASE

SQL - CHECK DATABASE

Executes an integrity check and in case of a repair of the database. This command analyzes the following things:

  • buckets: all the pages and records are scanned and checked if can be loaded (no physical corruption)

  • vertices: all the vertices are loaded and all the connected edges are checked. In case some edges point to records that have been deleted they can be fixed automatically if the FIX option is enabled.

  • edges: scan all the edges and check the incoming and outgoing links are consistent in the relative vertices. If not, the edges can be automatically removed if the FIX option is enabled.

  • TimeSeries types: every shard’s mutable bucket, sealed store and shared tag dictionary is validated, including the CRC32 of every sealed block. See TimeSeries integrity below.

  • indexes: the structure of each index is validated. A corrupted index is dropped and rebuilt if the FIX option is enabled, keeping its name and its configuration - a full-text index’s analyzers and BM25 parameters, a geospatial index’s precision, a vector index’s dimensions.

the index check is structural: it validates the index’s own pages, not that every record of the type appears in the index. An index that is intact but incomplete - for example one whose build was interrupted by a crash - is therefore reported as healthy. If you suspect that, run REBUILD INDEX on it: rebuilding re-reads every record and also preserves the index configuration.

When a vertex’s edge list is unreadable (for example a lost or corrupted chunk of the adjacency linked list), the FIX mode does not delete anything: every edge record stores its own endpoint identities, so the adjacency list is rebuilt from the surviving edge records. After the rebuild, a full-scope FIX (no TYPE/BUCKET filter) also reclaims the orphaned edge-list segments the old, unreachable chains left on disk.

The other shape is a dangling reference: an adjacency entry that points at a record which no longer exists, typically left behind by a vertex deleted before 26.8.1, when edge removal became strict. There is nothing to rebuild in that case - the record the entry names is gone - so FIX drops the entry instead.

Since v26.9.1, an application that reaches such a record before the repair has run is told so plainly, and always in the same way. Removing an entry from the edge list of a vertex whose own record is missing, adding one to it, and deleting that vertex outright all raise VertexNotFoundException, a RecordNotFoundException (answered as HTTP 404) that names the vertex and points at CHECK DATABASE FIX. Creating an edge whose endpoint is such a record raises it too, naming the endpoint - which is not the vertex the statement named. It is deliberately not a NeedRetryException: the record is gone rather than temporarily unreadable, so a retry can only fail identically, and a job walking possibly-stale references can skip that entry instead of spending its retry budget and rolling back the whole transaction. An unreadable chunk of a vertex that does exist keeps raising the retryable ConcurrentModificationException described above - the two are told apart by which record the failure names.

A DELETE VERTEX refused this way is refused before it touches anything: the vertex’s record is checked first, so the failure does not cost a walk over an adjacency list that is about to be rolled back.

the same answer now covers the genuinely concurrent case. A vertex that another transaction deletes while this one is removing its edges used to be reported as a retryable ConcurrentModificationException; it is a VertexNotFoundException like every other "the vertex is not there", because the record is equally gone either way and a retry could only rediscover that on the next attempt. An application that wraps its vertex deletes in its own catch (ConcurrentModificationException) retry loop no longer catches that case: catch RecordNotFoundException alongside the conflict and treat it as already deleted, which is the conclusion the loop would have reached after spending its retries.

That up-front check reads the vertex’s record, so deleting a vertex needs the readRecord permission on its type as well as deleteRecord (see groups). This is not a new requirement in practice, since every way of obtaining the vertex in order to delete it already reads it, but a group granting deleteRecord alone now fails at the first statement of the delete rather than part of the way through it.

the RECORD scope cannot repair a dangling reference. The RID in the message is the record that is missing, so there is no edge list to rebuild there, and the entry to drop lives on the vertex at the other end. No index maps a vertex back to the lists that reference it, so finding that vertex takes a scan - which is why the message names the database-wide form.

While the check runs, its progress (current step and percentage) is visible live in the console, in Studio, and through the HTTP progress endpoint. Run FIX in a maintenance window: like every repair, it assumes no concurrent writers.

Syntax

CHECK DATABASE [ TYPE <type-name>[,]* ] [ BUCKET <bucket-name>[,]* ] [ RECORD <rid>[,]* ]
              [ FIX ] [ DELETE ORPHANS ] [ RECLAIM UNREFERENCED FILES ] [ DEEP ] [ COMPRESS ]

The optional clauses must appear in the order shown above.

  • <type-name> Optional, if specified limit the check (and the fix) only to the specific types

  • <bucket-name> Optional, if specified limit the check (and the fix) only to the specific buckets

  • <rid> Optional, limits the check to the listed records instead of scanning whole types. Cannot be combined with TYPE or BUCKET. Useful when a single vertex’s edge list is broken and a full scan is not worth its cost. The listed records must exist: repairing a reference that points at a deleted record is a database-wide FIX, not this scope

  • FIX Optional, if used auto fix the issue found with the check

  • DELETE ORPHANS Optional, requires FIX. Removes orphan edge records: edge records whose endpoints are valid vertices but which no vertex’s edge list references. A clause of its own because the same shape is also produced by a vertex that merely lost its head-chunk pointer, whose edges are still recoverable. Without it the finding is still reported, under unreachableEdgeRecords

  • RECLAIM UNREFERENCED FILES Optional, requires FIX. Deletes files this node holds that no schema component was ever built for. Without it the finding is still reported, under unreferencedFiles

  • DEEP Optional, adds the checks that decode the stored data rather than reconciling what describes it. Today only the TimeSeries pass has a deeper tier - see TimeSeries integrity. Independent of FIX: nothing this tier finds is repairable

  • COMPRESS Optional, if used compresses the database with the check, packing every page to reclaim the space a deleted or shrunk record left behind. Independent of FIX: while it runs, it also removes any record-table slot whose declared size cannot fit the page - a physical-corruption shape it walked past and left in place before 26.9.1 - so a database compressed with this release repairs that class of corruption even without FIX

The command returns the integrity check report in one record with the fields:

Field Description

avgPageUsed

average space used in a page. This is a percentage value, where 100% means all the pages are full. If a record is too big, a page is split into multiple pages and multiple reads are necessary. Sometimes it helps to create buckets with larger pages to accommodate large records, see the bucketDefaultPageSize setting.

warnings

set of warning messages.

totalSurrogateRecords

number of records that have been moved between pages. This happens when a record is updated and doesn’t fit the original page, so a surrogate (placeholder) is set to point to the new page/position or it’s a piece of a record stored on multiple pages. Internal, can be ignored.

totalPlaceholderRecords

see above. Internal, can be ignored.

pageSize

internal, can be ignored.

totalDeletedRecords

number of records deleted in the database (allocated minus active). When a record is deleted, it’s RID is not recycled!

totalAllocatedRecords

total number of records created in the database. A record can be of type document, vertex or edge.

totalAllocatedDocuments

see above, but only documents records.

totalAllocatedVertices

see above, but only vertex records.

totalAllocatedEdges

see above, but only edge records.

totalActiveRecords

total number of records that are "active" (not deleted). A record can be of type document, vertex or edge.

totalActiveDocuments

see above, but only vertex document.

totalActiveVertices

see above, but only vertex records.

totalActiveEdges

see above, but only edge records.

totalMaxOffset

internal, can be ignored.

deletedRecordsAfterFix

holds the list of RIDs deleted after a fix.

corruptedRecords

records that are present but cannot be read or are inconsistent. This should be zero; if not, it means there is a bug in the graph operations that doesn’t keep operations 100% transactional. Please report this. A record that simply no longer exists is not listed here: an edge whose vertex was deleted is reported under topMissingReferences instead, and only the edge itself is counted as corrupted. The distinction matters because FIX rebuilds every index on a bucket that holds a corrupted record, which is wasted work when the record is not there at all.

totalPages

total number of pages in the database.

rebuiltIndexes

list of indexes that have been rebuilt automatically if corrupted records were found.

topMissingReferences

records an edge points at that could not be loaded, each with the number of edges that reference it - the usual shape after vertices were deleted while their edges survived. One line per target rather than per edge, so a single missing vertex referenced by millions of edges stays readable.

distinctMissingReferences

how many distinct targets topMissingReferences summarises.

operation

can be ignored, set by the console.

invalidLinks

similar to corrupted records; should be zero.

totalErrors

counter of errors encountered while checking the database. It should be zero.

autoFix

counter of autofix operations executed by the check database under the hood when corrupted records have been encountered. Some operations, like broken edges, can be fixed and restored automatically.

orphanedEdgeSegments

number of unreachable edge-list segments found in the dedicated edge-list buckets (garbage left behind by repaired adjacency lists). Only present on a full-scope FIX.

orphanedEdgeSegmentsReclaimed

number of orphaned edge-list segments actually deleted by the FIX. Only present on a full-scope FIX.

totalTimeSeriesTypes

number of TimeSeries types the check walked. Zero on a database with none, so a clean report can be told apart from a pass that never ran.

totalTimeSeriesShards

number of TimeSeries shards walked, across all types.

totalTimeSeriesSamples

number of samples those shards hold, mutable and sealed together.

totalTimeSeriesSealedBlocks

number of compressed columnar blocks in the sealed stores.

corruptedTimeSeries

names of the TimeSeries types with at least one finding. The findings themselves are in warnings.

repairedTimeSeries

number of TimeSeries repairs a FIX applied. Deliberately not folded into autoFix, which counts record actions only.

timeSeriesRepairs

one line per repair applied, naming the type, the shard and what was rewritten.

TimeSeries integrity

A TimeSeries type has neither record buckets nor indexes, so it is walked by a pass of its own that reads the three on-disk formats it owns: the mutable bucket (.tstb), the shared tag dictionary (.tstd) and the sealed store (.ts.sealed). The default tier reconciles every header and directory against the data it describes and verifies the CRC32 of every sealed block - which means it reads the whole sealed file, the same cost class as the record scan the check already runs over every bucket. A block is otherwise CRC-checked only on the first read that touches it, so a block nothing queries is a block nothing verifies.

DEEP adds the checks that have to decompress. A CRC proves the bytes are the bytes that were written; it proves nothing about whether they mean what the block claims. Three things every read path answers queries from, without ever looking at a value, are verified only under DEEP:

  • the block’s timestamps are sorted — the range iterator binary-searches them, so an unsorted block silently returns a subset of the rows that match

  • the block’s declared per-column min/max/sum are what its values actually add up to — aggregation push-down answers MIN/MAX/SUM/AVG straight from them without decompressing

  • the block’s declared distinct tag values cover the values it holds — block pruning skips a whole block whose declaration does not list the value being filtered on

Each of those, when wrong, produces a wrong answer rather than an error, so nothing else in the engine reports it.

FIX repairs only what is derived and never a sample: the mutable bucket’s page-0 counters, the sealed header’s block count and global timestamp bounds, and the tail of an interrupted sealed append. Those matter beyond tidiness - a wrong global bound is read straight out of the header when the file is opened, so a range query pruned against it silently misses data the file holds, and a tail no reader can see also hides every block appended after it.

A sealed block that fails its CRC, or that DEEP finds inconsistent, is reported and left untouched. It is the only copy of the samples in it, so discarding it is a decision for an operator, who under HA can also rebuild a sealed store by recompacting from the replicated mutable pages.

Examples

  • Execute the integrity check of the entire database without fixing any issue found.

ArcadeDB> CHECK DATABASE
  • Execute the integrity check of the types 'Account' and 'Bill' without fixing any issue found.

ArcadeDB> CHECK DATABASE TYPE Account, Bill
  • Execute the integrity check only on the bucket 'Account_Europe' without fixing any issue found.

ArcadeDB> CHECK DATABASE BUCKET Account_Europe
  • Execute the integrity check of the entire database and auto fix any issues found.

ArcadeDB> CHECK DATABASE FIX
  • Execute the deeper integrity check, which decodes the stored TimeSeries data instead of only reconciling what describes it.

ArcadeDB> CHECK DATABASE DEEP
  • Execute the deeper check on one TimeSeries type, repairing the derived counters it finds wrong.

ArcadeDB> CHECK DATABASE TYPE SensorReading FIX DEEP

SQL - EXPORT DATABASE

Exports a database in the exports directory under the root directory where ArcadeDB is running.

Syntax

EXPORT DATABASE [<url>] [WITH ( <setting-name> = <setting-value> [,] )* ]
  • <url> Optional, defines the location of the file to export. Use:

    • file:// as prefix for files located on the same file system where ArcadeDB is running. For security reasons, it is not possible to provide an absolute or relative path to the file

    • By default the file name is set to exports/<db-name>-export-<timestamp>.<format>.tgz.

  • WITH Optional settings as comma-separated key/value pairs:

    • format The format of the export. Default is jsonl.

      • jsonl exports in JSONL format - a newline-delimited JSON variant

      • graphml exports in the popular GraphML format. GraphML is supported by all the major Graph DBMS. This format does not support complex types, like collection of elements. Using graphson instead of graphml is recommended

      • graphson exports in the GraphSON format supported by all the major Graph DBMS

    • overwrite Set to true to overwrite the export file if it already exists. Default is false.

Examples

  • Export the current database under the exports/ directory with the default format and filename:

EXPORT DATABASE
  • Export the current database to a specific file:

EXPORT DATABASE file://database.jsonl.tgz
  • Export the current database in GraphSON format, overwriting any existing file:

EXPORT DATABASE file://Movies.graphson.tgz WITH format = 'graphson', overwrite = true
  • Export overwriting a previously exported file without specifying a URL:

EXPORT DATABASE WITH overwrite = true

SQL - IMPORT DATABASE

Executes an import of the database into the current one. Usually an import database is executed on an empty database, but it is possible to execute on any database. In case of conflict (unique index key already existent, etc.), the conflicting records will not be imported. The importer automatically recognize the file between the following formats:

  • OrientDB database export

  • Neo4J database export

  • GraphML database export. This format does not support complex types, like collection of elements. Using GraphSON instead of GraphML is recommended

  • GraphSON database export

  • JSON documents or responses

  • JSON Lines documents or responses

Syntax

IMPORT DATABASE <url> [WITH ( <setting-name> = <setting-value> [,] )* ]
  • <url> Defines the location of the file to import. Use:

    • file:// as prefix for files located on the same file system where ArcadeDB is running.

    • https:// and http:// as prefix for remote files.

Examples

  • Import the public OpenBeer database available as demo database for OrientDB and exported in TGZ file

IMPORT DATABASE https://github.com/ArcadeData/arcadedb-datasets/raw/main/orientdb/OpenBeer.gz
  • Import the Movie database used in Neo4j’s examples:

IMPORT DATABASE https://github.com/ArcadeData/arcadedb-datasets/raw/main/neo4j/movies.graphson.tgz
  • Import a JSON response into document type mytype

IMPORT DATABASE http://echo.jsontest.com/key/value/one/two WITH documentType = 'mytype'
  • Test data source

IMPORT DATABASE http://echo.jsontest.com/key/value/one/two WITH probeOnly = true
  • Import a graph from CSV, with vertices in a vertices.csv which contains a column Id, and edges in a edges.csv which contains columns From and To, both placed in the ArcadeDB base folder:

IMPORT DATABASE file://empty.csv WITH vertices="file://vertices.csv", verticesFileType=csv, typeIdProperty=Id, typeIdType=Long, edges="file://edges.csv", edgesFileType=csv, edgeFromField="From", edgeToField="To"

See also Importer for a description of the settings.