Backup & Recovery

Two ways to back up an Engrava database, what each one covers, and how to restore and verify. The most important things to know up front: a logical snapshot does not include the audit journal, the export is a consistent point-in-time snapshot even while writes continue (see below for what that guarantees and what it does not), and a naive file copy in WAL mode can lose data — all three are explained below.

Two kinds of backup

Pick the logical snapshot for portability and selective restore; pick a physical backup when you need the audit journal preserved or point-in-time file recovery — see below for which physical method gives you a byte-exact copy and which gives you a consistent one.

Logical snapshot and restore

engrava --db engrava.db snapshot -o backup.jsonl   # export
engrava --db fresh.db   restore  -i backup.jsonl   # import into a fresh db

The snapshot is JSONL: a metadata header line, then one record per thought / edge / embedding / action. The export opens one read transaction before its first read and holds it across the metadata header and all four table scans, so the snapshot is a consistent point-in-time view rather than four independent scans a concurrent writer could interleave with — a writer committing a new thought and its edge between two scans can no longer leave the edge in the snapshot while the thought it references is missing. It writes to a temporary file next to -o and publishes it only once that read has closed successfully, so a failing or cancelled export leaves an existing file at -o untouched.

A snapshot does NOT include the audit journal. The journal_entry table — the tamper-evident hash chain — is not exported by engrava snapshot, and therefore is not recreated by restore. What that leaves in the target’s journal depends on how you restored:

• A fresh target starts with an empty journal — there was no prior journal for it to have.
• restore --clear empties journal_entry along with the four core tables it wipes, so the journal does not outlive the data it described.
• A restore without --clear merges into an existing target instead — and if that target’s journal is non-empty, it refuses any record that collides with an existing row (a primary key or UNIQUE constraint), rolling the whole restore back rather than letting the merge replace a row the journal still describes. --orphan-journal-entries allows the merge anyway, accepting that the journal may end up describing data the merge discarded, and verify will keep reporting the chain as valid regardless. The gate never fires against an empty journal, which is the ordinary case since journaling is opt-in. See CLI reference → restore for the exact refusal message.

If audit continuity matters, use a physical file backup (which copies the journal verbatim), not a logical snapshot. See Audit Trail.

restore options worth knowing (see the CLI reference for the full list): --clear to empty the target’s four core tables and its journal (not _metadata, extension_schema_versions, or extension-owned tables) rather than wipe the whole target database, --skip-embeddings / --re-embed to control embedding handling, --orphan-journal-entries to allow a merge the collision gate above would otherwise refuse, and --service for multi-service targets. When --clear encounters a persisted sqlite-vec index, restore drops the derived table transactionally and the next configured open rebuilds it. Keep engrava[vec] installed for that virtual-table reset. Every restore — merge or --clear — also checks for the full-text search index and, when it exists, rebuilds it from the target’s thought rows before committing, so a restore that replaces a colliding record does not leave keyword search matching stale content.

Embedding handling during restore

A normal restore imports the embedding rows carried by the snapshot. --skip-embeddings discards those rows and restores text/graph/action data without vectors.

--re-embed also discards the snapshot vectors, then generates new vectors for every imported thought. The provider must come from the configuration passed via --config. A top-level provider supports single-database restore and acts as the fallback for configured services:

database:
  path: ./fresh.db
embeddings:
  provider: sentence-transformer
  model: sentence-transformers/all-MiniLM-L6-v2

engrava --db fresh.db --config engrava.yaml restore -i backup.jsonl --clear --re-embed

A service can inherit that top-level provider or override it:

embeddings:
  provider: ollama
  model: nomic-embed-text
services:
  data_dir: ./data
  default_service: main
  configs:
    main:
      embeddings:
        provider: sentence-transformer
        model: sentence-transformers/all-MiniLM-L6-v2

engrava --config engrava.yaml restore -i backup.jsonl --service main --clear --re-embed

services.configs.<name>.embeddings has precedence over the top-level fallback. Without either provider, --re-embed fails before importing records. Retain the snapshot embeddings or use --skip-embeddings when no provider is available. The re-embed and its model/dimension/prefix identity update commit atomically. Use a fresh target or --clear: a target with existing embeddings is rejected without --clear so surviving vectors cannot be mislabeled as the new corpus. The general --clear sqlite-vec reset described above also protects this path. --skip-embeddings and --re-embed are mutually exclusive.

Trust boundary

restore is designed for snapshots you produced yourself with engrava snapshot — your own trusted backups. That is the supported input boundary.

Restore does not blindly trust the file. Every non-empty line must be a JSON object — blank lines are ignored — and each record targeting a known table (thought, edge, embedding, action) is validated against a fixed, code-owned schema: its columns must be a known subset of that table’s columns, the required columns must be present and non-null, and each value must match its column’s type (an imported embedding vector must be valid base64 — the base64 check applies only when embeddings are actually imported, not under --skip-embeddings or --re-embed, which never read the stored vectors). A record whose type is not one of those tables is skipped, so a newer snapshot restored into an older build ignores record types it does not understand rather than failing. The whole restore runs as one transaction: if any record is malformed or carries a bad value, the transaction is rolled back and no rows are written — not even a --clear that ran first, and not the records that validated before the bad one. Column names never come from the file; the SQL uses a fixed, built-in column set, so a hand-edited or corrupted key cannot alter the statements that run.

What restore does not do is vouch for the content of a snapshot from someone else. The thought text, metadata, and other values are restored verbatim, so a snapshot obtained from an untrusted third party is untrusted data in your database — treat it with the same caution as any external import. Restore a snapshot you trust the origin of; if you must ingest an external one, review it first. Restore also assumes the file is not being modified while it runs; point it at a completed backup, not a snapshot that is still being written.

Physical file backup (WAL-safe)

Engrava runs in WAL mode, where recently-written data lives in the -wal file until it is checkpointed into the main .db. A plain file copy is only safe under specific conditions, so choose the method by whether the database is live (being written) or stopped.

If the database is live (writers running)

A file copy of a database under active writes is not reliable — the .db and -wal change during the copy and can be captured inconsistently. Use a method that produces an internally consistent copy without stopping writers. Neither of these two produces a byte-exact copy of the original file — see the checkpoint method below for that.

SQLite Online Backup API — a hot, consistent backup driven from your own code via Python’s sqlite3 backup API (source.backup(dest)). This is the recommended way to back up a running database, and it supports incremental copies.

VACUUM INTO — writes a fresh, consistent, compacted copy of the database to a new file. SQLite serialises it correctly against ongoing activity:

sqlite3 engrava.db "VACUUM INTO 'engrava-backup.db';"

Both produce a single clean .db you can store or move; neither requires copying the -wal/-shm files, and neither is guaranteed byte-identical to the source file — VACUUM INTO rebuilds the database into a fresh, compacted file, which typically differs in size and byte layout even though its rows match.

If you can stop or quiesce writers

When you can take the database offline (or guarantee no writes for the duration), a copy is safe once it captures every committed change: copy the main file only after a checkpoint has folded the WAL back into it and the checkpoint reports it completed, or copy the whole file set (below). Stopping writers alone does not guarantee the checkpoint completes — an open reader can still pin WAL frames — so check its result before copying rather than copying unconditionally.

Checkpoint, then copy only if it completed:

# With no writers (and, ideally, no readers) active. The PRAGMA prints
# busy|log_frames|checkpointed_frames. busy = 0 means the checkpoint
# completed, so the main file alone holds every committed change. A
# non-zero busy means it did not (an open reader is a common cause), and
# the main file may then lack changes still in the WAL. `test -f` comes
# first because the sqlite3 CLI creates an empty database at a path that
# does not exist. The copy goes to a temporary name and replaces the
# backup only once complete, so a failed copy leaves an earlier backup
# intact.
test -f engrava.db &&
  [ "$(sqlite3 engrava.db 'PRAGMA wal_checkpoint(TRUNCATE);' | cut -d'|' -f1)" = "0" ] &&
  cp engrava.db engrava.db.bak.tmp &&
  mv engrava.db.bak.tmp engrava.db.bak ||
  { echo "no backup made: missing file, incomplete checkpoint, or failed copy" >&2; false; }

wal_checkpoint(TRUNCATE) returns three integers: a busy flag, the number of WAL frames, and the number checkpointed. An open reader can hold WAL frames pinned and prevent a truncating checkpoint even with no writer active — the PRAGMA then reports a busy result, and copying the main file anyway captures an incomplete copy.

Or copy the file set (engrava.db + -wal + -shm) as one atomic unit — e.g. via a filesystem-level snapshot (LVM, ZFS, a cloud volume snapshot) that captures all three at the same instant. A plain cp of the three files of a live database is not atomic and can still be inconsistent; only do the multi-file copy when writers are stopped or behind a consistent snapshot.

Do not rely on a bare cp engrava.db backup.db — or even a non-atomic cp engrava.db engrava.db-wal engrava.db-shm ... — while the database is being written. For a live database use the Online Backup API or VACUUM INTO.

Restoring

• From a snapshot: engrava --db <target> restore -i backup.jsonl. Restore into a fresh database (optionally --clear an existing one) — the snapshot never carries the journal, so what that leaves in the target’s own journal depends on which of the three restore shapes above you used. For --re-embed, pass --config with a top-level provider or a per-service provider override.
• From a physical backup: stop the process, put the backed-up file in place, and start again. A backup made with the Online Backup API, VACUUM INTO, or a checked checkpoint-then-copy is a single self-contained .db. If instead you captured a multi-file filesystem snapshot, restore engrava.db, engrava.db-wal, and engrava.db-shm together as the unit they were snapshotted in.

Verify a restore

After restoring, confirm the database is readable and the counts look right:

engrava --db restored.db info     # reports counts; confirms the schema is readable

For a snapshot restore you can compare info counts against the source. If you rely on the audit journal and restored from a physical backup, also re-run journal verification (see Audit Trail) to confirm the chain is intact.

Multi-service backups

With EngravaManager, each service is its own database file under the shared data directory. Back them up the same way — either snapshot each service (snapshot --service <name>) or take a WAL-safe physical copy of each <name>.db (plus its -wal/-shm). Because services are independent files, you can back up, restore, or delete one without touching the others.

For the exhaustive package reference, see the OSS backup-and-recovery.md docs.