Data lifecycle, retention & deletion

How a thought moves through its lifecycle, how time-to-live expiry works, and — importantly for privacy and compliance — what it takes to truly erase data, including the residue a naive delete leaves behind.

Compliance note. This page describes the mechanics honestly so you can build a correct retention/erasure process. The default expiry strategy archives (does not erase), and a hard delete can still leave content in the audit journal and in backups. Read the GDPR / hard deletion section before relying on TTL for “deletion”.

Lifecycle states

Every thought carries a LifecycleStatus. There are four states:

You set the status on the ThoughtRecord you create, and update it over the thought’s life. Archiving is the soft-retire step: an ARCHIVED thought still exists (and its content is still stored) until you garbage-collect it.

The ordinary lifecycle state machine is:

CREATED -> ACTIVE -> DONE -> ARCHIVED
              \--------------> ARCHIVED
ARCHIVED --restore_thought()--> ACTIVE

The direct ACTIVE -> ARCHIVED transition is valid; a thought does not have to pass through DONE before being manually archived. restore_thought() is the canonical reverse path: it returns an archived thought to ACTIVE and clears the hygiene archive stamps. Two maintenance mechanisms have dedicated archival paths outside the ordinary evolve() sequence:

• TTL cleanup with the archive strategy archives any expired row that is not pinned, clears its expires_at, and clears the hygiene-specific archive stamps.
• Memory Hygiene may archive an eligible ACTIVE or CREATED row, clears its expires_at, and stamps archived_at_cycle plus archived_at for its restore windows.

Both remain reversible through restore_thought() until a hard-delete or garbage-collection stage removes the row.

ARCHIVED is excluded from default retrieval, but not unconditionally counted. Marking a thought ARCHIVED is a retention state — the row and its content stay in the database — but an archived thought is dropped from default ranked retrieval: search_hybrid / recall / search_fts / search_similar no longer return it unless you pass include_archived=True (and restore_thought re-activates it). This retrieval exclusion is the retrieval side of Forgetting (Memory Hygiene) — an opt-in, off-by-default hygiene loop — but the exclusion itself applies to any archived row (including one archived by the TTL archive strategy or manually), whether or not that loop is enabled. count_thoughts() / list_thoughts() include an archived row by default only if it is not also expired — those two methods are not ranked retrieval and stay lifecycle-neutral, but they still apply the same default expiry exclusion as everything else; pass include_expired=True to include an archived-and-expired row there too. Expired thoughts are likewise excluded from the four ranked retrieval methods listed above; those methods do not offer an include_expired escape hatch — that parameter belongs only to list_thoughts() and count_thoughts(). A retired REFLECTION (one whose lifecycle_status is no longer ACTIVE) stays out of search by a type-specific freshness floor even under include_archived=True.

Time-to-live (TTL) and expiry

A thought can carry an expiry time. Three ways to set it:

• Per-thought, absolute: set ThoughtRecord.expires_at to a timestamp.
• Per-thought, relative at create time: pass expires_after_seconds= to create_thought(...), which computes expires_at for you.
• A default for the whole store: ttl.default_ttl_seconds in config applies a default TTL to new thoughts that don’t set their own (see Configuration → ttl).

When more than one is available at creation, precedence is explicit and deterministic: the expires_after_seconds argument wins, then the record’s own expires_at, then the configured store default. The default is used only when neither per-call nor per-record expiry was supplied.

expires_at is stored in one canonical UTC form (2026-07-01T00:00:00+00:00), whichever ISO-8601 form you pass: an offset-aware value is converted to UTC, and a value with no offset is read as UTC. Expiry compares that stored value against the current UTC instant, so 2026-07-01 00:00:00, 20260701T000000, and 2026-07-01T02:00:00+02:00 all expire at the same moment.

Expiry is not automatic on a timer. Expired thoughts remain until a cleanup pass runs (see running cleanup below). By default, expired thoughts are excluded from count_thoughts(...) and list_thoughts(...) — pass include_expired=True to include them:

async def compare_counts(store) -> None:
    live = await store.count_thoughts()  # excludes expired
    everything = await store.count_thoughts(include_expired=True)

Ranked retrieval (search_hybrid, recall, search_fts, and search_similar) always excludes rows whose expires_at has passed, even before cleanup has archived or deleted them. include_expired belongs only to the list/count APIs; it is not a ranked-search parameter.

A pinned thought is never expired. A cleanup pass skips every row with pinned = True, whether it runs through cleanup_expired(), the ttl.check_every_n_operations cadence or engrava gc --expired: the row is neither archived nor deleted, and it is counted in pinned_kept_count (below) instead of expired_count. A protected priority (P1 by default) is not exempt: that protection belongs to Forgetting, and a TTL is the row’s own explicit lifetime. Only pinned exempts a row from expiry.

Archive vs. delete

What a cleanup pass does to an expired thought is governed by the store’s TTL strategy, set via ttl.strategy in config (see Configuration → ttl):

The default is archive — chosen so expiry is non-destructive and auditable. This means expiry alone does not erase anything under the default configuration. To make expiry actually remove rows, set ttl.strategy: delete.

Running cleanup

Expiry is applied by an explicit cleanup pass — nothing happens on a timer.

From Python: cleanup_expired() returns a CleanupResult:

async def run_cleanup(store) -> None:
    result = await store.cleanup_expired()
    print(result.expired_count)  # how many thoughts were expired
    print(result.strategy_applied)  # "archive" or "delete" (per config)
    print(result.timestamp)  # ISO-8601 time of the pass
    print(result.pinned_kept_count)  # past-TTL rows kept because they are pinned

You can also have the store run cleanup automatically via ttl.check_every_n_operations (default 0 = manual only). This counts completed thought create/update calls, not every store operation — a read, a delete, or an edge write does not advance the counter.

From the CLI: engrava gc --expired runs the expiry cleanup per your TTL strategy. What it does next depends on that strategy:

engrava gc --expired            # run expiry cleanup (per ttl.strategy)
engrava gc --expired --dry-run  # preview the pass, change nothing
engrava gc                      # delete ARCHIVED thoughts + their edges/embeddings/actions

• With ttl.strategy: delete: the expired rows are deleted outright, and the same pass then garbage-collects any pre-existing ARCHIVED thoughts.
• With ttl.strategy: archive (default): the expired rows are archived (marked ARCHIVED), and a real run stops there — but only if it archived at least one row. (Collecting the rows it just archived would defeat the soft-retire.) With no expired rows to archive it falls through to collecting pre-existing ARCHIVED rows, exactly as a plain engrava gc would. So a repeated gc --expired under the archive strategy does eventually collect: it stops on the run that archives something and collects on the run that does not. --dry-run does not follow that same stop condition, because a dry run never actually archives anything: it always evaluates as if the pass archived nothing, so it can preview the archived-collection deletion pass in the same invocation where it also reports expired rows it would archive — a preview a real run under the same starting conditions would not reach until its next invocation.

Plain engrava gc (no --expired) removes ARCHIVED thoughts together with every edge touching one on either end — including edges whose other end is still live — their embeddings and the actions sourced from them, then reconciles the vector index by removing every vec0 row whose embedding row is missing or no longer joins to a live thought. This is how archived data is finally deleted from the live table.

gc keeps pinned and protected-priority rows by default. An ARCHIVED thought that is pinned, or at a priority in the hygiene policy’s protected set, survives a plain gc, which prints how many it kept. Pass --include-pinned to delete them too. See Forgetting → Protection for the default protected set (P1); --config’s hygiene_policy: section overrides it. When --config names an engrava.yaml with journal.enabled: true, gc also appends a DELETE_THOUGHT journal entry for each row it deletes, in the same transaction as the delete.

gc refuses to delete on a vec0-indexed store without the vector extra. If the database carries an embedding_vec table and sqlite-vec cannot be loaded — most commonly because engrava[vec] is not installed, though an unsupported build, an OS error or a SQLite error fail the same way — a pass that is about to physically delete stops before deleting anything and exits 1 with:

Error: This database has a sqlite-vec index, and collecting thoughts without removing their vectors would strand them in it. Install 'engrava[vec]' and retry.

Removing the rows without removing their vectors would strand those vectors in an index nothing can then reach them through. Only deleting passes are refused: --dry-run is never refused, and neither is a run with nothing to delete. Read the archive strategy carefully, though — gc --expired under the default ttl.strategy: archive stops after archiving only when it actually archived something. With no expired rows to archive it falls through to the archived-collection pass, which is refused when there are archived rows to collect. Install the extra (pip install 'engrava[vec]') and retry.

GDPR and hard deletion

If you must erase a user’s data (e.g. a GDPR erasure request), be aware that neither archiving nor a single delete is sufficient on its own. The procedure below covers three things a hard erasure has to address — live rows, the audit journal, and backups — and it is not exhaustive: it does not cover SQLite’s own WAL/freelist residue or a remote embedding provider that already received the text. See the callouts after the procedure for both.

1. Archive does not erase. Under the default ttl.strategy: archive, an “expired” thought is only marked ARCHIVED — the row and its content remain in the database. Note that engrava gc --expired under the archive strategy archives the rows and stops only if it archived at least one — on that run it does not also delete archived rows. With no expired rows to archive the same command falls through and does collect pre-existing ARCHIVED rows, so do not read “it only archives” as a guarantee that nothing was deleted. To remove the row deliberately, run a separate engrava gc, or use ttl.strategy: delete so the row is deleted outright. If the row is pinned, or at a protected priority (P1 by default), engrava gc keeps it: pass --include-pinned when an erasure target may be one of those.
2. Any journal entry referencing the thought’s id can carry its content — for journal-enabled store methods. If the audit journal is enabled, an INSERT_THOUGHT entry’s delta.after, an UPDATE_THOUGHT entry’s delta.before and delta.after, and a DELETE_THOUGHT entry’s delta.before can each hold the thought’s full content — so the data can survive in journal_entry after the row itself is gone, in more than one entry and more than one mutation type. Whether a given entry exists depends on whether journaling was on when that mutation happened: a thought created while journaling was off, or brought in by restore, has no INSERT_THOUGHT entry. The CLI GC paths (engrava gc, gc --expired) construct their own connection, and journal a deletion only when --config names an engrava.yaml with journal.enabled: true. Without --config, or with journaling disabled there, a deletion made that way records no DELETE_THOUGHT entry, even when a store opened elsewhere has journaling on. Either way, an earlier INSERT_THOUGHT/UPDATE_THOUGHT entry carrying the content stays. A true erasure must find every journal_entry row whose target_id matches the thought, regardless of mutation_type, and purge all of them. Purging a non-tail entry breaks the retained chain’s linkage from that point and needs an explicit re-baseline if you depend on verification; purging a self-consistent suffix, including every entry, does not by itself make local verification fail — see Audit Trail → Security model.
3. Backups. Any snapshot or file backup taken before the deletion still contains the data. Erasure must extend to your backup retention.

Don’t treat “the thought no longer appears in search” as “the data is gone” — work through all three above for the deletion path you actually used.

A database below the core-12 schema can carry a deleted thought’s embedding row. That schema has no ON DELETE CASCADE on embedding, so a delete made by an older engrava build could leave the row, vector included, behind. Reconciliation (the pass that runs when a sqlite-vec backend opens) backfills a vector only for an embedding row whose owner id matches a row in thought, and delete_thought issues its own delete for the thought’s embedding row. The core-12 step of engrava migrate purges the dangling embedding rows that earlier deletes left behind. Full mechanism: Known Limitations → Deletion on a database that has not been migrated.

Memory-hygiene GC is a hard delete, not erasure. The opt-in Forgetting (Memory Hygiene) garbage-collection stage removes archived rows exactly like the deletes above: it reclaims the live/queryable working set, but when journaling is on the content survives in the DELETE_THOUGHT journal entry. Treat it as cognitive cleanup, not guaranteed erasure.

What this procedure does not cover. SQLite’s own on-disk residue — pages in the -wal file or the freelist that still hold pre-deletion bytes — is not addressed by any step above; a caller with that threat model needs a stronger disk-level answer than this page provides. Nor does it cover an export or snapshot file made before the delete, still holding the deleted record, that has not yet been rotated out — distinct from the file backups point 3 above already names. Neither does it cover a remote embedding provider that already received the thought’s text when it was embedded: that provider’s own retention is outside Engrava’s control.

Reclaiming disk space

Deleting rows — whether via ttl.strategy: delete, engrava gc, or a hard erasure — does not by itself shrink the database file under SQLite’s default auto_vacuum=NONE (Engrava does not force a different setting): the freed pages return to an internal free-list and are reused for future writes, so the file stays the same size on disk. A database externally configured with auto_vacuum=FULL is the exception: SQLite can truncate freed tail pages automatically at commit, without a VACUUM.

To reclaim file size on the default configuration you must run VACUUM, which rebuilds the database into a compact file. Plan for its cost:

• Write access, not full exclusivity. VACUUM requires write access and blocks competing writers for its duration. Engrava runs in WAL mode, so an existing reader with an established snapshot can continue reading while VACUUM runs on another connection — it is not a blanket “no concurrent reads or writes” lock.
• Temporary space. It writes a fresh copy before swapping, so it needs additional temporary disk space whose size depends on the database’s contents and SQLite’s own temporary-storage behavior — check the headroom for your deployment rather than assuming a fixed multiple.
• Off-peak. On a large database this can take a while; schedule it off-peak.

VACUUM;                 -- rebuild in place (blocks writers; needs temp space)
VACUUM INTO 'copy.db';  -- write a compacted copy without locking in place as long

Until you VACUUM (or unless auto_vacuum=FULL is configured), expect the file size to reflect the high-water mark, not the live row count — this is normal SQLite behaviour, not a leak.

For the exhaustive package reference, see the OSS data-lifecycle.md docs.