Upgrade from 0.6 to 0.7

Upgrading the library is a normal package upgrade:

pip install --upgrade engrava

0.7 is a schema-changing minor upgrade, so do not roll it across old and new workers sharing one database file.

Already on 0.7.0? Read From 0.7.0 to 0.7.1: a patch release with no schema change.

Before upgrading

Stop the 0.6 workers (or otherwise guarantee no writes for the duration), then back up the database.

result=$(sqlite3 my-data.db "PRAGMA wal_checkpoint(TRUNCATE);")
busy=$(echo "$result" | cut -d'|' -f1)
if [ "$busy" = "0" ]; then
  cp my-data.db my-data.db.bak
else
  echo "checkpoint busy ($busy); WAL still holds data, not copying" >&2
fi

The checkpoint must actually complete before the copy is trusted — an open reader can leave it busy even with no writer active, and copying the main file anyway captures an incomplete backup silently. See Backup & Recovery for the full explanation.

If the database is in WAL mode and you do not checkpoint first, copy the database together with its -wal and -shm files. Let one 0.7 process run the migration to completion, then start the rest.

engrava --db my-data.db migrate

The migration also runs on the first ensure_schema() call — automatically, if your application already makes that call. Opening the store through SqliteEngravaCore.from_config() makes it for you. The direct constructor, SqliteEngravaCore(conn, ...), does not.

The full OSS upgrade guide has the longer backup and recovery notes: Upgrade Guide.

What changes in 0.7

The core schema advances from user_version = 20 to 21 in one additive step, applied on the first ensure_schema() call.

If the warning appears, find the values in the named columns that are not ISO-8601 timestamps and repair them — the migration does not guess at them. Nothing else about the rewrite changes an instant, only the stored text: a value without an offset was compared as text on 0.6.x, so an expires_at or valid_from/valid_until written without one could expire early or compare against the wrong row. See Bi-temporal Model.

Breaking changes to review

Breaking changes first, most upgraders affected by more than one of these.

A real revision guard now backs every guarded update, and a competing write you never used to see now raises StaleDataError. update_thought and restore_thought already compared a version on write, but against updated_cycle — a value nothing in the store ever advanced on its own, so an ordinary concurrent edit passed silently. The guard now compares and increments revision instead, and, for the first time, update_edge and update_action gain the same guard — before 0.7, neither could ever raise StaleDataError. This also closes a real cross-connection gap: a second connection or process writing the same file can now be detected, not only a second task in the same process. Code that never saw a conflict because conflicts were invisible needs a retry idiom: re-read the row, recompute the change, reissue the call. See Concurrency → Optimistic concurrency and StaleDataError.

Two new exceptions — WriteContentionError and WriteLockTimeoutError — now surface where 0.6.x either duplicated a row, leaked a raw sqlite3.OperationalError, or hung. The dedup window (create_thought with deduplicate=True, get_or_create, upsert_by_hash, bulk_store(deduplicate=True)) now opens with BEGIN IMMEDIATE — unless your own code already opened a transaction directly on the connection — and every guarded write now serialises on one task-reentrant lock per store instance, with a bounded wait instead of an unbounded one — closing the deadlock a task spawned and awaited from inside another task’s own suspend_auto_commit() window used to cause. A same-task re-entry into the dedup window itself now raises DedupLockReentryError instead of blocking on itself forever. If your exception handling around any of these calls already names sqlite3.OperationalError for lock contention, it stops firing — a busy/locked sqlite3.OperationalError now converts to WriteContentionError instead; a non-busy OperationalError still propagates unchanged. Add a WriteContentionError handler, and place it before any broader except EngravaError or bare except Exception in the same try: Python runs the first clause that matches, not the most specific one, so a broader handler written first silently intercepts it. See Concurrency → Busy timeout and the error-handling decision table.

flush_access_buffer() now takes the store’s write lock, so it can raise WriteLockTimeoutError. With access tracking on, a flush that runs while a different task holds the write lock waits up to write_lock_acquire_timeout_seconds; 0.6.x took no lock. A flush that cannot take the lock raises WriteLockTimeoutError and leaves the buffered accesses available for a later retry. A flush with tracking off, or with nothing buffered, returns 0 without asking for the lock. consolidate() flushes first, so it can raise the same error. close() flushes best-effort, does not raise on a failed flush, and writes nothing if that flush cannot take the lock.

gc and restore now refuse outright on a database that is not on this build’s head schema, instead of proceeding. On 0.6, on an unmigrated database with a sqlite-vec backend, the next open’s reconcile could put a deleted thought’s vector back, and the id could reappear as a search-similar hit. In 0.7, reconciliation, the vector-index purge and search resolve a vector through its thought row, and delete_thought, the TTL delete strategy and hygiene GC issue their own deletes for the thought’s edge, embedding and action rows. The read commands (info, verify, export, snapshot, recall on --db or the default path, and a query that parses as a read) now warn instead of staying silent about a behind schema; gc and restore below head now refuse and name engrava migrate; above head (a database written by a newer engrava), engrava migrate cannot help and the refusal says to upgrade engrava itself. recall under --config refuses below head too, and remember and link never migrate the core schema of an existing database: below head they refuse and name engrava migrate. If a gc or restore that used to run cleanly on an unmigrated database now exits non-zero, run engrava migrate first — that is this gate working as intended, not a regression. See CLI → Schema-version checks and Known Limitations → Deletion on a database that has not been migrated.

A merge restore (no --clear) into a target that already has journal rows can now refuse instead of silently replacing data. A fresh target’s journal starts empty, since a snapshot never contains journal_entry. Before 0.7, a merge restore replaced any colliding row unconditionally and never touched journal_entry at all, so a collision could silently orphan a journal entry with no warning; restore --clear didn’t clear the journal either. On 0.7, --clear empties journal_entry along with the data, and a merge restore into a non-empty journal refuses any colliding record unless you pass --orphan-journal-entries to accept the resulting audit gap. This only ever fires on a target you already chose to journal. See Audit Trail → Backup & retention note and CLI → restore.

A resolved recency weight of 0.0 now fully disables cognitive-cycle recency, including on the query-less fallback path — this can reorder results, not just change a score. On 0.6.x, a resolved recency_weight of 0.0 correctly kept recency out of backends_used, but the fallback path (FTS and vector both inactive) still computed a cycle-decayed score and used it ahead of the sort — the weight gated the label, not the number. On 0.7 that axis is genuinely inert: every row gets a flat 0.0 recency contribution. If another signal (commonly priority) is also active, whichever row now has the higher remaining score can win outright where a fresher-but-lower-priority row previously won on the strength of a recency leak the zero weight was supposed to have turned off. On 0.7.0 this also affects a directly constructed SqliteEngravaCore with no SearchConfig, which resolved an omitted recency_weight to 0.0; 0.7.1 resolves it to the documented 0.10, like every other construction path (see From 0.7.0 to 0.7.1). Re-check result order on any query that falls through to this path with heterogeneous priorities. See Hybrid Search → Graceful Degradation.

EngravaMetrics.schema_version moves from Literal[1] to Literal[2], and engrava --format json info drops its own schema_version key. The metrics snapshot gained a measured field, so its shape version bumped — this is the metrics snapshot’s own version, unrelated to the database’s PRAGMA user_version. Update any schema_version == 1 check. Separately, info’s JSON output never actually reported the database’s own schema version despite the ambiguous name: it now reports metrics_schema_version and database_schema_version as two distinct keys, and schema_version is gone outright — a script reading data["schema_version"] gets a KeyError. See Observability → Metrics Fields and CLI → info.

Overriding create_thought() no longer reaches every insert path — override prepare_thought_for_insert() instead. On 0.6.x, an override sitting on create_thought() also ran for get_or_create()’s and upsert_by_hash()’s miss branch, and for each row bulk_store() inserted, because all three called the virtual self.create_thought(...) internally. On 0.7 none of the three do — they reach internal primitives directly — so a create_thought() override now runs only on a direct create_thought() call (or remember()). Move validation or persisted enrichment into prepare_thought_for_insert(), the new seam that covers create_thought, get_or_create, upsert_by_hash, bulk_store and remember: on create_thought (and remember) it runs once per call that passes metadata and provenance validation, before the dedup branch; on get_or_create/upsert_by_hash it runs once only after an observed miss — a stable hit costs zero seam calls. Keep only one implementation — retaining both risks running your logic twice on a direct create_thought() call.

A store instance reused across embedding models or dimensions is no longer exempt after its first write. _ensure_embedding_model_lock used to skip its comparison once a store instance had passed it; it now compares the model name, dimension and document-prefix fingerprint on every store_embedding() call and every verify_embedding_model() call (a REFLECTION’s own centroid vector is exempt). A store instance intentionally pointed at a rotating or growing set of embedding configurations now raises EmbeddingModelMismatchError on the next write with a different identity, instead of accepting it silently. The identity lives in the database’s own _metadata table, so a new store instance against the same file does not help — construct one against a different database per configuration, or catch the error and handle the switch explicitly. See Known Limitations → Embedding Dimension Consistency.

ConnectionQuarantinedError is now reachable from a plain delete_thought() call, with no derived-records seam required. delete_thought (and the TTL delete strategy, and hygiene GC) now delete a thought’s edge, embedding and action rows explicitly inside a savepoint; if the store’s own unwind of that savepoint fails for any reason — a cancellation racing the release, or a trigger/authorizer combination that defeats the rollback — the connection quarantines on the next guarded call. Previously this exception was reachable only through a failed derived-record compensation rollback. Treat it as reachable from any delete path, and replace the store instance on catching it — it is terminal by design. See Error Handling → Terminal connection quarantine.

The dreaming cohesion gate now approximates cosine similarity instead of a raw dot product. cluster_cohesion_score() used to divide by neither vector’s norm, so two members embedded by a provider whose output isn’t guaranteed close to unit length (only SentenceTransformerProvider explicitly normalises; the others pass through whatever their upstream API or callback returns) could look “more cohesive” purely because that provider produced longer vectors for them. If you run the dreaming consolidation loop with gates.cluster_quality_enabled on, re-calibrate cluster_quality_cohesion_threshold against a representative sample of your own clusters — the shift’s size and direction depend on your provider’s typical vector norms and is not the same for every pair in a corpus. See Dreaming → Reflections.

The vector-search backend’s logger moved with the module. It now reports as engrava.infrastructure.sqlite.vector_sqlite_vec instead of engrava.extensions.vector_sqlite_vec — the old import path still works, but a logger’s name follows the real module, not the import alias. Update any log-based alerting rule or dashboard filter keyed on the old name.

Also new in 0.7

Three one-shot CLI verbs: remember, recall, and link. Store a thought, search for one, or create a typed edge between two existing thoughts, each in a single command — no application code required. remember and link create a database that does not exist yet, but never migrate the core schema of one that does: below head they refuse and name engrava migrate. See CLI reference → remember, recall and link.

get_edges() takes an optional limit. The default limit=None returns every edge, as before. An integer >= 0 keeps that many of the highest-weight edges, and 0 returns an empty list; a negative value or a bool raises ValueError. See API reference → get_edges.

Cutover checklist

1. Stop the 0.6 workers.
2. Back up the database (checkpoint the WAL first).
3. Upgrade application dependencies to engrava>=0.7.
4. Run one 0.7 process, or engrava migrate, and let the migration complete.
5. Start the remaining 0.7 workers.
6. Re-check any exception handling around the dedup or guarded-write paths that names sqlite3.OperationalError, and add a WriteContentionError handler ahead of any broader one.
7. If you subclass the store and override create_thought() for validation or enrichment, move it to prepare_thought_for_insert().
8. If your resolved recency_weight is 0.0 with a cognitive-cycle reference present, re-check result order on the query-less fallback path.
9. If you run dreaming consolidation with the cohesion gate on, re-calibrate cluster_quality_cohesion_threshold.

From 0.7.0 to 0.7.1

0.7.1 is a patch release: no schema change (user_version stays at 21), so it is safe to roll across workers. Every change is a defect fix, and each one changes what some caller sees. Upgrading from 0.6 straight to 0.7.1 takes the 0.7 changes above and these together.

MindQL extension handlers receive a read-only accessor. A custom command’s handler used to receive the live database connection. It now receives a ReadOnlyAccessor, whose only method runs one SELECT. A handler that runs anything else, or calls a connection method such as commit, now fails. See Extensions → Custom MindQL Commands.

MindQL’s WHERE parser rejects two shapes it used to run. A leading, trailing or doubled AND used to be dropped, so the query ran without it; it now raises MindQLParseError. So does a WHERE nested more than 256 parentheses deep. Three shapes that failed now work: a value ending in or or and before an operator (source = vendor OR priority = 1), a quoted value that contains AND or ORDER BY, and a doubled quote inside a quoted value (source = 'O''Brien' matches O'Brien, in a comparison and in IN (...)). See MindQL.

The SELECT passthrough’s single-statement check reads comments and quoted identifiers. A ; inside either is accepted, and the statement may end with ;, whitespace and comments. A second statement hidden behind a quote inside a comment is now refused. The same check guards ReadOnlyAccessor.execute().

engrava gc keeps pinned and protected-priority archived rows unless --include-pinned is given, and reports how many it kept. With a journal-enabled --config, it journals each row it deletes. TTL cleanup never expires a pinned row; CleanupResult.pinned_kept_count counts them, and a protected priority still expires. The orphan-REFLECTION sweep keeps pinned and protected reflections, and retires a dreaming REFLECTION whose sources were all hard-deleted. See Data Lifecycle → Running cleanup and Dreaming → When a REFLECTION is retired.

EngravaManager can build fully configured stores. It used to give each service’s store only the PRAGMAs, the embedding provider and the vector backend. The new keyword base_config applies a full EngravaConfig to every store it builds, embedding_provider_factory supplies a provider per service, and EngravaManager.from_config() accepts the whole configuration. Without base_config, the manager wires each store as before; the other fixes on this list still apply to it. See Configuration → Multi-Service Usage.

A large top_k returns a full page. fts_top_k and vector_top_k used to cap each search arm’s candidates, so search_hybrid() and recall() returned fewer results than matched once top_k passed the pool (50 by default). Each arm’s pool is now at least top_k. A top_k at or below the pool is unchanged.

A directly constructed store applies recency. SqliteEngravaCore(conn, ...) with no search_config, and a store EngravaManager builds with neither base_config nor default_search, now resolve an omitted recency weight to 0.1, like every other construction path. With a recency reference (current_cycle, a cycle provider or recency_now), recency now contributes, which can reorder results. To keep the old ranking, pass recency_weight=0.0 to search_hybrid(), or construct the store with search_config=SearchConfig(default_recency_weight=0.0); recall() has no per-call override. See Hybrid Search → Signal model.

The VS Code MCP example loads as shipped. examples/mcp-client-config.vscode.json now holds servers at the top level, with "type": "stdio". See MCP server.

engrava-mcp users need no new server release: engrava-mcp 0.7.0 accepts 0.7.1, but an existing install can keep engrava 0.7.0 until you upgrade engrava with pip install -U engrava, uv tool upgrade engrava-mcp or uvx --refresh engrava-mcp, whichever matches how you installed it. See MCP server → Install.

The package guide has the full notes: Upgrade Guide → 0.7.0 → 0.7.1.

Coming from 0.5 or earlier

0.5 → 0.6 is a separate hop — an additive schema migration with its own behaviour changes — documented in Upgrade to 0.6. Run that upgrade first, then work through the checklist above.