Error Handling and Recovery

This guide describes the error and persistence contracts of Engrava v0.7. It is for application code that must decide whether to fix an input, retry a remote call, reconcile a partial write, or replace a store.

The central rule is: classify the failure and establish what committed before retrying. An exception does not universally mean “nothing was written,” and Engrava does not make every write idempotent.

Decision table

Caller, data, and configuration errors

These failures require a changed request, not backoff:

• ConfigError means configuration loading, provider resolution, or a bounded configuration value failed validation.
• ValueError is also used by some public write boundaries, including duplicate thought IDs and invalid metadata/provenance. Pydantic may raise its own validation error while constructing records.
• InvalidFilterError and InvalidFilterPathError reject an invalid compiled metadata or visibility filter before query execution.
• RecencyModeConflictError, InvalidRecencyArgumentError, and CycleProviderError identify an invalid recency request or provider result.
• EmbeddingModelMismatchError, EmbeddingQueryPrefixMismatchError, and VectorDimensionMismatchError protect the corpus from incompatible vectors. Restore the matching model/prefix/dimension or deliberately re-embed the corpus; repeating the same call cannot repair the mismatch.
• EmbeddingProviderContractError identifies a custom embedding provider that does not expose a required EmbeddingProviderProtocol member — today a public dimension. It names the provider class and the member; retrying cannot help, add the property. A provider that has the property and fails inside it does not raise this: its own exception propagates unchanged, and is classified by whatever raised it.
• ThoughtNotFoundError, ActionNotFoundError, SourceThoughtNotFoundError, and ReferentialIntegrityError identify a missing target or parent.
• DuplicateEdgeError identifies an existing (from, to, type) relationship; callers can safely use it as the idempotent-create signal without parsing SQLite error text.
• InvalidTransitionError rejects an illegal lifecycle/action transition, and ReadOnlyViolationError rejects a write through ReadOnlyEngrava.
• StaleDataError means a guarded update matched no row: another guarded write bumped the row’s revision between your read and your write, or the row was deleted in that window. The error does not say which, and nothing of the rejected update was written. It is recoverable only through a new read-modify-write cycle: re-read the record, recompute the change, and do not replay stale changes without checking the newer record. The engine bumps revision on every guarded write, so this fires whatever field the other write touched. Between ordinary tasks sharing one store, the write lock (Concurrency) already serialises guarded writes, so it almost never fires there; the shapes it does catch are a write from a second store on the same file and a same-task nested write reached through a caller-owned hook — see Concurrency. Its absence is not evidence that no one else wrote.
• WriteLockTimeoutError and DedupLockReentryError are raised instead of a hang. The first means a task waited longer than write_lock_acquire_timeout_seconds for the store’s write lock; the exception alone does not say whether a task was spawned and awaited from inside another task’s open suspend_auto_commit() window (a caller bug: issue the window’s writes from the one task that opened it) or the bound is too small for a slow embedding provider or large batch (raise it). The second means the same task tried to re-enter the dedup window it already holds — typically an update_thought() override reached from upsert_by_hash()’s hit branch that calls back into create_thought(deduplicate=True), get_or_create(), upsert_by_hash() or bulk_store(deduplicate=True). Repeating the call unchanged cannot help; see Concurrency.
• DerivedRecordError means a producer violated a derivation gate or collided with an unrelated identity. The source thought is already durable on the automatic post-store path; some earlier derived children may also be durable.

JournalIntegrityError and ExtensionMigrationError are operator failures, not caller mistakes. Do not keep opening the same files in a write-capable process. Preserve the database and WAL files, inspect the reported sequence or migration, and follow Backup & Recovery.

SchemaVersionError and CoreMigrationError are the same kind of failure, raised by ensure_schema() and so by from_config(), EngravaManager.get_store(), and engrava migrate. SchemaVersionError carries current_version and a reason:

• newer_than_head: the file was written by a newer Engrava. engrava migrate cannot help; upgrade Engrava before opening it.
• populated_sub_floor and stale_shape_sub_floor: the file is stamped below the bootstrap floor and this build cannot tell which older version produced it, so it refuses instead of stamping it current. Do not delete or re-initialise it — preserve it and follow Backup & Recovery.

CoreMigrationError means a migration step returned without leaving the structure it targets (a column, table, index, foreign key, or trigger) in place. The version is not stamped past the last fully applied step, so the next ensure_schema() retries the remaining steps rather than treating a half-migrated file as current; repeating the open without changing anything is not a fix. Replace the database from a backup, or use a build that matches its schema history.

The API Reference lists the documented exceptions.

Remote embedding failures

Provider behavior is deliberately provider-specific:

• OpenAICompatibleProvider retries transport timeouts/network errors and HTTP 408, 409, 425, 429, 500, 502, 503, and 504. It makes at most max_attempts attempts (default 3), waiting base_retry_delay_s * attempt_number before each next attempt (default waits: 1 second, then 2 seconds). Other non-success statuses, such as 400, 401, 403, and 404, fail immediately.
• OllamaProvider and HuggingFaceProvider do not add an Engrava retry loop. Their dependency/provider exceptions propagate.
• Local, callback, and third-party providers define their own failure behavior.

Once provider attempts are exhausted, auto-embedding on the thought a caller explicitly created or updated never swallows the error: with require_embedding=true, Engrava wraps a failure of the provider’s embed call in EmbeddingGenerationError; otherwise the provider’s original exception is re-raised (see Embeddings for what that covers). Strict mode changes the exception type, not the single-item commit boundary. A derived child’s own embedding failure is a separate path — under the default on_error="log" derivation-gate policy, it is logged and derivation continues, so the call that created or updated the source does not see it — see Derived records below.

For a search that only failed while generating its query vector, retrying the search after a transient provider recovery does not replay a graph write. For a write with auto-embedding, inspect the persistence boundary below before doing anything else. See Embeddings for provider setup and retry controls.

Persistence boundaries

Single thought creation

Outside suspend_auto_commit(), create_thought() (and therefore remember()) commits the thought and its journal entry before auto-embedding, TTL cleanup, on_store, and automatic derivation run. A later failure leaves the source thought durable.

Consequences:

• An embedding-provider failure leaves a newly-created thought without an embedding. It remains available to direct/FTS retrieval but is absent from vector search.
• An on_store or derivation failure does not roll back the source. Depending on where derivation failed, earlier derived children may already be committed.
• Retrying create_thought() with the same ID raises ValueError. Retrying remember() with the default deduplicate=False persists a new randomly identified row; with deduplicate=True, a matching row is reused instead and its confirmation count is incremented. Neither the default call nor the deduplicated one is a repair for a failed embedding.

Use strict embedding mode when callers need the typed thought ID for repair:

from engrava import EmbeddingGenerationError


async def create_with_partial_state(store, thought):
    try:
        return await store.create_thought(thought), True
    except EmbeddingGenerationError as exc:
        persisted = await store.get_thought(exc.thought_id)
        if persisted is None:
            raise RuntimeError("embedding failure did not leave the expected thought") from exc
        return persisted, False

The False result means “the thought exists but embedding repair is required,” not “creation failed atomically.”

If the application retains the provider, it can rebuild the same payload and role-aware dispatch used by auto-embedding, then upsert the missing vector:

from engrava import RoleAwareEmbeddingProvider, ThoughtNotFoundError


async def repair_embedding(store, provider, thought_id):
    thought = await store.get_thought(thought_id)
    if thought is None:
        raise ThoughtNotFoundError(thought_id)

    essence = thought.essence.strip()
    content = thought.content.strip()
    payload = thought.content if content.startswith(essence) else f"{thought.essence}\n{thought.content}"

    if isinstance(provider, RoleAwareEmbeddingProvider):
        vector = await provider.embed_document(payload)
    else:
        vector = await provider.embed(payload)

    return await store.store_embedding(
        thought_id,
        vector,
        model_name=provider.model_name,
    )

Use the same provider/model and role prefixes that built the corpus. The store’s model lock rejects an incompatible model or dimension.

Thought updates

update_thought() commits the updated row and journal entry before re-embedding when essence or content changed, provided this call owns its outermost transaction. Nested inside the caller’s own suspend_auto_commit() window, nothing is durable yet on any path — the outermost window’s exit decides, rolling back an uncaught failure and everything it inserted. If the update does commit and query generation then fails, the row contains the new text but:

• a thought that had no embedding remains unembedded, and stays unfindable by vector search;
• a thought that had an embedding retains the old, now stale vector, and stays findable by vector search against that outdated content;
• dependent REFLECTION centroids may not yet have been rebound.

Re-read the thought and repair/recompute enrichment. Do not replay the update unchanged and assume that doing so re-embeds: an update that no longer changes essence or content does not enter the re-embedding branch.

Bulk ingest

bulk_store() is different. Source rows, journal entries, and batch-generated embeddings run inside one suspend_auto_commit() transaction. Called on its own, any exception during that phase, including a provider exception or a task cancellation, rolls the batch back, with the exceptions listed under Transaction context behavior below. The require_embedding option controls whether a failure of the provider’s embed call is wrapped as EmbeddingGenerationError; it does not change the rollback (see Embeddings for what that covers).

Nested inside a caller’s own suspend_auto_commit() window, the batch shares that outer transaction: a row error still aborts the batch’s own inserts, but only the outermost window’s exit decides commit or rollback. If the caller catches the propagated exception and the outer window then exits cleanly, the batch’s successful prefix commits along with the rest of the outer block’s work instead of being rolled back.

Automatic derivation runs only after the batch commits. A derivation failure can therefore leave the complete source batch durable and zero or more derived children durable. Reconcile sources by ID/content and use derive_existing() to complete deterministic derivation rather than blindly replaying bulk_store().

Derived records

The source is durable before automatic derivation. Outside a caller-held transaction, each derived child’s row is committed as its own unit before its embedding and DERIVED_FROM edge are completed, and a failed child undoes only its own failing step — the row and its journal entry, the edge and its journal entry, or an embedding attempt — never an earlier child’s work. Re-running derive_existing(source_id) reuses deterministic children and edges and fills missing enrichment, provided the producer obeys the deterministic content contract.

Inside a caller’s own suspend_auto_commit() window, derive_existing() does not defer — the source is already durable — but its children join that transaction instead of committing per-child: nothing is durable until the window’s own final commit, and a failure that escapes the window uncaught rolls back everything the window holds, earlier children included — unless the connection is already quarantined, in which case the window attempts no rollback of its own (see Terminal connection quarantine below). A raw BEGIN differs again: a child’s own successful step still commits the shared transaction as soon as that step succeeds.

Transaction context behavior

suspend_auto_commit() groups writes on one store connection. A normal exit of the outermost call commits once; an exception that escapes that outermost block — including task cancellation — causes a rollback of the transaction and is re-raised, unless the connection is already quarantined, in which case no rollback is attempted (see Terminal connection quarantine). A nested call’s own exit neither commits nor rolls back anything by itself; only the outermost one decides, so an exception raised inside a nested window but caught by your own code before it reaches the outermost block still lets that block commit normally.

async def store_atomically(store, first, second, link):
    async with store.suspend_auto_commit():
        await store.create_thought(first)
        await store.create_thought(second)
        await store.create_edge(link)

Operational rules:

• Let an error escape the async with block. Catching it inside and then leaving normally tells the context manager to commit.
• suspend_auto_commit() holds a task-reentrant lock for its whole duration: a different task’s write on the same store instance now waits for the window to close instead of joining its transaction — bounded past write_lock_acquire_timeout_seconds, where it raises WriteLockTimeoutError — and nesting on the same task is supported (only the outermost call commits or rolls back). Drive the window from one task at a time regardless: a second task’s write is safe from corruption, but it still simply waits, so interleaving unrelated writes through a long-running window just serialises them for no benefit. See Concurrency for the full contract.
• Automatic on-store derivation is skipped inside a caller-held transaction. After commit, invoke derive_existing() explicitly for sources that need it.
• The rollback branch catches BaseException at every nesting level, so asyncio.CancelledError is caught the same way an ordinary exception is; it rolls back the transaction only when it escapes the outermost block uncaught, exactly like any other error. One exception is a cancellation that lands during the outermost window’s own commit(): the commit runs after the body has already returned, outside the rollback branch, so it may or may not have completed by the time the cancellation propagates. Treat the outcome as indeterminate: reconcile the affected IDs from a new store before repeating anything, and do not let a later commit decide the fate of an indeterminate transaction.

For the supported writer topology and task boundaries, see Concurrency.

SQLite lock contention

SQLite permits many readers but only one writer. Stores opened through SqliteEngravaCore.from_config() or EngravaManager set PRAGMA busy_timeout=5000, so a competing connection waits for up to five seconds before surfacing database is locked. A manually supplied connection keeps its own pragma settings.

create_thought(deduplicate=True), get_or_create(), upsert_by_hash(), bulk_store(deduplicate=True) and the revision-guarded updates (update_thought, restore_thought, update_edge, update_action) convert lock contention into WriteContentionError, carrying operation and attempts. The failed attempt wrote nothing, so retrying that call outright is safe. Other write paths can still surface the raw OperationalError from the table above.

Prefer removing contention over adding a generic retry loop:

1. Keep write transactions short.
2. Serialize writes per store/process.
3. Partition independent writers into separate database files with EngravaManager.
4. Increase busy_timeout only when the added tail latency is acceptable.

busy_timeout is a latency knob, not a correctness one. It decides how long a connection waits for the file lock — except a content-changing update_thought(), which can fail immediately without waiting; it does nothing about two stores interleaving inside one of engrava’s read-modify-write operations, which is why only one store may write a given file (Concurrency).

If an operation is known to be read-only and repeatable, bounded retry is reasonable. This example deliberately retries journal verification, not an arbitrary write:

import asyncio
import sqlite3


async def verify_journal_with_lock_retry(store, attempts=3):
    for attempt in range(1, attempts + 1):
        try:
            return await store.verify_journal()
        except sqlite3.OperationalError as exc:
            locked = "locked" in str(exc).lower() or "busy" in str(exc).lower()
            if not locked or attempt == attempts:
                raise
            await asyncio.sleep(0.1 * attempt)
    raise AssertionError("unreachable")

Do not wrap every Engrava method in this helper. After an ambiguous write error, query the durable state from a healthy connection and choose a method whose repetition is known to converge.

Self-healing search and counters

Search has two intentional degradation paths:

• If the primary FTS5 MATCH raises OperationalError, Engrava increments fts_match_failure_count, logs a warning, sanitizes the original query into bare mode, and retries once. If that fallback also fails, the FTS arm returns no results; hybrid search can still use its other active arms.
• An all-zero, NaN, or infinite query vector of the store’s declared length has no cosine direction. search_similar() increments vector_arm_degradation_count and returns []. A wrong-length vector is not degraded: the dimension check runs first and raises VectorDimensionMismatchError — including for an empty vector, which is a length-0 mismatch whenever the store declares a dimension. Only when the store declares no dimension does the dimension check never fire, so an empty vector falls through to the same degenerate-vector handling as the other two shapes. See API Reference for the exact ordering.

The counters are cumulative per store instance and reset when a new store is constructed. Alert on growth/deltas, not merely on a non-zero lifetime value:

async def recall_with_degradation_flags(store, query):
    fts_before = store.fts_match_failure_count
    vector_before = store.vector_arm_degradation_count

    result = await store.recall(query)

    return (
        result,
        store.fts_match_failure_count > fts_before,
        store.vector_arm_degradation_count > vector_before,
    )

An unavailable FTS5 index returns an empty FTS arm. Failure to load the optional sqlite-vec extension at store setup logs a warning and falls back to exhaustive NumPy search; it does not increment vector_arm_degradation_count. Runtime backend failures outside the two cases above are not covered by a blanket self-healing guarantee.

See Search for the result-level signals.

Terminal connection quarantine

ConnectionQuarantinedError means Engrava could not prove that a rollback or unwind it depends on completed, or that the connection’s last operation ever reported. The old connection may have an indeterminate open transaction, so the store revokes all new operations and replaces its internal connection with a failing proxy. This state never clears. Causes include:

• the savepoint unwind of a guarded write (update_thought, restore_thought, update_edge, update_action, a derived child’s row insert or edge insert) failed;
• the savepoint unwind of a delete_thought failed, including a delete made through the TTL delete strategy or hygiene garbage collection — an ordinary error such as a trigger veto can cause this, not only a cancellation;
• close() gave up waiting on the physical close after close_timeout_seconds — see Deployment.

The call that triggered the quarantine can surface its original or derivation error (or a cancellation that outranks it); a close() whose bound expired raises ConnectionQuarantinedError itself, unless a cancellation of that call is pending. Once quarantine is installed, the next public operation that reaches the connection raises ConnectionQuarantinedError — an operation that returns before touching the connection is unaffected. A second close() is the exception: its outcome depends on the state of the abandoned physical close and on whether that call is itself cancelled: see Deployment.

Close the old store, open a new one, and inspect durable state before deciding what to repeat:

from engrava import ConnectionQuarantinedError, SqliteEngravaCore


async def replace_and_reconcile(store, config_path, source_id):
    try:
        source = await store.get_thought(source_id)
    except ConnectionQuarantinedError:
        await store.close()
        store = await SqliteEngravaCore.from_config(config_path)
        source = await store.get_thought(source_id)

    if source is not None:
        await store.derive_existing(source.thought_id)
    return store

The caller owns the returned replacement and should close it during normal shutdown.

Do not retry against the quarantined instance, including through its journal. Quarantine detaches the real connection and schedules its best-effort close even when the store was built with the manual constructor. Any external reference to that same connection must still be treated as revoked and discarded.

Idempotency and convergence

Use these contracts narrowly:

When an API is not listed as idempotent, assume that retry requires an application-level operation key and a read-back/reconciliation step.

For the exhaustive package reference, see the OSS error-handling.md docs.