Known Limitations

This document covers platform-specific notes, constraints, and known issues. For the consolidated threat and trust-boundary model, see Security.

Search is unscoped by default (multi-tenant caveat)

search_hybrid() / recall() / search_similar() span the entire store by default — there is no implicit per-user or per-session boundary. If you keep multiple tenants’ thoughts in one database file, an unfiltered search can return another tenant’s memories.

Two supported ways to isolate:

  • A file per tenant (the strongest boundary) — one store per tenant, managed via EngravaManager. Isolation is then the file boundary itself.
  • Scoped retrieval within one file — pass filters= / visibility= to search_hybrid() or recall() to constrain results to a metadata scope (e.g. an owner or session key). The public search_similar() and search_fts() methods do not accept those metadata filters. See Scoped retrieval.

Choose file-per-tenant when tenants must never share a file; use scoped filters for soft, in-file partitioning.

Dreaming / consolidation: mechanism, not a proven retrieval lift

The built-in DreamingExtension’s default signals make no LLM calls, and its reflection content is built by a deterministic structural function (clustering, centroids, keyword counts) — but a custom signal registered via custom_signals runs whatever code it contains, including a call to an LLM, if you write one that does. Consolidation is: scoring, and, conditional on gates, thresholds, and caps, promotion to priority boost, association edges, and reflections. With a fixed store, configuration, cycle, embedding inputs, and deterministic custom signals, its result is reproducible. Those are real mechanical ranking effects, but Engrava makes **no claim that enabling dreaming improves retrieval accuracy. The v0.5/v0.6-candidate frozen synthetic snapshot measured aggregate recall@5 at 0.80 with Dreaming off and 0.70 with it on, even though the separate curated release gates passed. Enable it for the cognitive-hygiene mechanics it provides, not for an expected accuracy gain. See Dreaming and the current benchmark evidence.

Embedding-provider conformance is not checked at construction

EmbeddingProviderProtocol requires public dimension and model_name members, but the protocol is structural: a store accepts any object passed as embedding_provider= and only discovers a missing member when it reads one. A provider that keeps the dimension privately (self._dimension) with no public property therefore constructs cleanly and fails later — on the first vector search that has to ask the provider — with EmbeddingProviderContractError naming the class and the member. A store whose vector backend is sqlite-vec takes the dimension from its dimension-typed vec0 table instead, so that search path does not ask the provider for it; verify_embedding_model() asks regardless.

This is deliberate. Construction does not read the member, and a store that never searches by vector — and never calls verify_embedding_model() — never needs it, so failing at construction would reject stores that work today. Call verify_embedding_model() after construction when you want the failure at startup instead.

Note that the requirement is not new: dimension has been a required member of the protocol since the first public release. See Upgrade.

Validity intervals cannot be inverted

The bi-temporal valid_from / valid_until bounds on a ThoughtRecord or EdgeRecord describe a forward interval. When both bounds are set, engrava rejects an inverted interval (valid_from strictly after valid_until):

  • On write — construction fails with a ValidationError; the store’s update path re-validates the whole record, so a change that would invert a stored interval is refused; and invalidate_thought() / invalidate_edge() reject a valid_until earlier than the record’s stored valid_from.
  • On read — the invariant lives in the domain model, and the store reconstructs that model whenever it hydrates a full record (get_thought(), get_edges()). A row that became inverted out of band — for example one written by an older engrava build before this validation existed, or edited directly in the database file — therefore raises a ValidationError when it is read back that way, rather than silently returning a corrupt interval. Ranked search (search_hybrid(), recall(), search_similar(), search_fts()) returns thought ids and scores without reconstructing the record, so an inverted row can still surface in a ranked result; the error only appears once you hydrate that id. This is a deliberate fail-loud choice for the reads that build the full record. If you are upgrading a database that may contain such rows, repair them (set the offending bound to NULL, or correct the order) before reading.

Bounds are compared as UTC-normalised instants, so differing offsets are reconciled before the check. Two cases are not inversions and remain accepted:

  • Equal bounds (valid_from == valid_until) — a zero-length interval is a legitimate instantaneous fact.
  • An open bound (valid_from or valid_until is None) — the interval is open on that side (see The bi-temporal model).

There is no way to store a fact “valid from June to March”; express an open-ended fact with a None bound instead.

macOS SQLite Extension Loading

macOS ships with a system SQLite that has extension loading disabled by default. If you use the vec extra (pip install engrava[vec]), you may encounter:

sqlite3.OperationalError: not authorized

Workaround: Install Python via Homebrew or pyenv, which links against a full-featured SQLite build:

brew install python@3.12
# or
pyenv install 3.12

aiosqlite Proxy Architecture

engrava uses aiosqlite which runs SQLite on a dedicated background thread and proxies calls via asyncio. This has implications:

  • Connection objects should not be shared across event loops.
  • Long-running SQL statements block the background thread: it runs one call at a time, so a slow statement delays every other call on that connection. An open transaction alone does not — while a suspend_auto_commit() window sits idle, other tasks’ reads still run. What the window does hold is the store’s write lock, so a different task’s write waits for it — for at most write_lock_acquire_timeout_seconds — and raises WriteLockTimeoutError after that, rather than blocking on the worker thread itself. Keep transactions short.
  • WAL mode is used by default for concurrent read access. Writes are serialized by SQLite’s single-writer lock.

sqlite-vec Pre-v1 Status

The sqlite-vec extension is pre-v1. engrava pins >=0.1.0,<0.2.0 to avoid breaking changes. When sqlite-vec reaches 1.0, the pin will be relaxed.

Without the vec extra, engrava falls back to a brute-force cosine scan done with NumPy. Cost grows with the size of the eligible embedding population, not a fixed cliff. For larger collections, run pip install 'engrava[vec]' to use the compact compiled vec0 backend, but note that the pinned sqlite-vec 0.1.x line still performs an exhaustive linear KNN scan. It stores vectors in a dedicated vec0 table rather than loading them into NumPy; it is not an approximate or sub-linear index, and relative latency and memory use should be measured for your deployment rather than assumed. Measure your own p95 latency and see Performance.

FTS5 Availability

FTS5 is included in the standard SQLite build since version 3.9.0 (2015). Most Python distributions include it. If FTS5 is not available, creating the schema fails: ensure_schema() — and so opening a new store — raises sqlite3.DatabaseError. Separately, on a store that is already open, search_fts() degrades rather than raising when the thought_fts table does not exist: it returns an empty list.

To verify FTS5 support:

import sqlite3
conn = sqlite3.connect(":memory:")
conn.execute("CREATE VIRTUAL TABLE test USING fts5(content)")
conn.close()
print("FTS5 is available")

Concurrent Write Safety

SQLite supports one writer at a time. With WAL mode, readers do not block writers and vice versa. aiosqlite marshals every call onto one background thread, so concurrent tasks’ statements do not run at the same time.

That is statement-level serialisation, and by itself it would not be operation-level safety — but a task-reentrant write lock closes that gap in-process. When they make a real change, update_thought, restore_thought, update_edge, and update_action read the row, apply the change in memory, then write; a changed upsert_by_hash hit delegates to update_thought. Two genuinely concurrent tasks sharing one store instance no longer corrupt each other’s read-modify-write this way. Every core row also carries a revision column that update_thought/restore_thought/update_edge/update_action each check and increment atomically (a changed upsert_by_hash hit goes through update_thought) — update_edge and update_action carry this guard for the first time — so a competing write that does land in a real gap (a same-task nested call reached through a caller-owned hook, or a write from a second store on the same database file) makes the guard match no row and raises StaleDataError instead of silently overwriting it, whether or not the two edits share a field. See Concurrency for the full contract.

Only one store may write a given database file. The locks that order engrava’s own operations live on the store instance, so a second store — in another process, or a second connection in this one — is outside all of them. WAL and busy_timeout keep the file intact under that topology, and the revision guard above keeps a lost update from landing silently; neither makes multiple concurrent writers on one file a supported topology.

The full contract, the guarantees that do hold, and the idioms that close the gap are in Concurrency. For multi-service setups via EngravaManager, each service has its own database file with independent locking — that is the supported way to run independent writers.

Separately: if the store ever quarantines its connection (an internal safety response to an indeterminate transaction), quarantine revokes admission synchronously: every new operation that reaches the connection then fails fast with ConnectionQuarantinedError, so no write can flush an orphaned transaction. An operation that returns before touching the connection is unaffected — a degenerate query vector short-circuits search_similar() to [] before it reaches the connection, and search-latency recording lives entirely in an in-memory buffer — so quarantine is not a universal fail-fast over every store or journal call. It does not retract an operation already in flight — a reader admitted just before quarantine may complete a possibly-stale read on the pre-quarantine connection (never a commit). During quarantine the physical connection close is a detached, best-effort cleanup.

Embedding Dimension Consistency

All embeddings for a given database must use the same dimensionality. Engrava validates the stored model identity against the model name and dimension being written or verified, and raises EmbeddingModelMismatchError on a mismatch. The first store_embedding() or explicit verify_embedding_model() call on a fresh database locks that identity in; every later call is compared against it too — there is no instance-level cache that lets a call skip the comparison once one has passed, so a store instance handed a different model, dimension, or document prefix on a later call is refused just as reliably as one constructed fresh with the mismatched provider. store_embedding() runs this comparison before it writes the base row or the vector index, so a mismatch on any of the three — model name, dimension, or document prefix — is caught the same way regardless of vector backend: there is no write-time path left where a wrong dimension reaches the vec0 table or a plain numpy corpus unchecked. The one exemption: an embedding recorded under the internal centroid tag — a REFLECTION’s own computed centroid, not a configured provider’s output — skips this comparison and never changes the locked identity either way.

A deliberate CLI re-embed is available while restoring a snapshot into a configured direct database or service. Pass --config: direct mode uses the top-level embeddings provider, while service mode prefers its per-service override and otherwise uses the top-level provider. Without a configured provider, use --skip-embeddings or preserve the source vectors. See CLI restore.

Query-vector dimension mismatch

Distinct from the DB-wide Embedding Dimension Consistency above (which is about the stored corpus all sharing one dimension): this is about a single query vector whose length does not match the store’s embedding dimension.

search_similar() (and the vector arm of search_hybrid()) computes cosine similarity between the query vector and stored embeddings, which is only defined when both share a dimension. A wrong-length query vector is a caller-contract violation, so it is raised loudly as VectorDimensionMismatchError (carrying expected and actual) rather than silently returning an empty result. The check is dimension-only and runs before the degeneracy check, so a wrong-length all-zero vector is a dimension error, not a degenerate-vector degradation.

Behaviour change — catch the typed error. The wrong-dimension case now raises VectorDimensionMismatchError (a subclass of EngravaError, not of ValueError). Any caller that previously caught a plain ValueError around a vector search must catch VectorDimensionMismatchError (or EngravaError) instead.

Archived thoughts and default retrieval

Archived thoughts (lifecycle_status = ARCHIVED) are excluded from default retrieval on every ranked read — search_hybrid(), recall(), search_fts(), and search_similar(). This is the retrieval side of Forgetting — an opt-in, off-by-default hygiene loop — but the exclusion applies to any archived row, whether or not that loop is enabled: a forgotten (archived) thought stops surfacing without being deleted.

  • Behaviour change: previously an ARCHIVED thought still appeared in search. It no longer does. This also affects the TTL archive strategy — a TTL-expired, archived thought now drops out of default search too.
  • Reversible, with one exception: restore_thought() re-activates a row, and include_archived=True re-admits archived rows for a single call without restoring them — except a retired REFLECTION (one whose source cluster left ACTIVE), which a separate freshness rule keeps excluded even under include_archived=True; use restore_thought() to bring it back.
  • Not applied to counts/listing: count_thoughts() and list_thoughts() still include archived rows lifecycle-neutrally (they are not ranked retrieval), but apply their own default expiry exclusion — an archived-and-expired row needs include_expired=True too. Filter on lifecycle_status yourself if you need archived rows excluded there.

Deletion on a database that has not been migrated

A database below the core-12 schema (the migration that adds foreign-key ON DELETE CASCADE to edge, embedding and action) has no cascade to remove a deleted thought’s child rows. What the code does about that:

  • Child rows are deleted explicitly. delete_thought, the TTL delete strategy and hygiene GC each issue their own deletes for a thought’s edge, embedding and action rows, in the same savepoint as the parent delete, which runs first, instead of relying on the cascade.
  • Reconciliation only backfills for an owner that matches a thought. The reconcile that backfills the vector index on a sqlite-vec-enabled open backfills a vector only for an embedding row whose owner id matches a row in thought.
  • The purge is the same join. The sweep that removes orphaned index rows treats a vector as orphaned when its embedding row is gone or its owner id matches no row in thought.
  • Search resolves through a join that requires the thought row. The rowid-to-id resolution and the post-search eligibility check confirm that a thought row with that id exists (and is otherwise eligible); an id that matches no thought row is dropped.

Historical residue. On a pre-core-12 schema, a delete made by an older engrava build left the thought’s embedding row behind, since nothing removed it. Until it is purged, that row stays in the file with its vector, and reconciliation matches it to a thought by owner id alone, so a new thought that reuses the deleted id can be matched to it. engrava migrate (or opening the store through from_config(), which calls ensure_schema()) runs the core-12 step, which recreates the three child tables with ON DELETE CASCADE and purges the orphan rows that had already accumulated.

engrava --db engrava.db migrate

engrava gc refuses instead of running on a database below head. gc and restore now exit non-zero and name engrava migrate rather than proceeding on an unmigrated database. A read-only command (info, verify, export, snapshot, recall on --db or the default path, and a query that parses as a read) still runs against a behind schema, but warns on stderr that the schema is behind. recall under --config refuses instead, and remember and link refuse too; see CLI → Schema-version checks.

Maximum Database Size

SQLite supports databases up to 281 TB (theoretical). In practice, engrava has been tested with databases up to ~10 GB (millions of thoughts) without issues. Performance depends on index coverage and query patterns.

HybridSearchResult.backends_used Is an Open Set

backends_used is a frozenset[str] that may grow as new scoring signals are added (e.g. "priority" was added in v0.3.0). Do not compare it with exact equality (== {"fts5", "vector"}). Use subset checks instead:

assert {"fts5"} <= result.backends_used  # preferred

For the exhaustive package reference, see the OSS known-limitations.md docs.