Performance & scaling

How Engrava behaves as data grows, where the limits are, and the two levers that matter most: the vector backend and batched writes. The numbers that matter for your workload depend on corpus size, embedding dimension, query mix, and hardware — measure on your own data rather than trusting a single headline figure. This page explains what drives cost so you know what to measure.

For the dreaming quality benchmark (does consolidation help retrieval), see the OSS Benchmarks docs. For the hard platform constraints, see Known Limitations.

Where the cost is

A query touches up to five signals; each scales differently:

When the vector arm is active, its exhaustive scan over the eligible candidate set can dominate large-store queries; both backends perform exhaustive search over their respective eligible/indexed candidate sets, not necessarily every stored embedding. Actual dominance depends on which signals are enabled, candidate counts, query shape, and hardware.

The brute-force ceiling (and how to pass it)

Without the vec extra, vector search is a brute-force cosine scan done with NumPy: every search_similar / search_hybrid query performs an exhaustive scan over the eligible stored embeddings (live, unexpired, and matching any active filter). This is simple and dependency-free. Latency grows with the size of the eligible embedding population; measure your own workload and switch when it stops meeting your latency budget.

The fix is the sqlite-vec backend, which stores vectors in a dedicated, compact vec0 virtual table. In the pinned sqlite-vec 0.1.x line a vec0 query is still an exhaustive k-nearest-neighbour scan — not an approximate or sub-linear index — but over a tightly packed, chunked columnar store rather than loading vectors into NumPy. Relative latency and memory use should be measured for your deployment rather than assumed. FTS5 scales independently of the vector backend.

Cost grows with the eligible embedding population — there is no fixed cliff — see Known Limitations → sqlite-vec. Measure your own p95 query latency and switch when it stops meeting your budget.

Filtered search and the vector fast path

search_hybrid() / recall() with a filters= or visibility= argument scope the ranked query (see Scoped retrieval). There is a performance consequence worth knowing.

An active filter disables the sqlite-vec fast path. With no filter, the vector arm runs on the compiled vec0 table. With a filter, the vector arm instead runs an exhaustive NumPy cosine over the rows that match the filter — because the filter is an arbitrary metadata predicate that vec0 cannot apply before it picks its k nearest neighbours. The filtered cost is:

• ~O(N_total) for the scan that selects the eligible rows, plus
• O(N_eligible × dims) for the cosine over those eligible embeddings.

Note the first term: a selective filter shrinks the cosine work, but the eligibility scan still visits the whole embedding-bearing population. Both the filtered (NumPy) and unfiltered (vec0) paths are already exhaustive in the pinned sqlite-vec 0.1.x line (see the brute-force ceiling). Filtering changes the execution path and adds SQL eligibility work; depending on selectivity and constants, total latency may increase or decrease, while worst-case corpus scaling stays linear either way.

When it matters. A selective filter (one project, one session, one user) keeps the eligible set small — the filtered path stays acceptable, and can be faster than scanning the whole vec0 table when the filter is highly selective (the eligibility scan still runs, so this is not guaranteed). The slow corner is a broad filter on a large store at the same time — for example visibility= admitting public-or-mine where “mine” is most of the store, on a base approaching the brute-force ceiling. If that is your workload, prefer one store per tenant via EngravaManager (see scoping & multi-tenancy): each tenant’s store is small, so the exhaustive scan stays small — and that is real isolation, which a metadata filter is not.

Switching to sqlite-vec (incl. migrating an existing database)

The migration is designed to be turnkey: your embeddings already live in the embedding table, so switching backends only builds and backfills the vec0 vector table — you do not re-embed anything.

1. Install the extra.

pip install 'engrava[vec]'

2. Set the backend in your config.

extensions:
  vector:
    backend: sqlite-vec      # default is "numpy"
    dimension: 384           # must match your embedding model

3. Open the store with from_config. On open, Engrava creates the vec0 virtual table and backfills every existing embedding into it automatically (idempotent — safe to run repeatedly). From then on, new writes keep the index in sync.

from engrava import SqliteEngravaCore


async def main(query_vector: list[float]) -> None:
    # from_config wires the vector backend; the index is created and
    # back-filled on open. A plain SqliteEngravaCore(conn) constructor stays
    # on numpy.
    async with await SqliteEngravaCore.from_config("engrava.yaml") as store:
        result = await store.search_similar(query_vector, top_k=5)

That’s the whole migration — no manual re-index step, and no re-embedding, because the vectors are reused from the existing embedding table.

Important caveats.

• Use from_config. Only the from_config path configures the vector backend. If you build the store directly with SqliteEngravaCore(conn), it stays on the numpy backend regardless of the YAML.
• Graceful fallback, not a hard error. If the sqlite-vec package is missing or the extension can’t load, Engrava logs a warning and falls back to numpy rather than crashing — so a “switch” that silently kept numpy usually means the extension didn’t load.
• macOS system SQLite blocks extensions. The most common load failure is macOS’s bundled SQLite, which disables extension loading. Install Python via Homebrew or pyenv (a full-featured SQLite build). See Known Limitations → macOS.
• Dimension must match. The index is created for a fixed dimension; it must equal your embedding model’s output. Mixing dimensions corrupts results (see Embedding Dimension Consistency).

SQLite tuning & hot-path indexes

Stores opened via from_config (and EngravaManager) come pre-tuned — you do not configure any of this:

• synchronous=NORMAL — the documented-safe companion to WAL. It cuts an fsync from every commit (lower write latency) while staying durable across an application crash; only the most recent transactions are at risk on an OS crash or power loss. (See Concurrency.)

• busy_timeout=5000 — when SQLite invokes the busy handler, this permits a wait of up to five seconds; a content-changing update_thought() can instead fail immediately during its FTS5-trigger lock upgrade.

• Four hot-path indexes back the equality filters and sort column that the common reads hit, turning what were full-table scans into index lookups:

  Index	Column	Speeds up
  idx_edge_to_thought	edge(to_thought_id)	get_edges (IN / BOTH) and the reflection-consolidation scan
  idx_embedding_owner	embedding(owner_id)	get_embedding lookups by thought
  idx_thought_updated_cycle	thought(updated_cycle)	list_thoughts recency ordering
  idx_thought_type	thought(thought_type)	thought_type equality filters

The 0.4 schema also adds valid-time indexes so the bi-temporal predicates are index-backed: valid_from, a partial valid_until index (only non-NULL upper bounds are indexed — an open upper bound is the common case and stays overhead-free), and a composite (valid_from, valid_until) range index, on both the thought and edge tables.

The indexes are created idempotently (CREATE INDEX IF NOT EXISTS) and an existing database gains them automatically through the additive 0.3 → 0.4 schema migration — zero data loss, no manual step. See the OSS Upgrade Guide docs.

Write throughput and bulk ingest

By default each mutating call commits its own transaction. For a bulk load that is the wrong granularity — one commit per row dominates wall-clock. Wrap the batch in suspend_auto_commit(). When that call is the outermost one on the store, it defers to a single commit on success and rolls the whole batch back on any error, task cancellation included — with the exceptions listed under Error handling:

async def bulk_load(store, items):
    async with store.suspend_auto_commit():
        for item in items:
            await store.create_thought(item, deduplicate=True)
    return await store.count_thoughts()

• deduplicate=True collapses identical content into one thought (bumping confirmation_count) instead of inserting duplicate rows — cheaper storage and fewer embeddings to compute. (Note the persistence default is deduplicate=False; opt in per call.)
• Keep each transaction short. A long-running SQL statement blocks aiosqlite’s background thread; an open suspend_auto_commit() window by itself does not — other tasks’ reads still run. What the window does hold is the store’s write lock, so a different task’s write waits for it, for at most write_lock_acquire_timeout_seconds, and raises WriteLockTimeoutError after that rather than joining the transaction (see Known Limitations → aiosqlite). For very large imports, batch in chunks (e.g. a few thousand rows per suspend_auto_commit() block) rather than one giant transaction, so a failure mid-import rolls back less work.
• Embedding cost dominates a bulk load when a provider is configured with auto_embed=True: each new thought is embedded on write. Pre-compute vectors and store them with store_embedding(...), use a batching local provider, or import in chunks so the encoder isn’t the bottleneck. See the Embeddings guide.

A runnable end-to-end bulk-import example lives in the migration guide.

Dreaming cost at scale

Dreaming runs off the hot path — you invoke run_consolidation() explicitly, on your own cadence; it is never run inside a CRUD or search call. A consolidation run still opens the store’s own suspend_auto_commit() windows for its writes, holding the task-reentrant write lock for their duration, so a guarded write from another task waits for it to finish (up to write_lock_acquire_timeout_seconds) rather than staying invisible to it. Its own cost scales with the number of candidate thoughts and the clustering algorithm:

• Run it periodically, not every turn (every N cycles, a cron job, or manually).
• candidates_limit bounds the promotion query and each per-type agglomerative candidate query; it is not a global upper bound on all thoughts examined during a pass, and it does not cap the LPA edge graph — keep it bounded on large stores regardless.
• Clustering has two backends via extensions.dreaming.clustering_backend ("numpy" default, or "python"); numpy is faster for the similarity math on larger candidate sets. This choice is unique to this page — see Dreaming for the algorithm tradeoffs the clustering backends implement.

Checklist: scaling Engrava

1. Missing your vector-search latency budget? Switch to sqlite-vec (above).
2. Bulk loading? Batch writes with suspend_auto_commit() and consider deduplicate=True.
3. Embedding is the bottleneck? Use a batching provider or pre-compute vectors.
4. Multi-tenant? One database file per tenant via EngravaManager keeps each store smaller and independently lockable (see the scoping section).
5. Dreaming heavy? Cap candidates_limit per query, run it on a schedule, pick the right clustering_backend.

For the exhaustive package reference, see the OSS performance.md docs.