Spec: db layer restructuring#

Status: draft

Tracking: beads epic (see bd show, filed alongside this spec)

Motivation#

The db layer works, but its correctness rests on conventions that every

callsite must independently remember. Concretely:

Db (src/db/mod.rs:48) exposes every keyspace as a public Keyspace

field plus pub inner: Arc<Database>. Any module can write raw bytes to

any keyspace with hand-built keys; db.inner.batch() appears in 27+ files.

Four access styles coexist: half-dead static async helpers (Db::get,

with insert/remove/contains_key marked #[allow(dead_code)]),

direct sync calls inside spawn_blocking closures, direct sync calls on

dedicated blocking threads, and free functions taking (batch, db, ...).

Key formats live in free functions returning Vec<u8> (src/db/keys/)

with prefix constants scattered per module — db/filter.rs maintains its

own SEP/prefix constants separate from keys::SEP. The authoritative

documentation of key formats is comments (duplicated again in AGENTS.md).

Serialization helpers exist (ser_repo_state etc.) but ~31 files call

rmp_serde directly, several deserializing RepoState raw and bypassing

the helpers (backfill/worker/process.rs:58, backfill/manager.rs:122,

backfill/worker/task.rs:187).

None of these are hypothetical risks:

1. The record-import invariant is duplicated three times. A record write

must keep in sync: records ks, blocks ks, per-collection counts,

backlinks (cfg), a StoredEvent, and a jetstream event (cfg). This is

implemented once properly in ops::apply_commit (via RecordEmitter),

then re-implemented by hand in backfill/sparse.rs (persist_sparse_backfill

builds StoredEvent + event key + jetstream::stage_event manually, and

carries its own cfg'd backlinks block) and twice more in

backfill/worker/process.rs. Divergence between these is silent index

drift — missing events, missing backlinks, wrong counts.

2. Three manual registries per keyspace. keyspace_by_name

(db/keyspaces.rs), the /debug name→type deserializer match

(api/debug.rs:99), and the /stats endpoint must each be updated when a

keyspace is added. open.rs:209 admits this with a "when adding new

keyspaces, make sure to..." comment. That comment is the bug report.

3. The counts protocol is convention-only. Every write site must

remember: stage deltas into the batch → batch.commit() →

apply_count_deltas (plus lifecycle reservation apply). The pairing is

unenforced; a missed apply drifts in-memory counts from persisted state

until restart replay. Three separate subsystems (count deltas + watermarks,

LifecycleCountBatch, per-collection record count free fns) each carry

their own call-order rules.

4. Nothing pairs a keyspace to its key format or value type. Inserting

RepoState bytes into cursors, or building a record key against the

repos keyspace, compiles fine and fails at read time — or worse, doesn't.

5. Db::open is a 260-line monolith; check_poisoned is called at some

iteration sites and not others; every key construction heap-allocates.

What is already good and stays untouched: the migration framework (frozen

versioned wire types, ordered forward migrations), the per-mode keyspace

groups (IndexerDb/StreamDb/JetstreamDb/RelayDb + OpenCx), the

RecordEmitter ZST facade pattern, the count-delta crash-safe replay design,

and the per-keyspace tuning rationale.

Design#

One principle: **a keyspace is one type, declared in one table, and its

invariant-bearing writes have exactly one owner.** Everything else derives.

1. Keyspace schemas: `Ks<S>`#

Each keyspace gets a schema type pairing name, key codec, value codec, tuning,

and debug rendering:

pub trait Schema { const NAME: &'static str; type Key: KeyCodec; type Value: ValueCodec; /// per-keyspace tuning, applied at open (moves out of open.rs). fn options(cx: &OpenCx) -> KsOptions; /// debug rendering; default impl transcodes msgpack -> json. fn render(value: &[u8]) -> Result<serde_json::Value> { ... } } /// zero-cost typed handle; Clone is cheap (Keyspace is an Arc'd handle). pub struct Ks<S: Schema> { inner: Keyspace, _s: PhantomData<S>, } impl<S: Schema> Ks<S> { pub fn get(&self, key: &S::Key) -> Result<Option<S::Value>>; /// zero-copy escape for CAS blocks and other raw-value reads. pub fn get_raw(&self, key: &S::Key) -> Result<Option<Slice>>; pub fn contains(&self, key: &S::Key) -> Result<bool>; pub fn insert(&self, txn: &mut Txn, key: &S::Key, value: &S::Value); pub fn remove(&self, txn: &mut Txn, key: &S::Key); pub fn prefix(&self, p: impl Prefix<S>) -> impl Iterator<Item = Result<(S::Key, S::Value)>>; /// greppable escape hatch (migrations, debug endpoints). pub fn raw(&self) -> &Keyspace; }

Poison checks (check_poisoned) move into the read/iterator adapters so

every access is covered instead of the current ad-hoc sites.

KeyCodec replaces the keys/ free functions with structs that are the

documented format:

pub trait KeyCodec: Sized { /// KeyBuf = SmallVec<[u8; 64]>: typical keys never touch the heap. fn encode_into(&self, buf: &mut KeyBuf); fn decode(bytes: &[u8]) -> Result<Self, KeyError>; } /// {DID}|{COL}|{RKey} — the definition is the documentation. pub struct RecordKey<'a> { pub did: &'a str, pub collection: &'a str, pub rkey: &'a str, }

This deletes the duplicate SEP/prefix constants in db/filter.rs and makes

"wrong key against wrong keyspace" a type error.

ValueCodec has three impls: msgpack via rmp_serde (blanket over

Serialize + DeserializeOwned wire types), RawBytes (CAS blocks), and

BeU64/BeI64 (cursors, counts). The ~31 scattered rmp_serde callsites

and the ser_/deser_ helper pair in mod.rs collapse into this.

2. One declarative registry#

A single table declares every keyspace, grouped exactly like the existing

mode groups:

keyspaces! { core { repos: Repos, repo_metadata: RepoMetadata, cursors: Cursors, counts: Counts, filter: Filter, crawler: Crawler, } #[cfg(feature = "indexer")] indexer { records: Records, blocks: Blocks, pending: Pending, resync: Resync, resync_buffer: ResyncBuffer, } #[cfg(feature = "indexer_stream")] stream { events: Events, ... } #[cfg(feature = "jetstream")] jetstream { ... } #[cfg(feature = "relay")] relay { relay_events: RelayEvents, ... } #[cfg(feature = "backlinks")] backlinks { backlinks: Backlinks } }

The macro generates:

the Db field groups (matching today's IndexerDb etc. — this extends the

existing seam, it does not replace it),

open-time creation with S::options(cx) tuning (guts the open.rs monolith),

keyspace_by_name(&str),

fn all() -> impl Iterator<...> for /stats,

/debug rendering via S::render.

Adding a keyspace becomes: write the schema type, add one row. Forgetting the

row means the keyspace doesn't exist at all — no silent partial registration.

The open.rs:209 comment dies.

3. Visibility is the enforcement mechanism#

Registry-generated fields are private to db::. Read access is exposed

per keyspace where legitimately needed; the invariant-bearing keyspaces

(records, blocks, counts, events, jetstream_events, backlinks)

get no public write access at all. The only way to mutate a record is the

op type below. raw() escape hatches exist for db::migration and the

/debug write endpoint — both inside or routed through db::, both

greppable.

This is the "use the type system to encode correctness constraints" rule

applied with module privacy, which is the only enforcement Rust actually

gives us.

4. One write path for records: `RecordTxn`#

Layered write handles:

/// typed batch over fjall::Batch; plain writes (cursors, filter, pds_meta...). pub struct Txn<'db> { batch: fjall::Batch, db: &'db Db } impl Txn<'_> { pub fn commit(self, mode: PersistMode) -> Result<()>; } /// the ONLY path that may mutate records/blocks/counts/events/backlinks. pub struct RecordTxn<'db> { txn: Txn<'db>, counts: CountDeltas, lifecycle: LifecycleCountBatch, emitter: RecordEmitter, // existing ZST facade, reused as-is } impl RecordTxn<'_> { pub fn put_record(&mut self, did, coll, rkey, cid, block, action) -> Result<()>; pub fn delete_record(&mut self, did, coll, rkey, prev_cid) -> Result<()>; pub fn update_repo_state(&mut self, did, f: impl FnOnce(&mut RepoState)) -> Result<()>; /// stage deltas -> batch.commit -> apply_count_deltas -> lifecycle apply. /// the ordering exists exactly once, here. pub fn commit(self, mode: PersistMode) -> Result<()>; }

put_record/delete_record own the full invariant: records ks, blocks ks,

per-collection counts, backlinks (one cfg site, inside), StoredEvent

emission and jetstream staging via RecordEmitter.

ops::apply_commit, backfill/sparse.rs::persist_sparse_backfill, and both

backfill/worker/process.rs sites route through it. The three hand-rolled

StoredEvent constructions and sparse.rs's private backlinks cfg block are

deleted, not deprecated.

Drop-without-commit is the only failure mode, and it is exactly the case

the existing count-delta restart replay already handles. No new crash-safety

machinery — the existing design, minus the per-callsite ceremony.

This is not a transaction abstraction over fjall; batches are already atomic.

It is the existing protocol given a shape that cannot be half-followed.

5. Blocking discipline#

One rule: Ks methods are sync. Async contexts use one wrapper:

impl Db { /// spawn_blocking with the Db handle; the one async<->sync seam. pub async fn run<T>(&self, f: impl FnOnce(&Db) -> T + Send) -> T; }

Dedicated blocking threads (ingest/indexer/shard.rs, backfill retry worker)

call sync methods directly, unchanged. The static async helpers on Db

(get/insert/remove/contains_key, mostly dead code today) are removed.

Non-goals#

No storage-engine abstraction, no fjall replacement, no mock-store trait

for tests (fjall in a tempdir is fast; tests keep using the real thing).

No runtime mode selection. Modes stay compile-time; the registry table

keeps cfg at the composition root per the feature-gating conventions.

The migration framework is untouched. Migrations deliberately read old

formats and keep raw keyspace access via raw().

No change to on-disk formats, key layouts, or wire types. This is a

code-shape refactor; a fresh build must open an existing database and

produce byte-identical writes.

Phasing#

Each phase lands independently, warning-free through

nu tests/feature_matrix.nu (all 13 combos) and nu tests/run_all.nu.

Within a phase, old and new APIs may coexist; at each phase boundary the old

path is deleted — no lingering dual conventions.

1. Schema + registry. Schema, Ks<S>, ValueCodec, the keyspaces!

table; port every keyspace; derive keyspace_by_name, /stats, /debug

from it; move tuning out of open.rs. Pure mechanical, no behavior change.

(Fields stay pub(crate) during this phase.)

2. Key codecs. Key structs replace keys/ free functions; delete the

duplicate separator/prefix constants in db/filter.rs; poison checks fold

into accessors/iterators.

3. RecordTxn. Route apply_commit, sparse backfill, and backfill worker

processing through put_record/delete_record; delete the duplicated

StoredEvent/backlinks/counts blocks; flip invariant keyspaces to private.

4. Counts + cleanup. Fold count staging/apply and lifecycle reservations

into Txn/RecordTxn commit; remove loose apply_count_deltas callsites;

Db::run replaces the static async helpers; decompose what remains of

open.rs.

Ordering: 1–2 are low-risk mechanical enablers; 3 is the highest-value

correctness fix and depends on them; 4 rides on 3's types.

Verification#

Per phase: nu tests/feature_matrix.nu, nu tests/run_all.nu --skip-creds.

Phase 3 specifically (invariant-touching): repo_sync_integrity.nu,

backlinks.nu, stream_live_backfill.nu, stream_cursor_replay.nu, and

repo_count_resync.nu with credentials — these are exactly the tests that

observe the record-import invariant from outside.

Phase 4: repo_count_resync.nu again (count drift is its purpose), plus

ephemeral_ttl_*.nu for event pruning paths.

Open an existing pre-refactor database directory and verify /stats and

/debug/iter parity against a pre-refactor build (no format changes means

byte-identical output).

Risks#

Callsite breadth. Dozens of files touch keyspaces directly. Mitigation:

phase boundaries are keyspace-by-keyspace mechanical ports; the compiler

drives (private fields = exhaustive callsite list as errors).

Iterator decode overhead. Typed iterators decode per item; hot scans

that only need raw slices use prefix_raw variants. CAS block reads keep

zero-copy Slice via get_raw.

Monomorphization. Ks<S> per keyspace is ~20 instantiations of small

methods; negligible.

RecordTxn shape vs. sparse backfill. Sparse import builds records from

proof CARs, not commit diffs; put_record must accept a pre-parsed block

without forcing a re-parse. The API above takes (cid, block) raw for this

reason — verify during phase 3 that sparse's zero-copy Bytes::slice path

survives.

hydrant