# db layer everything hydrant stores goes through `src/db`. it opens one fjall database, gives each keyspace a typed handle, and owns the writes that have to keep several keyspaces in step, so the rest of the code can't half do them. the record keyspaces it guards are described in [record storage](https://hydrant.klbr.net/specs/block-gc/index.md). ## keyspaces and their schemas each keyspace is a type implementing `Schema` in `src/db/schema.rs`: ```rust pub trait Schema { const NAME: &'static str; const TRAIN: Option = None; fn options(cx: &OpenCx) -> KeyspaceCreateOptions; fn render_value(value: &[u8]) -> Value { ... } fn parse_debug_key(key: &str) -> Option> { ... } fn render_debug_key(key: &[u8]) -> String { ... } } ``` `NAME` is the name on disk and must never change. `options` is the keyspace's tuning (memtable size, block sizes, compression per level, bloom filters), with the reasons in comments next to each setting. `TRAIN` says whether `POST /db/train` builds a zstd dictionary for it. the render and parse functions are what `/debug/*` uses to show keys and values. `Ks` is the handle a `Db` holds for each keyspace: the fjall `Keyspace` plus the database's poison flag. its `get`, `contains_key`, `insert` and `remove` have the same signatures as `Keyspace`'s and check for poison. anything else (`range`, `prefix`, `iter`, batch writes) goes through a `Deref` to the raw `Keyspace`. `raw()` and `keyspace()` hand out the raw handle on purpose, for code that takes a `&Keyspace` like the redaction and erase helpers in `db::indexer`. `Ks` doesn't know its key or value types yet, so writing `RepoState` bytes into `cursors` compiles fine and only fails when something reads it. key formats are free functions in `src/db/keys/` returning a `Vec`, and values are msgpack through helpers like `ser_repo_state`, though plenty of code calls `rmp_serde` itself. ## the registry `src/db/registry.rs` lists every keyspace once, as a `Db` field path and its schema, with cfg on the row when it belongs to a feature: ```rust registry! { repos => schema::Repos, ... #[cfg(feature = "indexer")] indexer.heads => schema::Heads, #[cfg(feature = "indexer")] indexer.history => schema::History, ... } ``` the macro builds everything that has to name all the keyspaces: `Db::keyspace_by_name`, `Db::all_keyspaces` (which `/stats` and `POST /db/compact` walk), the `/debug` helpers `debug_value`, `debug_parse_key` and `debug_render_key`, `trainable()` for dictionary training and dictionary loading, and `names()`. the registry doesn't create the keyspaces. each mode group opens its own with `Ks::open(cx)`, which writes the name down, and at the end of `Db::open` the opened names have to equal `registry::names()` or open fails with "keyspace registry drift" (`assemble_keyspaces_and_verify` in `src/db/open.rs`). a keyspace with no row, or a row with no keyspace, fails at startup instead of going missing from `/stats` or `/debug`. to add a keyspace: write its schema type, add the field and its `Ks::open` line to its group (or to `Db` and `assemble_keyspaces_and_verify` if every mode has it), and add its registry row. ## per-mode groups `src/db/keyspaces.rs` declares the keyspaces that only some builds have, one struct per feature: `IndexerDb` (`indexer`), `StreamDb` (`indexer_stream`), `JetstreamDb` (`jetstream`) and `RelayDb` (`relay`). `Db` holds each one behind a single cfg, next to the keyspaces every mode has (`repos`, `repo_metadata`, `cursors`, `counts`, `filter`, `crawler`) and `backlinks`. a group's `open` opens its keyspaces, and the ones with a stream also hold its broadcast channel and its position counters, which their `init` resumes after the last stored event. `OpenCx` carries what a keyspace needs to open (the database, the config, the compression choice and the opened names), and its fields are private to `db` so nothing else can open a keyspace. ## who can write what module privacy is how the db layer keeps writes that have to happen together in one place. - `IndexerDb.heads`, `history`, `redactions`, `pending` and `pending_dids` are `pub(super)`, so only code in `db` can touch them. everything else reads through `head`, `head_prefix`, `head_range`, `history_range`, `is_pending`, `pending_iter` and `pending_did_range`, writes records through `RecordTxn`, and queues backfills through `stage_pending_insert` and `stage_pending_remove`, which always write both queue keyspaces in one batch. - `StreamDb.events` is `pub(super)`. events reach it only through a transaction's outbox, see [committing](#committing). - `stage_head` writes a head directly and only exists for test fixtures and the debug and bench seeding endpoints (it's compiled under `test` or `indexer_stream`). the rest isn't enforced yet. `Db.repos`, `Db.counts` and `Db.inner` are `pub`, and the event keyspaces of jetstream and relay, `StreamDb.event_bodies` and `Db.backlinks` are `pub(crate)`. so `repos` rows are only written through `Txn::put_repo` and `Txn::remove_repo` by convention, and code all over the crate still opens its own `db.inner.batch()`. ## transactions `Txn` (`src/db/txn.rs`) is one atomic write: a fjall batch plus everything that has to change when that batch commits. ```rust pub(crate) struct Txn<'db> { pub(crate) batch: fjall::OwnedWriteBatch, pub(crate) db: &'db Db, pub(crate) counts: CountDeltas, pub(crate) outbox: Outbox, ... } ``` - `counts` holds deltas for the global counters. they're written into the batch as entries in the `counts` delta log and only applied to the in-memory counts after the batch commits, so the memory never runs ahead of the disk. a restart replays the log past the last checkpoint. - per-collection record counts (`r|{did}|{col}`) and the lifecycle gauges go through `add_record_count`, `replace_record_counts`, `transition_lifecycle` and friends, which stage them on the same txn. - `outbox` holds the stream events the write produces, without positions yet. ### repo rows `Txn::put_repo` and `Txn::remove_repo` are the only writers of `repos` rows. each `k|p|{host}` counter counts the active repos on a pds host, and these move it by the row the write replaces. only a repo's owner shard writes its row, so that row can't change before the commit. `read_repo` reads the row through the txn and remembers its host, so a later `put_repo` in the same txn doesn't read it again. a raw `batch.insert(&db.repos, ..)` leaves a host's count wrong for good. ### committing `commit`, `commit_with` and `commit_assigned` all end in `commit_inner`, which does the steps in one order every time: 1. stage the record counts, the lifecycle gauges and the count deltas into the batch. 2. run the caller's `stage` closure, if any, for writes that have to land in the same batch. 3. commit through the outbox. with events to send, `Sequencer::commit` (`src/db/sequencer.rs`) takes its lock, gives the events their positions, writes their rows into the batch and commits it. 4. apply the counts in memory. 5. broadcast the events, still under the sequencer lock. since positions are only handed out inside that lock, position order is commit order, a broadcast is only sent once its row can be read, and broadcasts reach subscribers in position order across every writer. counts go before the broadcast, so a subscriber reading counts when an event arrives sees it counted. positions are reserved in blocks written to `cursors` in the same batch, so they never repeat across restarts, even when their rows get pruned. dropping a `Txn` without committing it throws away the batch and every delta with it. ### record writes `RecordTxn` is the one write path for records. `Txn::records(state, rev, did)` makes one for a live commit and reads the repo's exclude entry, and `Txn::backfill_step_records` makes one for a backfill step, which skips that read because the shard did it at the first step. it borrows the `Txn`, so its writes land in the same batch and commit with everything else. - `put_record(collection, rkey, block, action)` and `delete_record(collection, rkey)` keep everything about a record in step: the head in `heads`, the old body moving to `history`, the total and per-collection counts, backlinks (with the `backlinks` feature), and the `StoredEvent` and jetstream event, which `RecordEmitter` puts in `txn.outbox`. `put_record` takes a `CarBlock`, hashing it unless the car parse already did, so sparse backfill needs no second parse. - counts follow what happened to the head, not the op's claimed action, so replaying a commit is a no-op for them. - the count deltas go straight onto the `Txn` (`txn.counts` and its record counts), so there's nothing to flush, and several `RecordTxn`s in one `Txn` add up instead of overwriting each other. the resync drain uses that to apply a few buffered commits in one txn. - `read_repo` and `update_repo_state` go through `Txn::put_repo`. one catch: `RecordTxn` reads the current head from the database and not from the batch, so a second commit on the same record in one txn doesn't see what the first one staged. callers are `ops::apply_commit` for live commits, and for backfills the repo's owner shard (`src/ingest/indexer/bulk.rs`), which applies the steps that `backfill/worker/apply.rs` builds for full and sparse backfills alike. the chunked writers in `db::indexer` (a deleted account's wipe, an erase, a redaction) write `heads` and `history` directly, since they're inside `db`. ## blocking fjall calls are blocking. async code runs them through `Db::run` (`src/db/mod.rs`), which runs the closure on `spawn_blocking` with a clone of the `Db` and fails with `Cancelled` if the runtime shut down before it started. threads that are already blocking, like the indexer shards, call the db directly. a fjall call made straight from an async task blocks a runtime worker until it's done. ## poison once a disk write fails, fjall refuses every later write with `Error::Poisoned`. the first error like that has to reach `db.poison.check(..)`, or `check_report(..)` for a `miette::Report`, which ends `Hydrant::run` with `DatabasePoisoned` instead of exiting the process, since hydrant can be embedded. `Ks`'s point reads and writes do the check themselves. iterators reached through `Deref` and batch commits don't, so the code using them has to.