# record storage an indexer keeps the records it indexes in two keyspaces. `heads` holds the current body of every record, and `history` holds the bodies that got replaced or deleted, so replaying an old event from `/stream` can still show the record as it was. everything here is indexer only. a relay stores no records, and [ephemeral mode](#ephemeral-mode-is-its-own-layout) keeps bodies inside its events instead. before v10 the bodies lived in a content addressed `blocks` keyspace (`{col}|{cid}` → body) and `records` only mapped `{did}|{col}|{rkey}` to a cid. the v10 migration (`src/db/migration/v10.rs`) moved every body into `heads` and deleted both, so old code and old issues that talk about `records` and `blocks` mean that layout. ## the keys ``` heads: {did} 00 {col} 2f {rkey} -> body history: {did} 00 {col} 2f {rkey} 00 {death_rev} -> body, or empty for a delete ``` `{did}` is the trimmed did (`TrimmedDid`), `{rkey}` is the rkey as text and `death_rev` is an 8-byte big-endian tid. values are raw dag-cbor, and no cid is stored next to a body. anything that needs the cid hashes the body again. the keys are built in `src/db/keys/indexer.rs` (`record_key_trimmed`, `history_key`) and the keyspaces are `Heads` and `History` in `src/db/schema.rs`. ### why keys sort in mst order a repo's mst orders its records by the plain `{col}/{rkey}` text. heads keys put `00` after the did, `/` between collection and rkey, and the rkey as text, so a repo's stored keys sort in exactly the order its mst walks them. the separator matters for collections that prefix each other. `/` (`0x2f`) sorts below every character an nsid can have, so `app.bsky.feed.post/…` comes before `app.bsky.feed.postgate/…`, same as the mst. a separator like `|` (`0x7c`) sorts above them and puts `postgate` first, and storing rkeys tagged by kind would put every string rkey before every tid. `record_keys_sort_in_mst_order` in `src/db/keys/indexer.rs` checks this. a trimmed did:plc is 16 raw bytes and can hold a `00` itself. that's fine since plc segments always have the same length, and `trimmed_did_len` reads the length from the tag byte instead of scanning for `00`. what this buys is the resync diff. a full backfill has to bring the stored records in line with the fetched repo and emit an event for every record that changed, so `src/backfill/diff.rs` merges the repo's mst leaves against its stored heads. both come in the same order, so it's a streaming merge. `StoredHeads` (`src/backfill/worker/apply.rs`) reads heads a page of at most 4096 at a time and nothing ever holds a whole repo. the merge errors if the fetched side goes out of order, because merging out of order would delete and recreate records. ### why bodies are keyed by record and not by cid content addressing looks like it would save space by storing identical bodies once. measured on the live firehose, only 0.0242 % of writes land on a body another record already has, so that saves almost nothing. what it costs is a 36-byte cid that doesn't compress, in both the record entry and the block key, and a second point read on every `getRecord`. the little sharing there is also makes deletes unsafe. `chat.bsky.actor.declaration` bodies are byte-identical across accounts, so one block is the body of lots of records, and deleting it when one of them dies blanks the rest. knowing when it's safe needs refcounts or a reverse index over every live record, which costs more than the garbage it would free. keyed by record, one author's records of one collection also sit next to each other in tid order, and that compresses better. measured on 240 real repos (269 177 records, 62 MB of dag-cbor), modelled as 64 KiB lsm data blocks with zstd:3, `{col}|{cid}` came to 122.78 B/record and `{did}|{col}|{rkey}` to 76.24. most of the gap is the key. a cid compresses to 32.7 B/record and a did, collection and rkey to 6.9, since all three repeat between neighbours. bodies shrink too, from 80.8 to 67.1 B/record, because neighbours share an author, `createdAt` prefixes and often subjects instead of being shuffled into hash order. the old layout stored 173.6 B/record, 122.4 in `blocks` and 51.2 in `records`, so for the whole network that's: $$ 173.6 \rightarrow 76.0\ \mathrm{B/record} = -56\ \%,\qquad 4.7\ \mathrm{TB} \rightarrow \sim2.06\ \mathrm{TB} $$ these come from the model (which lands 0.3 % from the measured 122.4 B/record of the old `blocks`), not from a migrated database. putting the collection first instead (`{col}|{did}|{rkey}`) measured 76.06 B/record, a tie. so the did goes first, because then a repo is one range, which is what the diff, `generate_car` and a deleted account's wipe all want. interning collections as varints saved 0.06 B/record on disk and would break mst order, so collections stay text. ### why heads have no rev one keyspace could hold every version if each key ended in its rev. but a real rev changes in its low bytes on almost every key, so neither front-coding nor zstd can shrink it, and it measured +9.24 B/record (+12 %). every record would pay that for the ~6 % that ever get a second version. so heads have no rev, and only a body that dies gets one, as its key in `history`. ## compression `Heads::options` compresses at every level, zstd:3 down to L2 and zstd:5 below with the default `HYDRANT_DATA_COMPRESSION=zstd`, in 16, 64 and 128 KiB data blocks. compression decides whether this layout wins at all. uncompressed, a body plus its key is about 243 B/record (230.2 raw body, 12.9 front-coded key), worse than the 173.6 of the old layout. it starts at L0 because bulk loads like the migration write sorted tables that compaction moves down without rewriting, so an uncompressed first level would stay uncompressed through the whole tree. v10 had to build a new keyspace for the same reason: v9's `records` had "never compress" in its on-disk config, and fjall reads that at open. `History::options` uses the same compression in 64 and 128 KiB blocks and `expect_point_read_hits(false)`, since it's only read on replay and those reads mostly miss. neither keyspace trains a zstd dictionary (`Schema::TRAIN` is unset), because fjall compresses whole data blocks where zstd already has enough context, and a trained dictionary measured worse. bodies stay inline in data blocks instead of in fjall's blob files. 94.5 % of records are under 512 B and records of 1 KiB or more are 1.0 % of records and 8.1 % of bytes, and kv separation only pays off for values above about 1 KiB. ## what a write does every record write goes through `RecordTxn` (`src/db/txn.rs`), made for one commit and its rev. live commits come in through `ops::apply_commit`, and a backfill's writes come in steps on the repo's owner shard through `Txn::backfill_step_records` (`src/ingest/indexer/bulk.rs`), so a resync moves superseded bodies to history the same way a live commit does. `put_record` reads the current head first, since the firehose op says create or update but the head says what's actually there: - same cid, stored the same way: nothing changes. - same cid, but one side is a body and the other a cid marker: only the head is rewritten, which applies a redaction or fills in a marker once the body is around. - different cid: the old body goes to `history` at the death rev, then the new head is written. a cid marker isn't copied, and links-only mode writes no history. - no head: the new head is written and the record counted. `delete_record` moves the head's body to `history` (an empty entry if the head was a cid marker), removes the head and takes it off the counts. deleting a record with no head writes an empty entry, unless the record already has an entry at or after this commit's rev (`already_deleted`), so replaying a delete changes nothing. the empty entry is what tells "deleted at this rev" apart from "never existed". counts follow what happened to the head and not what the op claimed, so replaying a commit doesn't move them. ### the death rev history is keyed by when a version died, not when it was written. a head has no rev, so the rev it was written at isn't known when it dies, but the rev of the commit replacing it is. `RecordTxn::death_rev` takes the later of that commit rev and the local clock. retention ages history by the local clock, so a late commit with an old rev still keeps the body it replaced for the whole ttl, and a death always sorts after the events before it. ### deleted accounts an account deletion moves all of the repo's heads to history at one death rev, the later of the local clock and the repo's last rev plus 1 µs (`db::indexer::death_rev`). the repo's owner shard does it as a bulk job (`job|{did}` in `crawler`, which keeps the death rev), a chunk at a time through `wipe_heads` in `src/db/indexer.rs`, so a restart picks up where it stopped. it writes no tombstones and copies each head value as is, cid markers included. in links-only mode it copies nothing and only deletes the heads. ## who reads what `getRecord`, `listRecords`, `generate_car`, backlinks and per-collection counts only ever read `heads`. backlinks never index history, and counts count heads. `history` has one reader, event replay. `/stats` reports `history` from fjall's approximate length, so removed entries count until compaction rewrites their tables (see [retention](https://hydrant.klbr.net/concepts/retention/index.md)). ## replaying an old event in a permanent indexer a create or update event is stored as `StoredData::Ptr(cid)` plus the event's rev, without the body. `/stream` fills in the body when it sends the event, through `Db::resolve_event_record_body` in `src/db/indexer.rs`, which tries in order: 1. the head, if its body hashes to the event's cid. 2. `event_bodies` at `{record key} 00 {cid}`. v10 put the bodies of pre-v10 events there that no head or history entry could resolve, and nothing adds to it anymore. a body there that doesn't match its cid is an error. 3. the record's history entries that died after the event's rev, oldest first, skipping empty ones, until one hashes to the cid. it's strictly after the event's rev because a version that died at rev `R` was replaced by the commit at `R`, so an event at `R` points at the new body. it walks and checks cids instead of taking the first entry because the first death after `R` isn't always the version the event points at. when the local clock is ahead of `R`, the version the event's own commit replaced dies at that later time and lands in the range too, and a trimmed entry would make the next one look right. with the check, a body is either the right one or a miss. the range is `(K 00 R, K 01)` (`keys::history_range_after`), which holds exactly record `K`'s entries. without the `00` a longer rkey would fall inside it, since `abc` is a prefix of `abcd` and `d` sorts below `0xff`. `00` sorts below every byte an rkey can hold, so `[K 00 …, K 01)` can't reach `abcd`. `keys::history_hit_for` also checks that a hit is exactly `len(K) + 9` bytes and starts with `K`, and `history_key_brackets_and_hit_guard` tests both. a miss sends the event without its body and logs at `debug`. with `HYDRANT_HISTORY_TTL` set it's also counted as `history_trim_misses` in `/stats`, because there it's expected. ## retention without `HYDRANT_HISTORY_TTL`, history is kept forever. with it, `Db::open` installs `HistoryRetentionFilterFactory` (`src/db/compaction.rs`) on `history` and nothing else. the filter only looks at keys: the last 8 bytes are the death rev, its timestamp is the rev shifted right by 10 bits, and entries older than now minus the ttl are dropped with `Verdict::Destroy`, which writes no tombstone. it can't see a current body, since heads live in another keyspace. entries go when compaction next rewrites them, not at the deadline, so old data at the bottom of the tree can sit there until `POST /db/compact`. open keeps everything until migrations and the storage mode check are done, so a failed open can't trim anything. `Config::validate` refuses the ttl together with `HYDRANT_EPHEMERAL`, and it can change between restarts. ### keeping history forever the network deletes or replaces 2 211 850 records a day, 6.02 % of creates. at 122.4 B/record that's 271 MB/day, or 98.8 GB/year of superseded bodies, next to the growth of the live records: $$ \underbrace{36.7\,\mathrm{M} \times 173.6\ \mathrm{B} = 2.33\ \mathrm{TB/yr}}_{\text{live growth}} \qquad\text{vs}\qquad \underbrace{98.8\ \mathrm{GB/yr}}_{\text{history}} \;=\; 4.2\ \%\ \text{overhead} $$ about 4 % on top of growth you already pay for, so permanent is the default. in the current layout a history entry is around 85 B, which makes it about 68.6 GB/year. permanent starts when hydrant first indexes a repo though. a pds only serves a repo's current state, so nothing older than that can be filled in. ### event ttl and history ttl the two aren't tied together. only an ephemeral indexer expires its events, and it has no history. a permanent indexer keeps its events forever, so with a history ttl it ends up with events whose bodies are gone. that's safe because of the cid check in replay: a trimmed body is a miss, and replay never sends a newer head in its place. ## redactions an operator erase of one record (`/redactions`, `redact_record_bodies` in `src/db/indexer.rs`) swaps the bodies in `heads` and `history` for their 36-byte cids, deletes the record's `event_bodies` entries and writes `{record key} 00 {cid}` into `redactions` for each version. a cid marker can't be mistaken for a body, because a dag-cbor record is always a map and never starts with the cidv1 dag-cbor sha256 prefix (`is_cid_record_value`). history tombstones stay. it runs on the repo's owner shard, so it lands in order with the repo's commits. `put_record` checks `redactions` for every version it writes. a replay or a backfill that brings an erased version back stores the marker instead, drops its backlinks and emits the event without a body. markers don't expire with the history ttl. erasing a whole repo deletes its markers too, so tracking it again backfills it from nothing. ## links-only mode with `HYDRANT_ONLY_INDEX_LINKS` every head holds the record's cid marker instead of its body, and puts and deletes write no history. there are no bodies to serve or replay. it's fixed per database like ephemeral mode. ## ephemeral mode is its own layout with `HYDRANT_EPHEMERAL` an indexer writes no heads and no history. `RecordTxn` puts each create or update body inside its event as `StoredData::Block`, and the event ttl drops events with `drop_range`, so a body goes with its event. there's no `getRecord`, `listRecords` or car export. it's a separate layout and not a setting on top of `heads` because the two want opposite sort orders. lookups want key order, to find a body by record. cheap expiry wants time order, because `drop_range` only drops tables that sit fully inside the range, which is free when old data fills whole tables and useless when the expiring keys are spread across the keyspace. events are in time order and heads are in key order, and the only bridge between them would be an index as big as the live record set. so ephemeral picks time order and gives up lookups, and a permanent indexer picks key order and expires history with a compaction filter instead. the storage mode is written into the database on first open (`db::storage_mode`), and starting it with a different `HYDRANT_EPHEMERAL` or `HYDRANT_ONLY_INDEX_LINKS` is refused.