an indexer keeps the records it indexes in two keyspaces. heads holds the current body of every record, and history holds the bodies that got replaced or deleted, so replaying an old event from /stream can still show the record as it was. everything here is indexer only. a relay stores no records, and ephemeral mode keeps bodies inside its events instead.

before v10 the bodies lived in a content addressed blocks keyspace ({col}|{cid} → body) and records only mapped {did}|{col}|{rkey} to a cid. the v10 migration (src/db/migration/v10.rs) moved every body into heads and deleted both, so old code and old issues that talk about records and blocks mean that layout.

the keys

heads:   {did} 00 {col} 2f {rkey}                 -> body
history: {did} 00 {col} 2f {rkey} 00 {death_rev}  -> body, or empty for a delete

{did} is the trimmed did (TrimmedDid), {rkey} is the rkey as text and death_rev is an 8-byte big-endian tid. values are raw dag-cbor, and no cid is stored next to a body. anything that needs the cid hashes the body again. the keys are built in src/db/keys/indexer.rs (record_key_trimmed, history_key) and the keyspaces are Heads and History in src/db/schema.rs.

why keys sort in mst order

a repo's mst orders its records by the plain {col}/{rkey} text. heads keys put 00 after the did, / between collection and rkey, and the rkey as text, so a repo's stored keys sort in exactly the order its mst walks them. the separator matters for collections that prefix each other. / (0x2f) sorts below every character an nsid can have, so app.bsky.feed.post/… comes before app.bsky.feed.postgate/…, same as the mst. a separator like | (0x7c) sorts above them and puts postgate first, and storing rkeys tagged by kind would put every string rkey before every tid. record_keys_sort_in_mst_order in src/db/keys/indexer.rs checks this.

a trimmed did:plc is 16 raw bytes and can hold a 00 itself. that's fine since plc segments always have the same length, and trimmed_did_len reads the length from the tag byte instead of scanning for 00.

what this buys is the resync diff. a full backfill has to bring the stored records in line with the fetched repo and emit an event for every record that changed, so src/backfill/diff.rs merges the repo's mst leaves against its stored heads. both come in the same order, so it's a streaming merge. StoredHeads (src/backfill/worker/apply.rs) reads heads a page of at most 4096 at a time and nothing ever holds a whole repo. the merge errors if the fetched side goes out of order, because merging out of order would delete and recreate records.

why bodies are keyed by record and not by cid

content addressing looks like it would save space by storing identical bodies once. measured on the live firehose, only 0.0242 % of writes land on a body another record already has, so that saves almost nothing. what it costs is a 36-byte cid that doesn't compress, in both the record entry and the block key, and a second point read on every getRecord. the little sharing there is also makes deletes unsafe. chat.bsky.actor.declaration bodies are byte-identical across accounts, so one block is the body of lots of records, and deleting it when one of them dies blanks the rest. knowing when it's safe needs refcounts or a reverse index over every live record, which costs more than the garbage it would free.

keyed by record, one author's records of one collection also sit next to each other in tid order, and that compresses better. measured on 240 real repos (269 177 records, 62 MB of dag-cbor), modelled as 64 KiB lsm data blocks with zstd:3, {col}|{cid} came to 122.78 B/record and {did}|{col}|{rkey} to 76.24. most of the gap is the key. a cid compresses to 32.7 B/record and a did, collection and rkey to 6.9, since all three repeat between neighbours. bodies shrink too, from 80.8 to 67.1 B/record, because neighbours share an author, createdAt prefixes and often subjects instead of being shuffled into hash order. the old layout stored 173.6 B/record, 122.4 in blocks and 51.2 in records, so for the whole network that's:

these come from the model (which lands 0.3 % from the measured 122.4 B/record of the old blocks), not from a migrated database.

putting the collection first instead ({col}|{did}|{rkey}) measured 76.06 B/record, a tie. so the did goes first, because then a repo is one range, which is what the diff, generate_car and a deleted account's wipe all want. interning collections as varints saved 0.06 B/record on disk and would break mst order, so collections stay text.

why heads have no rev

one keyspace could hold every version if each key ended in its rev. but a real rev changes in its low bytes on almost every key, so neither front-coding nor zstd can shrink it, and it measured +9.24 B/record (+12 %). every record would pay that for the ~6 % that ever get a second version. so heads have no rev, and only a body that dies gets one, as its key in history.

compression

Heads::options compresses at every level, zstd:3 down to L2 and zstd:5 below with the default HYDRANT_DATA_COMPRESSION=zstd, in 16, 64 and 128 KiB data blocks. compression decides whether this layout wins at all. uncompressed, a body plus its key is about 243 B/record (230.2 raw body, 12.9 front-coded key), worse than the 173.6 of the old layout. it starts at L0 because bulk loads like the migration write sorted tables that compaction moves down without rewriting, so an uncompressed first level would stay uncompressed through the whole tree. v10 had to build a new keyspace for the same reason: v9's records had "never compress" in its on-disk config, and fjall reads that at open.

History::options uses the same compression in 64 and 128 KiB blocks and expect_point_read_hits(false), since it's only read on replay and those reads mostly miss. neither keyspace trains a zstd dictionary (Schema::TRAIN is unset), because fjall compresses whole data blocks where zstd already has enough context, and a trained dictionary measured worse.

bodies stay inline in data blocks instead of in fjall's blob files. 94.5 % of records are under 512 B and records of 1 KiB or more are 1.0 % of records and 8.1 % of bytes, and kv separation only pays off for values above about 1 KiB.

what a write does

every record write goes through RecordTxn (src/db/txn.rs), made for one commit and its rev. live commits come in through ops::apply_commit, and a backfill's writes come in steps on the repo's owner shard through Txn::backfill_step_records (src/ingest/indexer/bulk.rs), so a resync moves superseded bodies to history the same way a live commit does.

put_record reads the current head first, since the firehose op says create or update but the head says what's actually there:

• same cid, stored the same way: nothing changes.
• same cid, but one side is a body and the other a cid marker: only the head is rewritten, which applies a redaction or fills in a marker once the body is around.
• different cid: the old body goes to history at the death rev, then the new head is written. a cid marker isn't copied, and links-only mode writes no history.
• no head: the new head is written and the record counted.

delete_record moves the head's body to history (an empty entry if the head was a cid marker), removes the head and takes it off the counts. deleting a record with no head writes an empty entry, unless the record already has an entry at or after this commit's rev (already_deleted), so replaying a delete changes nothing. the empty entry is what tells "deleted at this rev" apart from "never existed".

counts follow what happened to the head and not what the op claimed, so replaying a commit doesn't move them.

the death rev

history is keyed by when a version died, not when it was written. a head has no rev, so the rev it was written at isn't known when it dies, but the rev of the commit replacing it is. RecordTxn::death_rev takes the later of that commit rev and the local clock. retention ages history by the local clock, so a late commit with an old rev still keeps the body it replaced for the whole ttl, and a death always sorts after the events before it.

deleted accounts

an account deletion moves all of the repo's heads to history at one death rev, the later of the local clock and the repo's last rev plus 1 µs (db::indexer::death_rev). the repo's owner shard does it as a bulk job (job|{did} in crawler, which keeps the death rev), a chunk at a time through wipe_heads in src/db/indexer.rs, so a restart picks up where it stopped. it writes no tombstones and copies each head value as is, cid markers included. in links-only mode it copies nothing and only deletes the heads.

who reads what

getRecord, listRecords, generate_car, backlinks and per-collection counts only ever read heads. backlinks never index history, and counts count heads. history has one reader, event replay. /stats reports history from fjall's approximate length, so removed entries count until compaction rewrites their tables (see retention).

replaying an old event

in a permanent indexer a create or update event is stored as StoredData::Ptr(cid) plus the event's rev, without the body. /stream fills in the body when it sends the event, through Db::resolve_event_record_body in src/db/indexer.rs, which tries in order:

1. the head, if its body hashes to the event's cid.
2. event_bodies at {record key} 00 {cid}. v10 put the bodies of pre-v10 events there that no head or history entry could resolve, and nothing adds to it anymore. a body there that doesn't match its cid is an error.
3. the record's history entries that died after the event's rev, oldest first, skipping empty ones, until one hashes to the cid.

it's strictly after the event's rev because a version that died at rev R was replaced by the commit at R, so an event at R points at the new body. it walks and checks cids instead of taking the first entry because the first death after R isn't always the version the event points at. when the local clock is ahead of R, the version the event's own commit replaced dies at that later time and lands in the range too, and a trimmed entry would make the next one look right. with the check, a body is either the right one or a miss.

the range is (K 00 R, K 01) (keys::history_range_after), which holds exactly record K's entries. without the 00 a longer rkey would fall inside it, since abc is a prefix of abcd and d sorts below 0xff. 00 sorts below every byte an rkey can hold, so [K 00 …, K 01) can't reach abcd. keys::history_hit_for also checks that a hit is exactly len(K) + 9 bytes and starts with K, and history_key_brackets_and_hit_guard tests both.

a miss sends the event without its body and logs at debug. with HYDRANT_HISTORY_TTL set it's also counted as history_trim_misses in /stats, because there it's expected.

retention

without HYDRANT_HISTORY_TTL, history is kept forever. with it, Db::open installs HistoryRetentionFilterFactory (src/db/compaction.rs) on history and nothing else. the filter only looks at keys: the last 8 bytes are the death rev, its timestamp is the rev shifted right by 10 bits, and entries older than now minus the ttl are dropped with Verdict::Destroy, which writes no tombstone. it can't see a current body, since heads live in another keyspace. entries go when compaction next rewrites them, not at the deadline, so old data at the bottom of the tree can sit there until POST /db/compact. open keeps everything until migrations and the storage mode check are done, so a failed open can't trim anything. Config::validate refuses the ttl together with HYDRANT_EPHEMERAL, and it can change between restarts.

keeping history forever

the network deletes or replaces 2 211 850 records a day, 6.02 % of creates. at 122.4 B/record that's 271 MB/day, or 98.8 GB/year of superseded bodies, next to the growth of the live records:

about 4 % on top of growth you already pay for, so permanent is the default. in the current layout a history entry is around 85 B, which makes it about 68.6 GB/year. permanent starts when hydrant first indexes a repo though. a pds only serves a repo's current state, so nothing older than that can be filled in.

event ttl and history ttl

the two aren't tied together. only an ephemeral indexer expires its events, and it has no history. a permanent indexer keeps its events forever, so with a history ttl it ends up with events whose bodies are gone. that's safe because of the cid check in replay: a trimmed body is a miss, and replay never sends a newer head in its place.

redactions

an operator erase of one record (/redactions, redact_record_bodies in src/db/indexer.rs) swaps the bodies in heads and history for their 36-byte cids, deletes the record's event_bodies entries and writes {record key} 00 {cid} into redactions for each version. a cid marker can't be mistaken for a body, because a dag-cbor record is always a map and never starts with the cidv1 dag-cbor sha256 prefix (is_cid_record_value). history tombstones stay. it runs on the repo's owner shard, so it lands in order with the repo's commits.

put_record checks redactions for every version it writes. a replay or a backfill that brings an erased version back stores the marker instead, drops its backlinks and emits the event without a body. markers don't expire with the history ttl. erasing a whole repo deletes its markers too, so tracking it again backfills it from nothing.

links-only mode

with HYDRANT_ONLY_INDEX_LINKS every head holds the record's cid marker instead of its body, and puts and deletes write no history. there are no bodies to serve or replay. it's fixed per database like ephemeral mode.

ephemeral mode is its own layout

with HYDRANT_EPHEMERAL an indexer writes no heads and no history. RecordTxn puts each create or update body inside its event as StoredData::Block, and the event ttl drops events with drop_range, so a body goes with its event. there's no getRecord, listRecords or car export.

it's a separate layout and not a setting on top of heads because the two want opposite sort orders. lookups want key order, to find a body by record. cheap expiry wants time order, because drop_range only drops tables that sit fully inside the range, which is free when old data fills whole tables and useless when the expiring keys are spread across the keyspace. events are in time order and heads are in key order, and the only bridge between them would be an index as big as the live record set. so ephemeral picks time order and gives up lookups, and a permanent indexer picks key order and expires history with a compaction filter instead.

the storage mode is written into the database on first open (db::storage_mode), and starting it with a different HYDRANT_EPHEMERAL or HYDRANT_ONLY_INDEX_LINKS is refused.