New: prepared statements, UPDATE and DELETE, and key lookups that seek — 0.143 ms through the gateway.Read the update

ULTRAFABRICData · Scale · Intelligence
DocsGet Started

Storage engine

The engine is log-structured: writes go to a write-ahead log and an in-memory table, which is later written out as sorted, checksummed table files and compacted in the background.

Writing

  • Write-ahead log. Every batch is a record with a length, a CRC-32C and a chained hash, written to segment files. A write is acknowledged only after it is durable.
  • Group commit. Batches waiting at the same time share one fdatasync. With 32 concurrent clients on a development machine, 17 writes shared each sync on average.
  • Fail closed. If a sync fails, the log refuses further writes rather than guessing what reached the disk.

Keeping

  • Table files. Sorted blocks with restart points, a Bloom filter, an index and a checksummed footer. Every block is verified when read.
  • Manifest. A log of which table files are live. A flush or compaction becomes visible in one atomic manifest record, after the new files are written and verified.
  • Incremental compaction. Merging happens in small steps alongside writes, with bounded memory, so a large merge never stalls a single write. On a development machine the worst write latency fell from 371 ms to 23 ms with this change.
  • Versions and garbage collection. Every write gets a commit timestamp. Old versions are kept until a durable horizon passes them; reading at a snapshot older than the horizon is an explicit error.

Reading

  • Point reads check the in-memory table, then table files newest first, using Bloom filters to skip files.
  • Range scans merge all layers lazily, reading only the blocks they need.
  • A byte-bounded block cache keeps raw blocks; entries are decoded one at a time from the nearest restart point.

Transactions

Transactions use snapshot isolation: a transaction reads from one snapshot and commits only if no other transaction committed a write to the same keys after that snapshot (first committer wins). Write skew is allowed under snapshot isolation; serializable isolation comes with the distributed phase.

Idempotent writes carry a client key. Retrying the same key returns the original outcome instead of writing twice; retrying it with a different batch is an error.

Recovery

On start the engine replays the log from the last flushed point. A torn record at the end of the log is cut off; damage in the middle of the log is reported as corruption, never silently skipped.