Node protocol v0
ufd serves one storage engine over a Unix socket. Protocol v0 is small, checksummed and strict. It is the engine's local interface, not the client protocol: applications connect through the gateway over the PostgreSQL protocol (see SQL and psql).
v0 is temporary. It will be replaced by a protobuf node protocol, and these numbers may change.
Running the server
ufd --dir <data dir> --socket <path> [--memtable-mb N] [--cache-mb N] [--query-mb N]
[--jwks <file> --issuer <iss> [--audience <aud>]] [--snapshot-retention-secs N]With --jwks, every connection must log in with a token before anything else (see Login). Without it, the server trusts its local callers, and the socket's file permissions are the only boundary.
Old versions are kept readable for --snapshot-retention-secs (default 60). After that the garbage-collection horizon moves forward on its own, and compaction drops what it released while the server is idle. With 0, the horizon moves only by request 7.
Frames
frame = body_len:u32 LE | crc32c(body):u32 LE | body (body ≤ 16 MiB)
request = kind:u8 | id:u64 | payload
response = status:u8 (0 ok, 1 error) | id:u64 | payload
error = code:u16 | message (rest of the body, UTF-8)
bytes = len:u32 | data
opt = flag:u8 (0 or 1) | [value]Decoders reject oversized frames, checksum mismatches, unknown kinds and trailing bytes, and never crash on malformed input.
Requests
| Kind | Request | Returns |
|---|---|---|
| 1 | Ping | Pong |
| 2 | Snapshot | The current read timestamp |
| 3 | Get (snapshot, key) | The value, or none |
| 4 | Write (batch of puts and deletes) | The commit timestamp |
| 5 | Write idempotent (client key, batch) | Applied, or the earlier outcome |
| 6 | Scan (snapshot, start, end, limit) | Key and value pairs; at most 10,000 |
| 7 | Set GC horizon (timestamp) | Done |
| 8 | Flush | Done |
| 9 | Stats | Tables, memory, cache and compaction counters |
| 10 | Metrics | Prometheus text, including per-request latency |
| 11 | Create collection (name, definition) | Done |
| 12 | Insert (collection, rows) | The commit timestamp |
| 13 | Query (plan, parameters, tenant, policy version, clearance) | Result rows |
| 14 | Catalog | Catalog and policy versions, and every collection with its fields and primary key |
| 15 | Login (token) | Done |
| 16 | Insert new (collection, rows) | The commit timestamp; an existing primary key is code 109 and nothing is written |
Writes that arrive together are committed as a group with one disk sync, so many small writers do not each pay for their own sync.
Collection rows are not part of the raw keyspace. Raw requests that name a row key are refused with code 108, and raw scans skip the rows: they return the raw keys before and after them, in order.
Queries
A query carries a plan in its canonical binary form, built against a catalog version. The node decodes it, rejecting any other encoding; checks the catalog version; runs the verifier for the requester; optimizes it; and executes it at a fresh snapshot. Each result row is the canonical encoding of the row as a list.
Login
With --jwks, a connection starts with Login and an RS256 token. The node checks the signature against the key set, the issuer and audience, and the validity window. It then holds that identity for the life of the connection:
- a query or an insert for any other tenant is refused (codes 110 and 112);
- creating collections, raw key requests and administrative requests need the
adminrole.
ufd listens only on its Unix socket. Network clients go through the gateway.
Error codes
| Code | Meaning |
|---|---|
| 1 | Write-ahead log failure; the log no longer accepts writes |
| 2 | Table file failure |
| 3 | I/O error |
| 4 | Invalid batch |
| 5 | Snapshot older than the GC horizon; read at a newer snapshot |
| 6 | Invalid GC horizon |
| 7 | Overloaded; retry after a pause |
| 8 | Idempotency key reused with a different batch |
| 9 | Corruption detected |
| 100 | The request could not be decoded |
| 101 | The query exceeded its memory budget or result limits |
| 102 | The plan was built against another catalog version; rebuild it |
| 103 | The verifier rejected the plan |
| 104 | Evaluation error: overflow, division by zero, a failed cast or a bad parameter |
| 105 | No such collection |
| 106 | The collection already exists |
| 107 | An invalid collection definition, or a row that does not fit its collection |
| 108 | A raw Get or Write touched a collection row key; use Query and Insert |
| 109 | The primary key already exists: always for Insert new, and for Insert into a columnar collection (columnar collections are append-only) |
| 110 | A row belongs to another tenant than the logged-in identity |
| 111 | Not logged in, or the token was refused |
| 112 | Not allowed for this identity, or a request that needs the admin role |