Updates

Drivers, updates and deletes, and key lookups 39 times faster

Applications can now talk to UltraFabric the way they talk to PostgreSQL. SQL and psql lists exactly what works.

• Prepared statements with $1 parameters work, so drivers in their default mode connect unchanged. The Go driver pgx runs the end-to-end tests this way.
• UPDATE and DELETE work. Each runs as one atomic step and changes only your tenant's rows. After the same statements, the results equal DuckDB's.
• Lookups by primary key read only the rows they need. One lookup now takes 0.14 ms instead of 5.5 ms, and eight clients reach about 10,600 lookups per second. These are indicative numbers from a shared development server.
• Managing the schema now needs an operator credential. It can never read or write tenant data, and an ordinary tenant token can never manage the schema.
• Found while testing: an UPDATE by someone who cannot see every column would have written the hidden values back as empty. Such updates are now refused instead.

psql connects

UltraFabric now speaks the PostgreSQL protocol. A gateway, written in Go, takes a token as the password, parses SQL with PostgreSQL's own parser, and turns each statement into a plan that the data node verifies before running it. SQL and psql lists exactly what works.

• The data node establishes the requester's identity itself. A plan that claims another tenant, or rows written for another tenant, are refused at the node.
• Queries never see or write a tenant column; every table is scoped to your tenant.
• The first SQL subset covers tables, inserts, filtered and grouped queries, and joins. Everything else is refused with a clear “not supported” error, never approximated.
• In the end-to-end test, two tenants never see each other's rows, and five queries over 10,000 rows give the same answers as DuckDB.
• Found while testing: SQL INSERT replaced an existing row instead of failing, because the storage layer's insert was an upsert. SQL inserts now use a separate request that fails with the PostgreSQL duplicate-key error. Also fixed: multiplying a numeric(12,2) by an integer gave four decimal places instead of two.
• First latency numbers, from a shared development server and so only indicative: a one-row insert takes about 1 ms end to end, but a lookup by primary key takes 5.5 ms, as long as a full aggregate, because queries do not seek by key yet and the server runs one request at a time. Key lookups and parallel reads are the next piece of work.
• Old row versions are now released on a schedule. They stay readable for 60 seconds by default, and compaction then reclaims the space.

Columnar collections, and analytic queries up to 44 times faster

A collection can now be stored column by column: rows arrive as usual, and are compacted into row groups whose columns are packed by type. A query reads only the columns it needs.

• TPC-H queries 1, 3, 5, 6 and 10 at scale factor 0.1 now take 0.07 to 0.37 seconds, down from 3.3 to 4.3 seconds three phases ago. Every answer equals DuckDB’s. These are indicative numbers from a shared development server, not a benchmark claim.
• On the way: scans decode only the columns a query can observe; filters, comparisons and decimal arithmetic run on typed columns; aggregation works on column batches; the optimizer prunes columns earlier.
• DuckDB is still 11 to 46 times faster on one thread. The remaining cost is in our executor, not in storage, and that is the next piece of work.
• Ten million generated queries still agree with the reference interpreter, and over a thousand generated queries give the same answers on row and columnar storage.

Ten million checked queries, proofs, and an honest speed number

The query layer's correctness work is complete. Its speed is not.

• Ten million generated queries: the executor and every optimizer rewrite agree with the reference interpreter, and one in ten is also checked for tenant isolation against a database that differs only in hidden data. No wrong answers, no isolation failures.
• Eight rewrite laws and the isolation guarantee for the core operators are proved in Lean 4.
• Raw key requests can no longer reach table rows; rows are read only through verified queries.
• Exact decimal multiplication and AVG are in the plan language.
• TPC-H queries 1, 3, 5, 6 and 10 give the same answers as DuckDB, but take 137 to 1 400 times longer. Almost all of the time goes to reading rows: each one is decoded in full, all 16 columns, when a query needs a few. Fixing the scan path is the next piece of work, and the 3× target stays.
• Found while testing: an error message could reveal more than it should. Recorded and closed before network access.

Queries run end to end

Collections now live in a catalog inside the storage engine, and the local server runs queries: it decodes the plan, verifies it for the requester, optimizes it, and executes it with a vectorized executor that reads only the tenant's part of the data.

• The executor and every optimizer rewrite agree with the reference interpreter on 20 000 to 25 000 generated queries each.
• Testing found two real bugs and fixed them: the verifier wrongly refused some joins, and canonical plan form could change results by reordering the inputs of a UNION feeding a LIMIT.
• Another fix: optimized plans keep the caller's parameter numbering, so a query binds the values it was sent.
• A query that tries to use too much memory fails with a clear error and leaves the server running.

Query plans and the security verifier

The foundation of the query layer is in place: the plan language, its canonical encoding and fingerprints, a validator that recomputes every type, a reference interpreter that defines correct answers, and the verifier that every data node runs before executing a plan.

• Found and fixed a gap in our own specification: its rules accepted a class of plans whose results could depend on other tenants' rows. The verifier now rejects them. Security model states the guarantee.
• Plan encodings match the reference protobuf compiler byte for byte; the BLAKE3 hash matches all 35 official test vectors.

Storage engine complete

Snapshot-isolation transactions, idempotent writes, and ufd, a local server with a checksummed protocol and Prometheus metrics.

• Incremental compaction cut the worst write latency on our development server from 371 ms to 23 ms.
• Group commit lets concurrent writers share disk syncs: 17 writes per sync with 32 clients.
• Restart points in table blocks and a faster checksum made point reads 4.4 times faster than the first measurement.

Compaction, caching and a RocksDB comparison

Block cache, two-level compaction, garbage collection of old versions with a durable horizon, and a memory governor that reads the system's memory limit. UltraFabric now runs the same randomized workloads as RocksDB and a simple model, and all three must agree.

• The first benchmarks found range scans reading far more data than needed; a lazy merge fixed it.

Project start

Specifications for the system and the plan language, a deterministic simulator with a fault-injecting disk, a TLA+ model of the write-ahead log checked with TLC, and the log itself, which survived a campaign of one million simulated crashes without losing an acknowledged write.