P2c differential: ten million plans
Ten million generated plans checked against the reference interpreter, every optimizer rewrite and the encoding round trip, plus isolation checks: 0 failures.
Run details
| Date | |
|---|---|
| Commit | 89054ec, 36fe928, eea0f9f (chunks); 69efb19 (final full run) |
| Host | sttiz, release build, 3 threads under the uf-remote caps. Times are indicative. |
| Tool | scripts/nightly-diff 2000000 <from>, run five times (ufdiff). |
| Raw results | docs/bench/results/diff-runs.jsonl |
| Report | docs/bench/2026-10-21-p2c-differential.md |
Indicative only. Measured on a shared development host with other services running, one node, data that fits in memory. These figures show direction, not performance to expect. The claims policy requires a dedicated host before any target is published.
The tables, as written
Transcribed from the report without rounding. Bracketed values are the previous measurement where the report gives one.
Chunked run, 2 000 000 seeds per chunk
| Seeds | Failures | With rows | Rewritten | NI checked | Seconds | Commit |
|---|---|---|---|---|---|---|
| 1 – 2 000 000 | 0 | 1 020 495 | 1 232 830 | 198 902 | 247.4 | 89054ec |
| 2 000 001 – 4 000 000 | 0 | 1 020 049 | 1 232 143 | 198 856 | 247.6 | 89054ec |
| 4 000 001 – 6 000 000 | 0 | 1 021 260 | 1 231 453 | 198 872 | 247.9 | 36fe928 |
| 6 000 001 – 8 000 000 | 0 | 1 020 574 | 1 231 902 | 198 904 | 278.5 | eea0f9f |
| 8 000 001 – 10 000 000 | 0 | 1 020 969 | 1 232 190 | 198 871 | 254.6 | eea0f9f |
| 10 000 000 | 0 | 5 103 347 | 6 160 518 | 994 405 | 1 276 |
Final run after the branch review (commit 69efb19)
| Seeds | Failures | Done | NI checked (each on plan + optimized plan) | Seconds |
|---|---|---|---|---|
| 1 – 10 000 000 | 0 | 10 000 000 | 994 405 | 1 352.7 |
Before and during this run
- Each seed generates a secured database and a verified plan, then checks the executor against the reference interpreter, the optimized plan in both engines against the original in the reference, the encoding round trip, and, every tenth seed, noninterference: a database that differs only in rows or fields the principal cannot see must give the same result in both engines.
- Earlier attempts in the same raw file stopped on resource limits, never on a result difference. Each stop became a regression test: correctness runs use a 1 GiB budget instead of the executor's default 64 MiB; the noninterference check uses the same options as the correctness check, and two budget errors count as one outcome; the row and work caps are lifted for correctness runs after a 143 208-row join result hit the default 100 000-row cap.
- The branch review found that a panicking worker was reported as a pass and that the noninterference check ran only on the unoptimized plan. Both were fixed (commit 69efb19) and all ten million seeds were run again with every check at full limits.
What the numbers say
The report's own reading, including what went wrong and what is still open.
- No wrong answer and no interference in ten million plans, in the chunked run and again in the final full run.
- Lifting a limit only changes seeds that hit it, and those failed their chunk, so the clean chunks run before each fix remain valid.
- The byte count of a budget error depending on hidden rows is a resource channel; it is recorded in the P2 gate report.
Files and tools
Report
docs/bench/2026-10-21-p2c-differential.md in the repository.
Raw results
docs/bench/results/diff-runs.jsonl
Access
The repository is private during development. Early-access teams receive the build, the benchmark scripts and the raw files.
See the measurements in context.
The performance page draws the latest figure from each area and states what is not measured yet.