Benchmarks and fault fixtures

The S26 tools measure verified CPU transfers and prepare the P10 lifetime matrix. They change no transfer policy. Results describe the recorded host and artifact; they are not hardware qualification or a general performance claim.

CPU sweeps

benches/run.py runs the public Rust API and a test-only C++ caller of frozen Mooncake v0.3.14. The C++ runtime is confined to the reference executable. CPU memcpy, ordinary TCP and io_uring have matched rows; HP-TCP and SHM have OrbiTE-only rows. HP-TCP uses 127.0.0.1 and 127.0.0.2 as two loopback rails.

The default workload is READ and WRITE at 4 KiB, 64 KiB, 1 MiB and 64 MiB, plus batches of 8 and 64 four-KiB blocks. Contiguous memory-target blocks exercise physical-owner merging. Gapped blocks use a stride of twice the block size and must remain separate owners. File batches preserve one owner per public request for independent CQE outcomes, including contiguous blocks; the runner checks that mapping too. There is one caller, two configured workers per engine, three warmup batches, 100 samples and three independent process trials. Trial order alternates the two implementations. Both disable engine failover, metrics, progress workers and runtime queues, and enable merging. TENT’s transport-local TCP retry defaults are pinned separately: three retries, 100 ms initial delay and a 2,000 ms cap. OrbiTE has no corresponding inner TCP retry loop. These are healthy-path measurements, not retry benchmarks. The manifest records these settings, actual process thread counts, compiler versions, affinity, source and binary hashes, filesystems and every command. Internal worker arrangements differ: these are matched caller/worker settings, not identical engine implementations. TENT’s common and TE RPC client pools are each pinned to the same worker count through their two explicit environment variables.

Each iteration changes the source pattern and resets the destination. Every payload byte, gap and guard is checked after every transfer, including warmup; corruption aborts without a result. The runner also corrupts a guard in each implementation and checks rejection. It rejects missing checks, wrong byte counts and incomplete/nonpositive samples. There is no unchecked mode.

Timing covers batch allocation, submission, the same polling schedule (128 hot polls, then exponential sleeps capped at 200 microseconds), and batch free. OrbiTE additionally checks local drain. Setup, registration, reset and full verification are outside the interval. CPU time uses the whole process clock, including both engines’ threads during that interval. Throughput divides verified payload bytes by the sum of measured batch times. p50 and nearest-rank p99 are empirical per-batch latency, not per-block latency; 100 samples give limited tail resolution. The CPU memcpy row is only a memory-copy control, not an engine submission. Raw samples are retained so results can be recomputed.

File offsets, lengths and buffer addresses are four-KiB aligned. The runner requires actual ext4 and tmpfs mounts and checks OrbiTE’s descriptor route. Both file profiles leave TCP installed for host registration (TENT does not include IOURING in its memory-registration transport list), but requests carry an explicit io_uring hint. TENT attempts O_DIRECT on both; on tmpfs the flag may be accepted without a direct path. Both harnesses open and preallocate the file once, reset it in place with pwrite, and finish fdatasync before the timing window. Buffered reset keeps verification simple without leaving dirty reset pages or allocation work for the measured direct I/O. Payload, gap and guard checks are unchanged. These are warmed measurements, not cold SSD throughput. Timed WRITE completion still excludes a durability barrier. Maximum payload per batch is 64 MiB; the allocation and guard extent is bounded separately.

On a Linux runner with the native dependencies from the CPU oracle, also install liburing-dev before building TENT. Enable io_uring in the kernel and choose writable ext4 and tmpfs paths:

python3 tests/reference/run.py --fetch-reference --output "$OUT/reference"
python3 benches/run.py --reference "$OUT/reference" --output "$OUT/sweep" \
  --ext4 "$EXT4_DIR" --tmpfs /dev/shm --workers 2 --warmup 3 \
  --samples 100 --trials 3

Both output directories must be fresh and outside the checkout. The Frozen TENT CPU comparison CI workflow runs this on stable and Rust 1.85 and uploads cpu-bench-* artifacts. GPU, RDMA and GDS are emitted as not run (hardware), without estimated numbers. The scoped summary lives in qualification.

TCP profile and connection reuse

The pre-change profile is run 42439141061, cpu-oracle (stable), revision ec103db (production sources unchanged from f0556448). Its host runnervm7aw5e exposes four AMD EPYC 7763 logical CPUs, Ubuntu 24.04, Linux 7.0.0-1017-azure and Rust 1.99.1. The reference is e1ea5d31bb5e46ff2df9f90e7c2fbaf52fcdee8c. Both toolchains passed all sixteen profile cases with byte checks. These are diagnostic observations, not timings used in the comparison table.

benches/run.py --profile-only uses the same executables, sizes, layouts, workers and warmup as S26. strace -f -c -w counts 100 samples minus a one-sample setup control, divided by 99. perf record samples user CPU at a 10-microsecond period with DWARF stacks and 1,000 samples per workload: millisecond sampling misses short-lived batch threads. perf stat separately records process CPU, context switches and migrations. Raw traces, call stacks and exact commands remain in the workflow’s cpu-bench-* artifacts, outside Git. Profiling includes verification/setup; latency measurements below exclude them.

Baseline calls per batch (READ / WRITE)	OrbiTE	TENT	Interpretation
TCP connect, single 4 KiB	1 / 1	0 / 0	TENT reuses warmed connections
TCP connect, gapped 64 × 4 KiB	64 / 64	0 / 0	One handshake per physical request is confirmed
TCP nanosleep, gapped 64 × 4 KiB	219 / 236	129 / 128	Traced polling sleeps include the common benchmark poll loop
TCP futex, gapped 64 × 4 KiB	429 / 431	481 / 469	This is not a count of locks or thread handoffs
Batch thread creation, every OrbiTE target	1 / 1	0 / 0	One caller→new batch-worker handoff, retained for a later PR
tmpfs ring setup / enter, single 4 KiB	1 / 1	1 / 1	No extra ring is created per request
tmpfs futex, single 4 KiB	13.7 / 13.0	approximately 0 / 0	Publication, progress and worker retirement remain measurable
ext4 ring enter, gapped 64 × 4 KiB	1.14 / 1.11	1 / 1	Reaping can need another enter
ext4 statx, gapped 64 × 4 KiB	65 / 65	0 / 0	Per-request file/descriptor checks remain; no safety checks removed

The single TCP READ’s user samples attribute 18.2% to recv, 14.4% to sched_yield, 12.1% to the main VDSO clock frame and 10.2% to TcpTransport::transfer. For gapped READ/WRITE, recv is 9.9%/10.2%, sched_yield 9.5%/9.2%, and ControlChannel::poll_read 2.1%/2.3%. Pattern generation/verification accounts for another 17% and is outside the benchmark interval. This supports removing direct TCP’s hot-poll/backoff loop as well as reconnects. No admission or loan frame reaches 1% of these samples; inlining and sample size prevent treating that as proof of zero bookkeeping cost. Those owners and checks stay unchanged.

The tmpfs READ profile instead attributes 23.0% to Batch::status, 4.6% to ProgressState::status, 19.4% to the VDSO clock frame and 2.0% to batch publication. The file path still creates a worker and publishes completion to the polling caller. Ext4 READ samples include 1.2% statx, 0.7% selector and 0.5% preparation; pattern generation alone is 46.1%. Aligned ext4 route checks pass, so these runs do not implicate an accidental buffered fallback. They do not isolate the historical p99 tail’s cause; storage and VM scheduling need a separate pass.

Waiting and working cannot be inferred by subtracting multi-thread process CPU from wall latency. In the separate perf stat run, mean wall / process CPU per batch was 285 / 398 microseconds for one TCP READ, 8,861 / 16,408 for gapped TCP READ, 174 / 176 for tmpfs READ, and 1,960 / 1,053 for gapped ext4 READ. Under strace, summed sleep / futex / epoll wall time for gapped TCP READ was 63.9 / 249.2 / 123.4 ms per batch, versus 3.0 / 46.4 / 21.3 ms for gapped ext4 READ (plus 0.39 ms in io_uring_enter). These include concurrent idle threads and tracer perturbation, not critical-path percentages. The CPU stack samples distinguish hot polling from sleeping; the untraced paired sweeps quantify the effect on actual batch latency and CPU cost.

The same perf stat processes report 23,674 / 30,312 context switches for OrbiTE’s gapped TCP READ / WRITE, versus TENT’s 46,849 / 46,749. Each process includes setup, three warmups and 100 samples. That is approximately 230 / 294 versus 455 / 454 switches per executed batch, including setup; it is not an exact count of owner handoffs. The one new thread per OrbiTE batch is the measurable caller-to-worker handoff. A syscall count alone cannot assign every remaining scheduler switch to an owner.

The change follows frozen HP-TCP’s clean/dirty lane ownership without adding a worker: one exclusive epoch-scoped stream, deadline-bounded socket waits, and source readiness monitoring in the existing control workers. Dirty streams are retired, never replayed; admission, backing loans, per-request snapshots, physical drain and sticky cleanup remain. The native frame bytes and offsets are unchanged; a stream now carries successive complete TCP frames. See the owner contract and its negative controls.

Paired results and regressions

Run 42441063520 completed 480 checked rows before and after on each toolchain, comparing f0556448 with de9fd60b. Later commits change fixtures and reporting only. Both sides of each pair share one GitHub ubuntu-24.04 VM, compiler, filesystems and parameters above. Stable (Rust 1.99.1) uses four EPYC 7763 logical CPUs, matching S26’s processor class; Rust 1.85 uses four EPYC 9V45 logical CPUs. Both report hostname runnervm7aw5e and Linux 7.0.0-1017-azure; that hostname alone does not identify a processor. Do not pool the toolchains’ absolute times. The order is before then after, with implementation order alternating across the three trials, not randomized across revisions.

The stable results below are medians of the three trial statistics. The PR’s complete tables include every workload/implementation case, throughput and every numerical regression; all 480 raw trial rows per side remain in cpu-bench-* artifacts. Reproduce those tables with python3 benches/compare.py BEFORE AFTER. It rejects mismatched manifests, workload hashes, missing trials and unchecked data.

Workload	OrbiTE p50 µs before → after	TENT p50 µs after	OrbiTE p99 µs before → after	OrbiTE CPU µs before → after
TCP, 1 × 4 KiB READ	275.3 → 153.5	284.2	320.4 → 261.8	370.3 → 190.3
TCP, 1 × 4 KiB WRITE	277.0 → 160.6	281.6	339.3 → 288.2	385.2 → 215.2
TCP, 64 × 4 KiB gapped READ	8,591.0 → 4,231.4	6,175.9	9,133.9 → 4,556.8	15,908.6 → 5,449.1
TCP, 64 × 4 KiB gapped WRITE	9,331.6 → 6,296.2	6,071.6	9,856.6 → 6,641.5	16,946.6 → 7,055.2
tmpfs, 1 × 4 KiB READ	156.8 → 155.4	82.4	329.3 → 309.8	168.0 → 164.8
tmpfs, 1 × 4 KiB WRITE	194.4 → 194.2	82.6	326.5 → 305.1	247.4 → 243.9
ext4, 64 × 4 KiB gapped READ	926.5 → 933.2	1,018.8	16,517.1 → 17,418.5	1,107.6 → 1,041.9
ext4, 64 × 4 KiB gapped WRITE	873.3 → 892.3	917.5	7,269.4 → 8,969.2	1,096.4 → 1,116.3

The after profile confirms zero warmed TCP connects and zero sched_yield calls per batch, versus 64 connects and 308/342 yields for scattered READ/WRITE. Futex calls fall from 429/431 to 226/242; epoll waits rise from 67/67 to 75/77. One batch-thread creation remains. Pattern generation now occupies 49%/48% of the scattered user-CPU stacks, outside the measured interval; the very short single-request after profiles have only 61/26 stack samples, so their individual frame percentages are not a reliable ranking. The untraced CPU times above are the evidence for the cost reduction.

The EPYC 9V45 pair independently reduces scattered READ/WRITE p50 by 46%/29% and CPU by 64%/58%. Its resulting p50s are 2.26/3.50 ms versus TENT’s 2.92/2.91 ms: WRITE remains 20% slower there. Regressions are not omitted:

• Stable TCP’s 64 MiB WRITE p50 rises 0.3% (overlapping trial ranges), while its p99 and CPU improve. On 1.85, 64 MiB READ p50 rises 2.8%, p99 10.7% and throughput falls 3.2%, while CPU falls 4.2%. Blocking wakeups can trade latency for CPU on a large transfer whose handshake is already amortized; the contribution is not isolated, and the latency trial ranges overlap. This is not a claim of a large-transfer improvement.

• Untuned paths also vary. Stable HP-TCP 1 MiB READ p50 rises 17.3%, with overlapping ranges for all four statistics. Stable ext4 1 MiB READ p50 rises 40.8% (nonoverlapping median ranges); unchanged TENT rises 23.1% on that case. Ext4 64-block contiguous WRITE p99 rises 51.7%, with overlapping trial ranges. On 1.85, ext4 eight-block gapped READ p99 rises 55.8%. Warmed page-cache/coherence and VM scheduling are plausible contributors; shared control-worker effects are not ruled out. The cause remains open.

• Every smaller regression is flagged in the complete tables. Memcpy and TENT controls do not execute the changed TCP pool; their changes show run, cache and scheduling variation. OrbiTE HP-TCP, SHM and file payload owners are unchanged, but their shared idle control worker changed. Range overlap is descriptive, not a significance test or permission to discard a result.

The standard 480-row run on both toolchains also passed, alongside all feature gates and 50-repeat controls and the installed-wheel matrix. These are single-caller cloud-loopback and warmed-file results. No concurrent-caller, physical two-host, GPU, RNIC or SSD performance claim follows. The small-file pass below measures storage latency; ext4 tails and remaining bookkeeping cost stay separate work.

Small io_uring submission profile

The historical ready-CQE pass reduced stable tmpfs 4 KiB READ p50 from 156 to 51 µs, but WRITE remained 193 µs versus TENT’s 84 µs: one nonblocking check often missed the WRITE CQE and started a batch worker. Its ext4 values used the old reset harness.

The before profiles at 578531b7 and after profiles at a5874066 use the current preallocated, synchronized reset and bulk verification. Before profiles use EPYC 7763; after uses EPYC 9V74 on stable and EPYC 7763 on 1.85. These are separate diagnostic runs; the untraced same-VM pairs below establish the timing comparison. Raw perf and strace output stays in cpu-bench-* artifacts. Setup-subtracted calls per 4 KiB tmpfs batch are:

Toolchain / operation	clone/clone3 before → after	futex before → after	io_uring_enter before → after
Stable READ	0.00 → 0.00	6.00 → 6.00	1.00 → 1.00
Stable WRITE	0.00 → 0.00	6.00 → 6.00	1.00 → 1.00
Rust 1.85 READ	0.00 → 0.00	6.00 → 6.01	1.00 → 1.00
Rust 1.85 WRITE	0.00 → 0.00	6.00 → 6.00	1.00 → 1.00

Both keep one ring setup per batch; the one lazy availability probe is outside measurement. TENT likewise records no per-batch clone, one setup/enter and approximately zero futex calls. Futex calls are not a handoff count. Tracing delays the initial CQ check enough to hide worker creation even on the old source, so these zero-clone traces cannot prove ordinary WRITEs avoided it. The less intrusive before WRITE CPU profiles still attribute 1.75%/9.72% to pthread_create on stable/1.85. Neither that frame nor Batch::start_worker appears in the after WRITE reports at the 0.5% cutoff. The 1.85 reports contain only 72 before and 118 after user-stack samples, so absence is not proof of zero calls. Its separate perf stat WRITE sample changes p50/process-CPU from 197/193 to 86/97 µs; process context switches remain 253→225, including setup and warmup. Blocking CQ waits still sleep and wake. These diagnostic runs do not replace the untraced paired comparison.

The caller now submits one contiguous host request of at most 64 KiB on an idle batch, checks the CQ once, then waits up to 200 µs, capped by the request deadline. At most three interrupted/unready waits share that window. A CQE finishes through the normal prefix and admission owners. A missed window or enter error moves the same ring to the existing drain worker without resubmission. Runtime queues, background progress, eligible fallback, CUDA/pinned producer waits, vectors and larger groups retain their worker path. No ring pool, thread pool or coordinator is added. The owner controls preserve preflight, partial-submit drain, exact short prefixes and quarantine. The late-CQE tmpfs/ext4 control counts zero worker starts; disabling only the wait makes it fail with one start. This is the deterministic handoff proof.

Small-file paired results

Run 42692901727 completed 480 checked rows before and after on each toolchain, comparing 578531b7 with a5874066. Each pair shares one VM, compiler, ext4/tmpfs mounts, workload hashes, reset and verification. Three trials, 100 samples, three warmups, one caller and two configured engine workers are unchanged. Revision order is before then after; implementation order alternates across trials. The frozen reference is Mooncake v0.3.14 e1ea5d31. No diagnostic pacing is used. Raw trials and profiles stay outside Git in the run’s artifacts.

The complete stable table and 1.85 table include all 160 workload/implementation cases per toolchain and every numerical regression; each case aggregates three byte-checked trial rows. Reproduce with python3 benches/compare.py BEFORE AFTER --changed-profile io_uring-ext4 --changed-profile io_uring-tmpfs. CPU is process time per batch; the tables show medians of trial statistics.

Stable: four logical CPUs (AMD EPYC 9V74), Ubuntu 24.04, Linux 7.0.0-1017-azure, rustc 1.99.1 (58cf8f9c3 2026-10-13), GCC 13.3.0.

Workload	OrbiTE p50 µs before → after	TENT p50 µs before → after	OrbiTE p99 µs before → after	OrbiTE CPU µs before → after
tmpfs, 1 × 4 KiB READ	33.9 → 37.1	62.8 → 57.5	67.5 → 56.6	37.9 → 39.5
tmpfs, 1 × 4 KiB WRITE	117.9 → 57.3	59.7 → 56.3	196.7 → 151.9	130.2 → 61.5
tmpfs, 1 × 64 KiB READ	45.6 → 44.3	71.4 → 69.5	108.7 → 131.6	51.1 → 48.3
tmpfs, 1 × 64 KiB WRITE	128.1 → 70.0	132.9 → 127.0	193.9 → 166.8	148.6 → 73.6
ext4, 1 × 4 KiB READ	178.1 → 122.7	142.0 → 143.4	259.5 → 252.6	143.0 → 68.1
ext4, 1 × 4 KiB WRITE	200.2 → 144.9	197.0 → 199.7	275.6 → 434.3	182.7 → 85.1
ext4, 1 × 64 KiB READ	193.7 → 150.1	208.2 → 204.4	239.2 → 268.0	148.6 → 75.9
ext4, 1 × 64 KiB WRITE	218.6 → 193.7	206.9 → 205.6	331.0 → 544.0	213.4 → 102.5

Rust 1.85: four logical CPUs (AMD EPYC 7763), Ubuntu 24.04, Linux 7.0.0-1017-azure, rustc 1.85.0 (4d91de4e4 2025-02-17), GCC 13.3.0.

Workload	OrbiTE p50 µs before → after	TENT p50 µs before → after	OrbiTE p99 µs before → after	OrbiTE CPU µs before → after
tmpfs, 1 × 4 KiB READ	45.3 → 46.5	80.9 → 77.5	87.6 → 84.2	49.6 → 53.3
tmpfs, 1 × 4 KiB WRITE	192.8 → 76.1	80.1 → 79.3	325.8 → 177.0	181.3 → 86.6
tmpfs, 1 × 64 KiB READ	58.5 → 58.0	88.5 → 88.3	157.2 → 156.1	63.4 → 61.9
tmpfs, 1 × 64 KiB WRITE	207.8 → 99.2	111.4 → 112.4	276.8 → 197.2	199.1 → 106.9
ext4, 1 × 4 KiB READ	272.5 → 148.6	190.9 → 172.2	378.9 → 419.2	202.3 → 97.4
ext4, 1 × 4 KiB WRITE	282.8 → 171.5	186.6 → 191.0	422.3 → 439.5	246.5 → 116.2
ext4, 1 × 64 KiB READ	370.4 → 259.1	326.0 → 376.2	469.1 → 468.9	215.5 → 122.7
ext4, 1 × 64 KiB WRITE	342.0 → 232.7	352.7 → 320.1	527.8 → 464.9	267.2 → 140.9

Tmpfs 4 KiB WRITE p50 falls 51% on stable and 61% on 1.85; CPU falls 53%/52%. After p50 is 1.02×/0.96× TENT, respectively. This removes the measured ordinary small-WRITE handoff gap on these runs, not every conditional handoff.

Regressions remain, with every case reviewed in the linked tables:

• Stable ext4 64 KiB WRITE p99 rises 331→544 µs (+64.4%), with separate three-trial ranges (284–403→513–618 µs), while p50 and CPU fall. TENT’s unchanged p99 also rises 805→1322 µs (+64.4%), with overlapping ranges. A CQE that misses the 200 µs window can pay that wait before worker handoff; this is a bounded latency tradeoff, not a uniform p99 improvement. The relative contribution of the wait and VM/device variation is not isolated.

• Other changed-path stable regressions are ext4 4 KiB WRITE p99 +57.6%, ext4 64 KiB READ p99 +12.1%, tmpfs 64 KiB READ p99 +21.0%, and tmpfs 4 KiB READ p50/CPU +9.3%/+4.3%. Their ranges overlap. TENT p99 moves +8.0%, +242.8% and −1.0% for the first three cases; TENT small READ p50/CPU moves −8.5%/−3.4%. Those reference movements do not explain away the regressions.

• On 1.85, ext4 4 KiB READ/WRITE p99 rises 10.6%/4.1%; tmpfs 4 KiB READ p50/CPU rises 2.6%/7.5%, with throughput down 6.6%. All those ranges overlap. TENT ext4 READ p99 is nearly unchanged and WRITE p99 rises 0.5%; its tmpfs READ p50/CPU moves −4.2%/−4.7%. Changed wake timing remains a possible cost.

• 101/160 stable and 95/160 1.85 cases have at least one numerically worse statistic. Only five/three are eligible caller-wait cases. Larger file groups, TCP, HP-TCP, SHM, memcpy and TENT payload code is unchanged. Every slower case lists its trial-range overlap and the unchanged TENT movement where that baseline exists. Sequential-run cache, scheduling, VM and shared code-generation effects are not causally separated. Overlap is descriptive, never proof of noise or a reason to discard a row. No virtual-disk tail investigation or tuning is added here.

The two toolchain jobs are separate VMs and are not pooled. p99 uses only 100 samples per trial. The timed WRITE is not a durability receipt. These results support only this CPU cloud runner and workload; no GPU, RNIC or physical-NVMe gain is claimed. A request that misses the bounded window still needs worker drain, and the virtual-disk tail remains a P11 hardware-batch limit.

HP-TCP watch item recheck

Run 42501030618 separately repeats all 480 checked rows before and after the TCP-pool change, f0556448 → edec7937, with the same S26 parameters. Each pair shares one VM; stable uses four EPYC 9V74 logical CPUs and 1.85 uses four EPYC 9V45 logical CPUs. These are separate from the caller-wait pairs above and are not pooled.

HP-TCP 1 MiB READ	p50 µs before → after	p99 µs before → after	CPU µs before → after
Stable	405.8 → 458.3 (+12.9%)	596.2 → 581.9	647.7 → 648.9
Rust 1.85	352.2 → 280.7 (−20.3%)	487.9 → 360.0	493.3 → 434.8

All four statistics, including throughput, have overlapping trial ranges in both pairs. Stable repeats the direction of the original +17.3% watch item, but its CPU cost is flat and p99 falls. The HP-TCP profiles at edec7937 show zero warmed connects, one batch-thread creation, 13.2/14.9 futex calls and 7.4/7.8 epoll waits per batch on stable/1.85. Pattern generation occupies 68%/78% of user-stack samples and is outside timing; no shared control-worker frame reaches 0.5%. There is no measured extra payload work or connection setup to remove. Wake timing and VM/cache scheduling can change the median without more CPU work; these profiles do not isolate their contribution or rule out a shared control-worker effect. The watch item is retained, not declared fixed. This PR changes no HP-TCP owner; its own paired run records READ p50 changes of −3.6%/−17.8%, also with overlapping ranges.

Ext4 tail diagnosis

The old harness truncated and rewrote the file through the page cache before every sample without syncing the reset. S26, the TCP/small-file tables above and the unchanged-code control below therefore include reset interference in their ext4 timings; they do not isolate transfer cost. Both harnesses now preallocate once and finish an in-place pwrite plus fdatasync before each window. The production owners, timed operations and byte/gap/guard checks are unchanged. Buffered reset was chosen to keep verification simple while moving its writeback and allocation work outside the measured interval.

Paired trace run 42536940169 compares old harness 521861db with fixed harness 19ec2009; both have the production sources of 4f6bd35c, against frozen TENT e1ea5d31. Each phase runs twenty implementation/filesystem/direction/shape cases, with 1,000 byte-checked samples and three warmups each. Each old/fixed pair shares one four-vCPU Ubuntu 24.04 / Linux 7.0.0-1017-azure VM: stable (Rust 1.99.1) uses EPYC 9V74, and Rust 1.85 uses EPYC 9V45. These are different hosts; do not pool their absolute times.

benches/run.py --tail-profile-only records monotonic sample windows and system-wide io_uring, block-I/O, thread-creation, fdatasync and scheduler tracepoints. The collector writes to tmpfs and copies evidence to ext4 only after each benchmark exits. Raw perf data, event formats, commands and windows remain in cpu-bench-* artifacts. Run python3 benches/analyze_io_tail.py "$OUT/orbite-profiles/tail" for per-sample submit/first-CQE/last-CQE/return, worker/caller wake and block timelines.

The analyzer pairs device/flags/sector/length receipts across the whole capture, including writes issued before a window that finish inside it. It counts same-device traffic from other processes too. Direct candidates must match the benchmark process, requested operation and synchronous block flags after SQE submission, and their total bytes must equal the payload. Missing ring/CQE or scheduler receipts and mismatched direct bytes fail the analysis. Block events have no SQE identity: this is correlation, not an invented per-SQE receipt. Extra traffic is reported without assuming that all of it belongs to the benchmark. Prefix-flush and FUA flags are counted separately; the analyzer has negative controls for both and for carry-in writes.

Each row below covers 8,000 ext4 windows per phase (4,000 READ and 4,000 WRITE). Counts are old → fixed. Extra writes exclude the measured direct payload; carry-in means the write was already pending at window start.

Toolchain / implementation	Extra writes during READ	Extra writes during WRITE	Carry-in writes, both directions	Flushes / FUA, both phases
Stable OrbiTE	6,001 → 1	6,001 → 1	12,000 → 1	0 / 0
Stable TENT	6,001 → 1	6,000 → 0	12,000 → 0	0 / 0
Rust 1.85 OrbiTE	6,000 → 0	6,001 → 0	12,000 → 0	0 / 0
Rust 1.85 TENT	6,000 → 2	5,999 → 0	11,999 → 0	0 / 0

The old reset overlaps 31,999 of 32,000 measured ext4 windows. After the fix, no reset writeback overlaps a window. The five remaining writes are 4 KiB each from other processes (three stable, two MSRV); one stable write straddles window start. There are three discard requests per fixed phase, zero flush/FUA requests and zero unmatched overlapping issues. This confirms a measurement artifact in the old harness, but does not explain every tail.

The following values are diagnostic traces, not benchmark p99s: each is the median of the slowest ten samples in its 1,000-sample case. CQE and block completion mean kernel events, not physical hardware timestamps. Runnable time covers observed wake/preemption intervals after the issuer first runs; initial thread creation/scheduling is in the before-first-SQE timeline.

64 × 4 KiB gapped	Batch ms, old → fixed	Fixed longest block wait ms	Fixed issuer blocked / runnable µs	Fixed last block → last CQE µs
Stable OrbiTE READ	30.30 → 29.11	28.41	28,185 / 61	17.2
Stable OrbiTE WRITE	75.46 → 23.79	23.06	23,160 / 9	20.6
Stable TENT READ	101.27 → 6.60	6.00	5,523 / 131	6.3
Stable TENT WRITE	4.98 → 4.32	3.59	3,538 / 66	15.7
Rust 1.85 OrbiTE READ	30.01 → 32.30	31.72	31,433 / 24	4.9
Rust 1.85 OrbiTE WRITE	5.97 → 31.81	31.31	31,252 / 8	10.0
Rust 1.85 TENT READ	26.22 → 24.98	24.13	23,257 / 368	6.1
Rust 1.85 TENT WRITE	4.67 → 4.19	3.16	3,411 / 61	15.1

In particular, the fixed MSRV OrbiTE gapped WRITE case has no extra writes, flushes, FUA or discards in any of its 1,000 windows, yet its long tails contain 31 ms block waits. Thus the clean reset removes a proven confounder; it does not establish that the whole S26 tail or the implementation gap was a harness artifact. The residual direct-I/O tail remains open. No production tuning or physical-SSD claim is justified by these traces.

The proposed engine explanations can be narrowed:

• Every measured gapped batch uses one ring, 64 SQEs and one submitting enter in both implementations. In the fixed slow-tail samples, OrbiTE uses a median 2–4 enters including zero-submission waits; TENT uses one submitting enter and polls the mapped CQ. These waits never resubmit an SQE.

• All 64-block CQEs are published on the submitting batch worker. There is no caller-to-worker ring handoff in this case. Worker drain has zero nanosleeps; the benchmark caller still uses its documented status backoff.

• Aligned gaps allocate no bounce buffer. The new aligned_file_batches_use_one_ring_without_bounce_for_gaps_or_contiguous_ranges control fails every bounce allocation and checks one ring, full bytes and guards for READ/WRITE, 8/64 blocks, contiguous/gapped, ext4/tmpfs.

• OrbiTE creates one batch worker per multi-request submission; TENT creates none. Before-first-SQE time in OrbiTE’s slow 64-block samples is 105–148 µs in the fixed paired traces, much shorter than the block wait. Removing thread creation cannot by itself explain or eliminate the measured millisecond tail.

These historical traces also isolated the small tmpfs WRITE handoff: the median sample created a batch worker after caller submission, versus zero for READ and TENT. Diagnostic WRITE medians were 137/118 µs (stable/MSRV), versus TENT’s 81/59 µs; they are not untraced S26 results. The small-file paired pass now measures a bounded caller CQ wait for the existing eligible request. Expiry still hands the same owner to the worker and is never a release receipt. The multi-request direct-I/O owner and the historical tail diagnosis remain unchanged.

Direct-I/O issue paths and verification load

The block-layer comparison uses production 82cf6cef with diagnostic harness 2fba491d. The counterfactual run uses dca107d, with the same production tree, against frozen TENT e1ea5d31. Each diagnostic case has three warmups and 1,000 byte-checked samples. The latter run uses four-vCPU Ubuntu 24.04 / Linux 7.0.0-1017-azure VMs: stable Rust 1.99.1 on EPYC 9V45, Rust 1.85 on EPYC 7763. The initial MSRV trace and stable counterfactual expose MSFT NVMe Accelerator v1.0, with a 256 KiB request limit; the counterfactual MSRV job exposes Msft Virtual Disk (sda), with a 2 MiB limit. Both report 512-byte logical / 4 KiB physical sectors and the none scheduler. Each within-job engine comparison shares its device; the jobs are distinct virtual disks, not installed NVMe qualification.

Both implementations open the payload descriptor with O_RDWR|O_DIRECT, without O_DSYNC or O_SYNC; OrbiTE also uses O_CLOEXEC. The separate OrbiTE buffered descriptor’s O_NONBLOCK is absent from the direct descriptor. strace records opens and subsequent fcntl calls. Plain ring setup, SQEs, file offsets and device requests agree. For each gapped 64-block WRITE:

Boundary	OrbiTE	Frozen TENT
SQE submission	Existing batch worker, one enter	Caller, one enter
Ring / SQEs	One / 64 WRITEV	One / 64 WRITEV
File offsets / lengths	4 KiB + request index × 8 KiB / 4 KiB	Same
Kernel issue	64 NOWAIT direct attempts, then 64 io-wq direct issues	Same
Bios / alignment	64 × 4 KiB, 4 KiB-aligned sectors, stride 16 sectors	Same
Merges / unplug width	Zero merges; 64 unplugs of one request	Same
Median of per-batch maximum driver-issue depth, initial MSRV run	9	10
Cached folios invalidated after reset	49: orders 0–2	14: orders 0–5
Untimed verification reads, per sample	32 × 4 KiB plus one large read	5 × 4 KiB plus five larger reads

The first repeatable divergence is verification before the next sample. Rust reads the whole file; the earlier C++ harness used an istreambuf_iterator with small buffered reads. Their readahead and cached-folio layouts differ even though both reset in place and finish fdatasync. In the initial MSRV trace, median between-window time is 1.72 / 2.81 ms (OrbiTE / TENT). All measured WRITEs are issued before the long waits: OrbiTE’s slowest-1% median last issue is 0.82 ms, while the longest block wait is 29.68 ms. No delayed submit, extra SQE, bounce allocation, flush/FUA or ownership wait explains that gap. The trace’s queue depth is this batch’s outstanding driver requests, not hardware-wide queue depth. Physical disk service is not visible to the guest.

Two controls leave production and the timed operation unchanged. A bulk-read C++ verifier preserves every byte/guard check and reproduces the long tail in TENT. An additional untimed 10 ms pause after reset reduces offered load. Numbers below are medians of the slowest ten traced samples, in milliseconds; they are not untraced benchmark p99s. Stable / Rust 1.85 stay separate.

64 × 4 KiB WRITE	Original verifier	TENT bulk verifier only	Untimed pause only
OrbiTE gapped	30.78 / 34.16	—	1.54 / 2.58
TENT gapped	3.15 / 3.35	29.83 / 28.79	2.83 / 3.29
OrbiTE contiguous	26.58 / 19.71	—	17.41 / 3.00
TENT contiguous	25.61 / 12.89	25.01 / 17.48	1.88 / 2.77

A reversed-order run on source fc3c2a5 runs bulk verification first and the legacy verifier later. TENT gapped tail medians are 27.03 / 30.12 ms with bulk and 3.28 / 3.39 ms with legacy verification. The effect persists in both orders.

The original TENT bulk-control longest block waits are 28.45 / 27.48 ms. Thus an unchanged TENT owner exhibits the purported OrbiTE-specific gapped tail when only verification changes. Lower offered load reduces it without replacing OrbiTE’s worker, ring or drain; it does not eliminate every outlier (notably stable contiguous OrbiTE). Periodic stalls after driver issue are consistent with virtual-disk rate limiting, but the host’s quota/controller is not exposed: that mechanism is an inference, not a measured throttle receipt. The evidence supports a verification/load-dependent comparison and a limit below guest issue, not a production ownership fix or a general storage speed claim.

Both normal harnesses now verify files with a bulk read. The legacy C++ verifier and pacing are diagnostic controls only; benches/run.py strips all inherited ORBITE_BENCH_* settings and records explicit diagnostic overrides. No pacing is added to the normal S26 sweep. The complete paired report records every row with the original trials, samples, warmup and byte checks. Use benches/compare.py BEFORE AFTER --changed-profile none --verification-comparison: the explicit mode permits the C++ verification transition only, keeps reset and Rust-workload checks, and labels affected TENT file rows V. TENT CPU/TCP rows remain noise references; changed TENT file rows do not. The clean-reset numbers below remain evidence for the earlier verification regime, not proof that OrbiTE issues intrinsically slower direct WRITEs. Current virtual-disk tails remain a P11 limit for the installed NVMe batch. The bounded small-WRITE wait is measured separately above; a missed window still needs worker drain.

Reset-only paired comparison

The untraced paired run passes all 480 old and 480 fixed rows on each toolchain, 521861db → 19ec2009. Each pair uses one four-vCPU EPYC 7763 VM, Ubuntu 24.04 / Linux 7.0.0-1017-azure, GCC 13.3.0, and stable Rust 1.99.1 or Rust 1.85.0. The original S26 three trials, 100 samples, three warmups, two configured workers, sizes/shapes and byte checks are unchanged. Production sources are identical. The complete stable table and 1.85 table include every ext4 row for both implementations, CPU/throughput and every numerical regression. Raw results remain outside Git.

These are medians of the three untraced trial p99s, in milliseconds:

Toolchain / 64 × 4 KiB shape	OrbiTE p99 old → fixed	TENT p99 old → fixed
Stable gapped READ	15.60 → 10.41	2.04 → 1.74
Stable gapped WRITE	8.08 → 32.98	1.67 → 1.82
Stable contiguous WRITE	1.09 → 14.09	1.56 → 3.97
Rust 1.85 gapped READ	17.17 → 10.63	2.44 → 1.59
Rust 1.85 gapped WRITE	10.83 → 25.98	2.67 → 1.76
Rust 1.85 contiguous WRITE	1.02 → 18.00	1.39 → 6.18

The reset artifact is removed, but the tail and implementation gap do not largely disappear. WRITE tails worsen, with nonoverlapping OrbiTE trial ranges in both shapes/toolchains; contiguous TENT WRITE worsens too. These regressions remain open. Small tmpfs WRITE p50 is 194→203 / 194→195 µs (stable/MSRV), versus TENT after 81/79 µs. HP-TCP 1 MiB READ p50 moves 605→548 / 548→588 µs; the MSRV increase has nonoverlapping ranges. No watch item is declared fixed.

Stable/MSRV have 110/114 of 160 cases with a worse statistic; 71 in each pair have all four trial ranges overlapping. Non-file rows do not execute reset and remain run/build-placement controls; their clock/cache/VM contributions are unisolated. File preparation intentionally changes allocation/cache state and I/O timing in both harnesses, so TENT file rows are not pure noise controls. Neither those controls nor range overlap explain away the file gap; range overlap is not a significance test. This is a measurement correction, not an engine speedup.

Reproduce with benches/compare.py BEFORE AFTER --changed-profile none --reset-comparison. The flag permits exactly the documented reset transition and changed harness hashes; all other workload/host/CPU checks remain. Instantaneous clock readings are reported separately, not normalized.

Unchanged-code paired control

The paired control run uses the old harness, 7e40c7b → 23f6911, on each toolchain: identical production sources and workload hashes, with the ordinary S26 three trials, 100 samples and byte checks. All 480 rows per side passed. The complete stable and 1.85 tables include every slower row. Stable uses four EPYC 7763 logical CPUs; 1.85 uses four Xeon 6973P-C logical CPUs. Each pair shares one VM with Ubuntu 24.04 / Linux 7.0.0-1017-azure. In the 1.85 pair, only the instantaneous lscpu scaling reading differs (116% → 154%); model, topology, flags and frequency limits match. Frequency was not locked or used to normalize times. Reproduce them with benches/compare.py BEFORE AFTER --changed-profile none.

Stable medians of the three trial statistics; all ext4 rows below use 4 KiB blocks:

Workload	OrbiTE p50 µs before → after	OrbiTE p99 µs before → after	TENT p99 µs before → after
ext4, 64 gapped READ	2,220.5 → 1,652.3	5,424.5 → 6,536.6	4,335.6 → 5,419.0
ext4, 64 gapped WRITE	853.6 → 842.5	2,517.7 → 4,486.7	1,762.1 → 2,851.6
ext4, 8 gapped READ	—	1,582.4 → 777.8	1,077.4 → 919.8
ext4, 8 gapped WRITE	—	643.4 → 681.9	891.2 → 1,443.6
ext4, 8 contiguous READ	—	833.8 → 803.1	986.0 → 819.1
ext4, 8 contiguous WRITE	—	757.7 → 673.7	593.9 → 755.4

Rust 1.85’s gapped 64-block READ/WRITE p99 changes 16.36→15.28 / 4.54→4.41 ms; TENT changes 1.35→3.40 / 2.15→2.28 ms. Stable tmpfs READ/WRITE p50 changes 52.5→48.2 / 198.9→190.8 µs, versus TENT after 83.9/83.5 µs: the WRITE handoff gap remains. HP-TCP 1 MiB READ p50 changes 543.6→541.0 µs on stable and 610.3→583.8 µs on 1.85; this does not establish a fix to its watch item.

All payloads are unchanged. Stable has 141/160 cases with at least one worse statistic, 105 with overlapping ranges for all four statistics; 1.85 has 104/160 and 84. Not all changes track TENT: stable single-block ext4 READ p99 rises 157% while TENT falls 5%, with nonoverlapping OrbiTE trial ranges. Conversely, contiguous 64-block READ p50 rises 45% while TENT rises 57%. The unchanged 1.85 SHM contiguous 64-block WRITE p50 rises 56%, despite no file I/O. These are run/build-placement variations, not optimization results; their clock, cache and VM contributions are not isolated. Every regression remains flagged in the tables. Range overlap is descriptive, not a significance test. Raw results remain outside Git; no physical-SSD claim follows.

CPU fault matrix

tests/faults/matrix.tsv is the executable fixture list: backend, fault, exact production-owner control and expected charge/release state. It reuses the existing socket, file, mapping and fallible-table controls. New TCP and HP-TCP cases kill a real requester process after a partial WRITE. TCP leaves the destination untouched; HP-TCP keeps the received prefix. Both retain the registration during unregister and close, then verify all bytes and guards after physical socket drain. The restart fixture keeps its reserved endpoint and bounds only setup-time AddrInUse retries: concurrent process creation can inherit a listener until exec closes it. Transfer deadlines, epoch rejection and drain assertions stay unchanged.

The matrix covers peer death, timeout, cancellation, partial submission, unregister/close during work, and restart authority. It distinguishes native socket failures from injected completion faults. io_uring uses real files/rings. A missing required kernel facility fails, never silently skips. SHM’s synchronous memcpy has no native cancel or partial-post queue; queued cancellation belongs to the common batch owner. Local files have no peer epoch: path replacement is tested against the admitted inode. Those dispositions are explicit, not counted as passes.

cargo test --locked --features io-uring --lib --no-run --message-format=json > "$OUT/build.json"
# BINARY is the compiler-artifact executable in build.json.
ORBITE_EXT4_DIR="$EXT4_DIR" ORBITE_TMPFS_DIR=/dev/shm \
  python3 tests/faults/run.py --binary "$BINARY" --output "$OUT/faults" --kill-repeats 50

CI executes the matrix on both toolchains with the full feature set. It repeats each new process-kill control 50 times. Existing native fault/stability gates remain in the matrix. This prepares P10; it does not close its installed hardware acceptance matrix.

Installed fault fixtures

The verification queue owns the exact commands and hardware leases. Reuse the installed RDMA fault runner’s installed-inflight-dma, installed-lost-completion, installed-cleanup-failure and restart modes: accepted posts remain charged until matching retirement receipts, and failed cleanup retains the native owner. The NVLink grouped requester-exit and graceful-close modes check cooperative CUDA import/event lifetime and guarded bytes. Exporter crash and physical device loss remain explicit limits, not simulated passes.

The additional gds-p10 packaged consumer runs one real native READ while a small, test-only cuFile observer withholds status receipts. drain checks a wait timeout and concurrent unregister/file close, then releases real CQEs and verifies the full payload and guards. quarantine withholds receipts past the submission deadline plus destruction window and checks Unquiesced, retained byte charges and GPU backing after every caller handle is dropped. It never fabricates a completion or retries a WRITE. The driver requires compatibility disabled, resource locks and matching nvidia-fs READ-counter evidence. Both modes are written and compiled in CI, not run. These are controlled receipt faults, not spontaneous device faults.