# Benchmarks and fault fixtures

The S26 tools measure verified CPU transfers and prepare the P10 lifetime
matrix. They change no transfer policy. Results describe the recorded host and
artifact; they are not hardware qualification or a general performance claim.

## CPU sweeps

[benches/run.py](https://github.com/OrbitInfra/OrbiTE/blob/main/benches/run.py)
runs the public Rust API and a test-only C++ caller of frozen Mooncake v0.3.14.
The C++ runtime is confined to the reference executable. CPU memcpy, ordinary
TCP and io_uring have matched rows; HP-TCP and SHM have OrbiTE-only rows. HP-TCP
uses `127.0.0.1` and `127.0.0.2` as two loopback rails.

The default workload is READ and WRITE at 4 KiB, 64 KiB, 1 MiB and 64 MiB, plus
batches of 8 and 64 four-KiB blocks. Contiguous memory-target blocks exercise
physical-owner merging. Gapped blocks use a stride of twice the block size and
must remain separate owners. File batches preserve one owner per public request
for independent CQE outcomes, including contiguous blocks; the runner checks
that mapping too. There is one caller, two configured workers per engine,
three warmup batches, 100 samples and three independent process trials. Trial
order alternates the two implementations. Both disable engine failover,
metrics, progress workers and runtime queues, and enable merging. TENT's
transport-local TCP retry defaults are pinned separately: three retries,
100 ms initial delay and a 2,000 ms cap. OrbiTE has no corresponding inner
TCP retry loop. These are healthy-path measurements, not retry benchmarks.
The manifest records
these settings, actual process thread counts, compiler versions, affinity,
source and binary hashes, filesystems and every command. Internal worker
arrangements differ: these are matched caller/worker settings, not identical
engine implementations. TENT's common and TE RPC client pools are each pinned
to the same worker count through their two explicit environment variables.

Each iteration changes the source pattern and resets the destination. Every
payload byte, gap and guard is checked after **every** transfer, including
warmup; corruption aborts without a result. The runner also corrupts a guard
in each implementation and checks rejection. It rejects missing checks,
wrong byte counts and incomplete/nonpositive samples. There is no unchecked
mode.

Timing covers batch allocation, submission, the same polling schedule (128
hot polls, then exponential sleeps capped at 200 microseconds), and batch free.
OrbiTE additionally checks local drain. Setup, registration, reset and full
verification are outside the interval. CPU time uses the whole process clock,
including both engines' threads during that interval. Throughput divides
verified payload bytes by the sum of measured batch times. p50 and nearest-rank
p99 are empirical per-batch latency, not per-block latency; 100 samples give
limited tail resolution. The CPU memcpy row is only a memory-copy control,
not an engine submission. Raw samples are retained so results can be recomputed.

File offsets, lengths and buffer addresses are four-KiB aligned. The runner
requires actual ext4 and tmpfs mounts and checks OrbiTE's descriptor route.
Both file profiles leave TCP installed for host registration (TENT does not
include IOURING in its memory-registration transport list), but requests carry
an explicit io_uring hint. TENT attempts O_DIRECT on both; on tmpfs the flag may be accepted without a
direct path. Both harnesses open and preallocate the file once, reset it in
place with `pwrite`, and finish `fdatasync` before the timing window. Buffered
reset keeps verification simple without leaving dirty reset pages or allocation
work for the measured direct I/O. Payload, gap and guard checks are unchanged.
These are warmed measurements, **not cold SSD throughput**. Timed WRITE
completion still excludes a durability barrier. Maximum payload per batch is 64 MiB; the
allocation and guard extent is bounded separately.

On a Linux runner with the native dependencies from the
[CPU oracle](cpu-oracle.md), also install `liburing-dev` **before** building
TENT. Enable io_uring in the kernel and choose writable ext4 and tmpfs paths:

```bash
python3 tests/reference/run.py --fetch-reference --output "$OUT/reference"
python3 benches/run.py --reference "$OUT/reference" --output "$OUT/sweep" \
  --ext4 "$EXT4_DIR" --tmpfs /dev/shm --workers 2 --warmup 3 \
  --samples 100 --trials 3
```

Both output directories must be fresh and outside the checkout. The
`Frozen TENT CPU comparison` CI workflow runs this on stable and Rust 1.85 and
uploads `cpu-bench-*` artifacts. GPU, RDMA and GDS are emitted as
**not run (hardware)**, without estimated numbers. The scoped summary lives in
[qualification](qualification.md#cpu-benchmark-evidence).

## TCP profile and connection reuse

The pre-change profile is [run 42439141061](https://github.com/OrbitInfra/OrbiTE/actions/runs/42439141061),
`cpu-oracle (stable)`, revision `ec103db` (production sources unchanged from
`f0556448`). Its host `runnervm7aw5e` exposes four AMD EPYC 7763 logical CPUs,
Ubuntu 24.04, Linux 7.0.0-1017-azure and Rust 1.99.1. The reference is
`e1ea5d31bb5e46ff2df9f90e7c2fbaf52fcdee8c`. Both toolchains passed all sixteen
profile cases with byte checks. These are diagnostic observations, not timings
used in the comparison table.

`benches/run.py --profile-only` uses the same executables, sizes, layouts,
workers and warmup as S26. `strace -f -c -w` counts 100 samples minus a one-sample
setup control, divided by 99. `perf record` samples user CPU at a 10-microsecond
period with DWARF stacks and 1,000 samples per workload: millisecond sampling
misses short-lived batch threads. `perf stat` separately records process CPU,
context switches and migrations. Raw traces, call stacks and exact commands
remain in the workflow's `cpu-bench-*` artifacts, outside Git. Profiling includes
verification/setup; latency measurements below exclude them.

| Baseline calls per batch (READ / WRITE) | OrbiTE | TENT | Interpretation |
|---|---:|---:|---|
| TCP connect, single 4 KiB | 1 / 1 | 0 / 0 | TENT reuses warmed connections |
| TCP connect, gapped 64 × 4 KiB | 64 / 64 | 0 / 0 | One handshake per physical request is confirmed |
| TCP nanosleep, gapped 64 × 4 KiB | 219 / 236 | 129 / 128 | Traced polling sleeps include the common benchmark poll loop |
| TCP futex, gapped 64 × 4 KiB | 429 / 431 | 481 / 469 | This is not a count of locks or thread handoffs |
| Batch thread creation, every OrbiTE target | 1 / 1 | 0 / 0 | One caller→new batch-worker handoff, retained for a later PR |
| tmpfs ring setup / enter, single 4 KiB | 1 / 1 | 1 / 1 | No extra ring is created per request |
| tmpfs futex, single 4 KiB | 13.7 / 13.0 | approximately 0 / 0 | Publication, progress and worker retirement remain measurable |
| ext4 ring enter, gapped 64 × 4 KiB | 1.14 / 1.11 | 1 / 1 | Reaping can need another enter |
| ext4 statx, gapped 64 × 4 KiB | 65 / 65 | 0 / 0 | Per-request file/descriptor checks remain; no safety checks removed |

The single TCP READ's user samples attribute 18.2% to `recv`, 14.4% to
`sched_yield`, 12.1% to the main VDSO clock frame and 10.2% to
`TcpTransport::transfer`. For gapped READ/WRITE, `recv` is 9.9%/10.2%,
`sched_yield` 9.5%/9.2%, and `ControlChannel::poll_read` 2.1%/2.3%.
Pattern generation/verification accounts for another 17% and is outside the
benchmark interval. This supports removing direct TCP's hot-poll/backoff loop
as well as reconnects. No admission or loan frame reaches 1% of these samples;
inlining and sample size prevent treating that as proof of zero bookkeeping
cost. Those owners and checks stay unchanged.

The tmpfs READ profile instead attributes 23.0% to `Batch::status`, 4.6% to
`ProgressState::status`, 19.4% to the VDSO clock frame and 2.0% to batch
publication. The file path still creates a worker and publishes completion to
the polling caller. Ext4 READ samples include 1.2% `statx`, 0.7% selector and
0.5% preparation; pattern generation alone is 46.1%. Aligned ext4 route checks
pass, so these runs do not implicate an accidental buffered fallback. They do
not isolate the historical p99 tail's cause; storage and VM scheduling need
a separate pass.

Waiting and working cannot be inferred by subtracting multi-thread process CPU
from wall latency. In the separate `perf stat` run, mean wall / process CPU per
batch was 285 / 398 microseconds for one TCP READ, 8,861 / 16,408 for gapped TCP
READ, 174 / 176 for tmpfs READ, and 1,960 / 1,053 for gapped ext4 READ.
Under `strace`, summed sleep / futex / epoll wall time for gapped TCP READ was
63.9 / 249.2 / 123.4 ms per batch, versus 3.0 / 46.4 / 21.3 ms for gapped ext4
READ (plus 0.39 ms in `io_uring_enter`). These include concurrent idle threads
and tracer perturbation, **not critical-path percentages**. The CPU stack
samples distinguish hot polling from sleeping; the untraced paired sweeps
quantify the effect on actual batch latency and CPU cost.

The same `perf stat` processes report 23,674 / 30,312 context switches for
OrbiTE's gapped TCP READ / WRITE, versus TENT's 46,849 / 46,749. Each process
includes setup, three warmups and 100 samples. That is approximately
230 / 294 versus 455 / 454 switches per executed batch, including setup;
it is not an exact count of owner handoffs. The one new thread per OrbiTE
batch is the measurable caller-to-worker handoff. A syscall count alone
cannot assign every remaining scheduler switch to an owner.

The change follows frozen HP-TCP's clean/dirty lane ownership without adding a
worker: one exclusive epoch-scoped stream, deadline-bounded socket waits, and
source readiness monitoring in the existing control workers. Dirty streams
are retired, never replayed; admission, backing loans, per-request snapshots,
physical drain and sticky cleanup remain. The native frame bytes and offsets
are unchanged; a stream now carries successive complete TCP frames. See
[the owner contract](cpu-tcp.md) and its negative controls.

### Paired results and regressions

[Run 42441063520](https://github.com/OrbitInfra/OrbiTE/actions/runs/42441063520)
completed 480 checked rows **before and after on each toolchain**, comparing
`f0556448` with `de9fd60b`. Later commits change fixtures and reporting only.
Both sides of each pair share one GitHub `ubuntu-24.04` VM, compiler,
filesystems and parameters above. Stable (Rust 1.99.1) uses four EPYC 7763
logical CPUs, matching S26's processor class; Rust 1.85 uses four EPYC 9V45
logical CPUs. Both report hostname `runnervm7aw5e` and Linux 7.0.0-1017-azure;
that hostname alone does not identify a processor. Do not pool the toolchains'
absolute times. The order is before then after, with implementation order
alternating across the three trials, not randomized across revisions.

The stable results below are medians of the three trial statistics. The
[PR's complete tables](https://github.com/OrbitInfra/OrbiTE/pull/180) include
**every** workload/implementation case, throughput and every numerical
regression; all 480 raw trial rows per side remain in `cpu-bench-*` artifacts.
Reproduce those tables with `python3 benches/compare.py BEFORE AFTER`. It
rejects mismatched manifests, workload hashes, missing trials and unchecked data.

| Workload | OrbiTE p50 µs before → after | TENT p50 µs after | OrbiTE p99 µs before → after | OrbiTE CPU µs before → after |
|---|---:|---:|---:|---:|
| TCP, 1 × 4 KiB READ | 275.3 → 153.5 | 284.2 | 320.4 → 261.8 | 370.3 → 190.3 |
| TCP, 1 × 4 KiB WRITE | 277.0 → 160.6 | 281.6 | 339.3 → 288.2 | 385.2 → 215.2 |
| TCP, 64 × 4 KiB gapped READ | 8,591.0 → 4,231.4 | 6,175.9 | 9,133.9 → 4,556.8 | 15,908.6 → 5,449.1 |
| TCP, 64 × 4 KiB gapped WRITE | 9,331.6 → 6,296.2 | 6,071.6 | 9,856.6 → 6,641.5 | 16,946.6 → 7,055.2 |
| tmpfs, 1 × 4 KiB READ | 156.8 → 155.4 | 82.4 | 329.3 → 309.8 | 168.0 → 164.8 |
| tmpfs, 1 × 4 KiB WRITE | 194.4 → 194.2 | 82.6 | 326.5 → 305.1 | 247.4 → 243.9 |
| ext4, 64 × 4 KiB gapped READ | 926.5 → 933.2 | 1,018.8 | 16,517.1 → 17,418.5 | 1,107.6 → 1,041.9 |
| ext4, 64 × 4 KiB gapped WRITE | 873.3 → 892.3 | 917.5 | 7,269.4 → 8,969.2 | 1,096.4 → 1,116.3 |

The after profile confirms zero warmed TCP connects and zero `sched_yield`
calls per batch, versus 64 connects and 308/342 yields for scattered READ/WRITE.
Futex calls fall from 429/431 to 226/242; epoll waits rise from 67/67 to 75/77.
One batch-thread creation remains. Pattern generation now occupies 49%/48% of
the scattered user-CPU stacks, outside the measured interval; the very short
single-request after profiles have only 61/26 stack samples, so their individual
frame percentages are not a reliable ranking. The untraced CPU times above
are the evidence for the cost reduction.

The EPYC 9V45 pair independently reduces scattered READ/WRITE p50 by 46%/29%
and CPU by 64%/58%. Its resulting p50s are 2.26/3.50 ms versus TENT's
2.92/2.91 ms: WRITE remains 20% slower there. Regressions are not omitted:

- Stable TCP's 64 MiB WRITE p50 rises 0.3% (overlapping trial ranges), while
  its p99 and CPU improve. On 1.85, 64 MiB READ p50 rises 2.8%, p99 10.7%
  and throughput falls 3.2%, while CPU falls 4.2%. Blocking wakeups can trade
  latency for CPU on a large transfer whose handshake is already amortized;
  the contribution is not isolated, and the latency trial ranges overlap.
  This is not a claim of a large-transfer improvement.
- Untuned paths also vary. Stable HP-TCP 1 MiB READ p50 rises 17.3%, with
  overlapping ranges for all four statistics. Stable ext4 1 MiB READ p50
  rises 40.8% (nonoverlapping median ranges); unchanged TENT rises 23.1%
  on that case. Ext4 64-block contiguous WRITE p99 rises 51.7%, with overlapping
  trial ranges. On 1.85, ext4 eight-block gapped READ p99 rises 55.8%.
  Warmed page-cache/coherence and VM scheduling are plausible contributors;
  shared control-worker effects are not ruled out. The cause remains open.
- Every smaller regression is flagged in the complete tables. Memcpy and
  TENT controls do not execute the changed TCP pool; their changes show run,
  cache and scheduling variation. OrbiTE HP-TCP, SHM and file payload owners
  are unchanged, but their shared idle control worker changed. Range overlap
  is descriptive, not a significance test or permission to discard a result.

The standard [480-row run on both toolchains](https://github.com/OrbitInfra/OrbiTE/actions/runs/42441786027)
also passed, alongside [all feature gates and 50-repeat controls](https://github.com/OrbitInfra/OrbiTE/actions/runs/42441786080)
and the [installed-wheel matrix](https://github.com/OrbitInfra/OrbiTE/actions/runs/42441786046).
These are single-caller cloud-loopback and warmed-file results. No concurrent-caller,
physical two-host, GPU, RNIC or SSD performance claim follows. The small-file
pass below measures storage latency; ext4 tails and remaining bookkeeping cost
stay separate work.

## Small io_uring submission profile

The historical [ready-CQE pass](https://github.com/OrbitInfra/OrbiTE/pull/183)
reduced stable tmpfs 4 KiB READ p50 from 156 to 51 µs, but WRITE remained
193 µs versus TENT's 84 µs: one nonblocking check often missed the WRITE CQE
and started a batch worker. Its ext4 values used the old reset harness.

The [before profiles](https://github.com/OrbitInfra/OrbiTE/actions/runs/42691895658)
at `578531b7` and [after profiles](https://github.com/OrbitInfra/OrbiTE/actions/runs/42692901727)
at `a5874066` use the current preallocated, synchronized reset and bulk
verification. Before profiles use EPYC 7763; after uses EPYC 9V74 on stable
and EPYC 7763 on 1.85. These are separate diagnostic runs; the untraced
same-VM pairs below establish the timing comparison. Raw `perf` and `strace` output stays
in `cpu-bench-*` artifacts. Setup-subtracted calls per 4 KiB tmpfs batch are:

| Toolchain / operation | clone/clone3 before → after | futex before → after | io_uring_enter before → after |
|---|---:|---:|---:|
| Stable READ | 0.00 → 0.00 | 6.00 → 6.00 | 1.00 → 1.00 |
| Stable WRITE | 0.00 → 0.00 | 6.00 → 6.00 | 1.00 → 1.00 |
| Rust 1.85 READ | 0.00 → 0.00 | 6.00 → 6.01 | 1.00 → 1.00 |
| Rust 1.85 WRITE | 0.00 → 0.00 | 6.00 → 6.00 | 1.00 → 1.00 |

Both keep one ring setup per batch; the one lazy availability probe is outside
measurement. TENT likewise records no per-batch clone, one setup/enter and
approximately zero futex calls. Futex calls are not a handoff count. Tracing
delays the initial CQ check enough to hide worker creation even on the old
source, so these zero-clone traces cannot prove ordinary WRITEs avoided it.
The less intrusive before WRITE CPU profiles still attribute 1.75%/9.72% to
`pthread_create` on stable/1.85. Neither that frame nor `Batch::start_worker`
appears in the after
WRITE reports at the 0.5% cutoff. The 1.85 reports contain only 72 before and
118 after user-stack samples, so absence is not proof of zero calls. Its
separate `perf stat` WRITE sample changes p50/process-CPU from 197/193 to
86/97 µs; process context switches remain 253→225, including setup and warmup.
Blocking CQ waits still sleep and wake. These diagnostic runs do not replace
the untraced paired comparison.

The caller now submits one contiguous host request of at most 64 KiB on an
idle batch, checks the CQ once, then waits up to 200 µs, capped by the request
deadline. At most three interrupted/unready waits share that window. A CQE
finishes through the normal prefix and admission owners. A missed window or
enter error moves the same ring to the existing drain worker without resubmission.
Runtime queues, background progress, eligible fallback, CUDA/pinned producer
waits, vectors and larger groups retain their worker path. No ring pool,
thread pool or coordinator is added. The [owner controls](storage.md) preserve
preflight, partial-submit drain, exact short prefixes and quarantine. The
late-CQE tmpfs/ext4 control counts zero worker starts; disabling only the
wait makes it fail with one start. This is the deterministic handoff proof.

### Small-file paired results

[Run 42692901727](https://github.com/OrbitInfra/OrbiTE/actions/runs/42692901727)
completed **480 checked rows before and after on each toolchain**, comparing
`578531b7` with `a5874066`. Each pair shares one VM, compiler, ext4/tmpfs mounts,
workload hashes, reset and verification. Three trials, 100 samples, three
warmups, one caller and two configured engine workers are unchanged. Revision
order is before then after; implementation order alternates across trials.
The frozen reference is Mooncake v0.3.14 `e1ea5d31`. No diagnostic pacing is
used. Raw trials and profiles stay outside Git in the run's artifacts.

The complete [stable table](https://github.com/OrbitInfra/OrbiTE/pull/193#issuecomment-6752994908) and
[1.85 table](https://github.com/OrbitInfra/OrbiTE/pull/193#issuecomment-6752602865)
include all 160 workload/implementation cases per toolchain and every numerical
regression; each case aggregates three byte-checked trial rows. Reproduce with
`python3 benches/compare.py BEFORE AFTER --changed-profile io_uring-ext4 --changed-profile io_uring-tmpfs`.
CPU is process time per batch; the tables show medians of trial statistics.

Stable: four logical CPUs (AMD EPYC 9V74), Ubuntu 24.04, Linux
7.0.0-1017-azure, rustc 1.99.1 (58cf8f9c3 2026-10-13), GCC 13.3.0.

| Workload | OrbiTE p50 µs before → after | TENT p50 µs before → after | OrbiTE p99 µs before → after | OrbiTE CPU µs before → after |
|---|---:|---:|---:|---:|
| tmpfs, 1 × 4 KiB READ | 33.9 → 37.1 | 62.8 → 57.5 | 67.5 → 56.6 | 37.9 → 39.5 |
| tmpfs, 1 × 4 KiB WRITE | 117.9 → 57.3 | 59.7 → 56.3 | 196.7 → 151.9 | 130.2 → 61.5 |
| tmpfs, 1 × 64 KiB READ | 45.6 → 44.3 | 71.4 → 69.5 | 108.7 → 131.6 | 51.1 → 48.3 |
| tmpfs, 1 × 64 KiB WRITE | 128.1 → 70.0 | 132.9 → 127.0 | 193.9 → 166.8 | 148.6 → 73.6 |
| ext4, 1 × 4 KiB READ | 178.1 → 122.7 | 142.0 → 143.4 | 259.5 → 252.6 | 143.0 → 68.1 |
| ext4, 1 × 4 KiB WRITE | 200.2 → 144.9 | 197.0 → 199.7 | 275.6 → 434.3 | 182.7 → 85.1 |
| ext4, 1 × 64 KiB READ | 193.7 → 150.1 | 208.2 → 204.4 | 239.2 → 268.0 | 148.6 → 75.9 |
| ext4, 1 × 64 KiB WRITE | 218.6 → 193.7 | 206.9 → 205.6 | 331.0 → 544.0 | 213.4 → 102.5 |

Rust 1.85: four logical CPUs (AMD EPYC 7763), Ubuntu 24.04, Linux
7.0.0-1017-azure, rustc 1.85.0 (4d91de4e4 2025-02-17), GCC 13.3.0.

| Workload | OrbiTE p50 µs before → after | TENT p50 µs before → after | OrbiTE p99 µs before → after | OrbiTE CPU µs before → after |
|---|---:|---:|---:|---:|
| tmpfs, 1 × 4 KiB READ | 45.3 → 46.5 | 80.9 → 77.5 | 87.6 → 84.2 | 49.6 → 53.3 |
| tmpfs, 1 × 4 KiB WRITE | 192.8 → 76.1 | 80.1 → 79.3 | 325.8 → 177.0 | 181.3 → 86.6 |
| tmpfs, 1 × 64 KiB READ | 58.5 → 58.0 | 88.5 → 88.3 | 157.2 → 156.1 | 63.4 → 61.9 |
| tmpfs, 1 × 64 KiB WRITE | 207.8 → 99.2 | 111.4 → 112.4 | 276.8 → 197.2 | 199.1 → 106.9 |
| ext4, 1 × 4 KiB READ | 272.5 → 148.6 | 190.9 → 172.2 | 378.9 → 419.2 | 202.3 → 97.4 |
| ext4, 1 × 4 KiB WRITE | 282.8 → 171.5 | 186.6 → 191.0 | 422.3 → 439.5 | 246.5 → 116.2 |
| ext4, 1 × 64 KiB READ | 370.4 → 259.1 | 326.0 → 376.2 | 469.1 → 468.9 | 215.5 → 122.7 |
| ext4, 1 × 64 KiB WRITE | 342.0 → 232.7 | 352.7 → 320.1 | 527.8 → 464.9 | 267.2 → 140.9 |

Tmpfs 4 KiB WRITE p50 falls 51% on stable and 61% on 1.85; CPU falls 53%/52%.
After p50 is 1.02×/0.96× TENT, respectively. This removes the measured ordinary
small-WRITE handoff gap on these runs, not every conditional handoff.

Regressions remain, with every case reviewed in the linked tables:

- Stable ext4 64 KiB WRITE p99 rises **331→544 µs (+64.4%)**, with separate
  three-trial ranges (284–403→513–618 µs), while p50 and CPU fall. TENT's
  unchanged p99 also rises 805→1322 µs (+64.4%), with overlapping ranges.
  A CQE that misses the 200 µs window can pay that wait before worker handoff;
  this is a bounded latency tradeoff, not a uniform p99 improvement. The
  relative contribution of the wait and VM/device variation is not isolated.
- Other changed-path stable regressions are ext4 4 KiB WRITE p99 +57.6%,
  ext4 64 KiB READ p99 +12.1%, tmpfs 64 KiB READ p99 +21.0%, and tmpfs 4 KiB
  READ p50/CPU +9.3%/+4.3%. Their ranges overlap. TENT p99 moves +8.0%,
  +242.8% and −1.0% for the first three cases; TENT small READ p50/CPU moves
  −8.5%/−3.4%. Those reference movements do not explain away the regressions.
- On 1.85, ext4 4 KiB READ/WRITE p99 rises 10.6%/4.1%; tmpfs 4 KiB READ
  p50/CPU rises 2.6%/7.5%, with throughput down 6.6%. All those ranges
  overlap. TENT ext4 READ p99 is nearly unchanged and WRITE p99 rises 0.5%; its tmpfs READ
  p50/CPU moves −4.2%/−4.7%. Changed wake timing remains a possible cost.
- **101/160 stable and 95/160 1.85 cases** have at least one numerically worse
  statistic. Only five/three are eligible caller-wait cases. Larger file
  groups, TCP, HP-TCP, SHM, memcpy and TENT payload code is unchanged. Every
  slower case lists its trial-range overlap and the unchanged TENT movement
  where that baseline exists. Sequential-run cache, scheduling, VM and shared
  code-generation effects are not causally separated. Overlap is descriptive,
  never proof of noise or a reason to discard a row. No virtual-disk tail
  investigation or tuning is added here.

The two toolchain jobs are separate VMs and are not pooled. p99 uses only 100
samples per trial. The timed WRITE is not a durability receipt. These results
support only this CPU cloud runner and workload; no GPU, RNIC or physical-NVMe
gain is claimed. A request that misses the bounded window still needs worker
drain, and the virtual-disk tail remains a P11 hardware-batch limit.

### HP-TCP watch item recheck

[Run 42501030618](https://github.com/OrbitInfra/OrbiTE/actions/runs/42501030618)
separately repeats all 480 checked rows before and after the TCP-pool change,
`f0556448` → `edec7937`, with the same S26 parameters. Each pair shares one VM;
stable uses four EPYC 9V74 logical CPUs and 1.85 uses four EPYC 9V45 logical
CPUs. These are separate from the caller-wait pairs above and are not pooled.

| HP-TCP 1 MiB READ | p50 µs before → after | p99 µs before → after | CPU µs before → after |
|---|---:|---:|---:|
| Stable | 405.8 → 458.3 (+12.9%) | 596.2 → 581.9 | 647.7 → 648.9 |
| Rust 1.85 | 352.2 → 280.7 (−20.3%) | 487.9 → 360.0 | 493.3 → 434.8 |

All four statistics, including throughput, have overlapping trial ranges in
both pairs. Stable repeats the direction of the original +17.3% watch item,
but its CPU cost is flat and p99 falls. The HP-TCP profiles at `edec7937` show
zero warmed connects, one batch-thread creation, 13.2/14.9 futex calls and
7.4/7.8 epoll waits per batch on stable/1.85. Pattern generation occupies
68%/78% of user-stack samples and is outside timing; no shared control-worker
frame reaches 0.5%. There is no measured extra payload work or connection
setup to remove. Wake timing and VM/cache scheduling can change the median
without more CPU work; these profiles do not isolate their contribution or
rule out a shared control-worker effect. The watch item is retained, not
declared fixed. This PR changes no HP-TCP owner; its own paired run records
READ p50 changes of −3.6%/−17.8%, also with overlapping ranges.

## Ext4 tail diagnosis

The **old harness** truncated and rewrote the file through the page cache
before every sample without syncing the reset. S26, the TCP/small-file tables
above and the unchanged-code control below therefore include reset interference
in their ext4 timings; they do not isolate transfer cost. Both harnesses now
preallocate once and finish an in-place `pwrite` plus `fdatasync` before each
window. The production owners, timed operations and byte/gap/guard checks are
unchanged. Buffered reset was chosen to keep verification simple while moving
its writeback and allocation work outside the measured interval.

[Paired trace run 42536940169](https://github.com/OrbitInfra/OrbiTE/actions/runs/42536940169)
compares old harness `521861db` with fixed harness `19ec2009`; both have the
production sources of `4f6bd35c`, against frozen TENT `e1ea5d31`. Each phase
runs twenty implementation/filesystem/direction/shape cases, with 1,000
byte-checked samples and three warmups each. Each old/fixed pair shares one
four-vCPU Ubuntu 24.04 / Linux 7.0.0-1017-azure VM: stable (Rust 1.99.1) uses
EPYC 9V74, and Rust 1.85 uses EPYC 9V45. These are different hosts; do not
pool their absolute times.

`benches/run.py --tail-profile-only` records monotonic sample windows and
system-wide io_uring, block-I/O, thread-creation, `fdatasync` and scheduler
tracepoints. The collector writes to tmpfs and copies evidence to ext4 only
after each benchmark exits. Raw `perf` data, event formats, commands and
windows remain in `cpu-bench-*` artifacts. Run
`python3 benches/analyze_io_tail.py "$OUT/orbite-profiles/tail"` for per-sample
submit/first-CQE/last-CQE/return, worker/caller wake and block timelines.

The analyzer pairs device/flags/sector/length receipts across the **whole
capture**, including writes issued before a window that finish inside it.
It counts same-device traffic from other processes too. Direct candidates
must match the benchmark process, requested operation and synchronous block
flags after SQE submission, and their total bytes must equal the payload.
Missing ring/CQE or scheduler receipts and mismatched direct bytes fail the
analysis. Block events have no SQE identity: this is correlation, not an
invented per-SQE receipt. Extra traffic is reported without assuming that all
of it belongs to the benchmark. Prefix-flush and FUA flags are counted
separately; the analyzer has negative controls for both and for carry-in writes.

Each row below covers 8,000 ext4 windows per phase (4,000 READ and 4,000 WRITE).
Counts are **old → fixed**. Extra writes exclude the measured direct payload;
carry-in means the write was already pending at window start.

| Toolchain / implementation | Extra writes during READ | Extra writes during WRITE | Carry-in writes, both directions | Flushes / FUA, both phases |
|---|---:|---:|---:|---:|
| Stable OrbiTE | 6,001 → 1 | 6,001 → 1 | 12,000 → 1 | 0 / 0 |
| Stable TENT | 6,001 → 1 | 6,000 → 0 | 12,000 → 0 | 0 / 0 |
| Rust 1.85 OrbiTE | 6,000 → 0 | 6,001 → 0 | 12,000 → 0 | 0 / 0 |
| Rust 1.85 TENT | 6,000 → 2 | 5,999 → 0 | 11,999 → 0 | 0 / 0 |

The old reset overlaps 31,999 of 32,000 measured ext4 windows. After the fix,
**no reset writeback overlaps a window**. The five remaining writes are
4 KiB each from other processes (three stable, two MSRV); one stable write
straddles window start. There are three discard requests per fixed phase,
zero flush/FUA requests and zero unmatched overlapping issues. This confirms
a measurement artifact in the old harness, but does not explain every tail.

The following values are **diagnostic traces**, not benchmark p99s: each is
the median of the slowest ten samples in its 1,000-sample case. CQE and block
completion mean kernel events, not physical hardware timestamps. Runnable
time covers observed wake/preemption intervals after the issuer first runs;
initial thread creation/scheduling is in the before-first-SQE timeline.

| 64 × 4 KiB gapped | Batch ms, old → fixed | Fixed longest block wait ms | Fixed issuer blocked / runnable µs | Fixed last block → last CQE µs |
|---|---:|---:|---:|---:|
| Stable OrbiTE READ | 30.30 → 29.11 | 28.41 | 28,185 / 61 | 17.2 |
| Stable OrbiTE WRITE | 75.46 → 23.79 | 23.06 | 23,160 / 9 | 20.6 |
| Stable TENT READ | 101.27 → 6.60 | 6.00 | 5,523 / 131 | 6.3 |
| Stable TENT WRITE | 4.98 → 4.32 | 3.59 | 3,538 / 66 | 15.7 |
| Rust 1.85 OrbiTE READ | 30.01 → 32.30 | 31.72 | 31,433 / 24 | 4.9 |
| Rust 1.85 OrbiTE WRITE | 5.97 → 31.81 | 31.31 | 31,252 / 8 | 10.0 |
| Rust 1.85 TENT READ | 26.22 → 24.98 | 24.13 | 23,257 / 368 | 6.1 |
| Rust 1.85 TENT WRITE | 4.67 → 4.19 | 3.16 | 3,411 / 61 | 15.1 |

In particular, the fixed MSRV OrbiTE gapped WRITE case has no extra writes,
flushes, FUA or discards in any of its 1,000 windows, yet its long tails contain
31 ms block waits. Thus the clean reset removes a proven confounder; it does
**not** establish that the whole S26 tail or the implementation gap was a
harness artifact. The residual direct-I/O tail remains open. No production
tuning or physical-SSD claim is justified by these traces.

The proposed engine explanations can be narrowed:

- Every measured gapped batch uses **one ring, 64 SQEs and one submitting
  enter** in both implementations. In the fixed slow-tail
  samples, OrbiTE uses a median 2–4 enters including zero-submission waits;
  TENT uses one submitting enter and polls the mapped CQ. These waits never resubmit an SQE.
- All 64-block CQEs are published on the submitting batch worker. There is
  no caller-to-worker ring handoff in this case. Worker drain has **zero
  nanosleeps**; the benchmark caller still uses its documented status backoff.
- Aligned gaps allocate **no bounce buffer**. The new
  `aligned_file_batches_use_one_ring_without_bounce_for_gaps_or_contiguous_ranges`
  control fails every bounce allocation and checks one ring, full bytes and
  guards for READ/WRITE, 8/64 blocks, contiguous/gapped, ext4/tmpfs.
- OrbiTE creates **one batch worker** per multi-request submission; TENT
  creates none. Before-first-SQE time in OrbiTE's slow 64-block samples is
  105–148 µs in the fixed paired traces, much shorter than the block wait. Removing thread creation
  cannot by itself explain or eliminate the measured millisecond tail.

These historical traces also isolated the small tmpfs WRITE handoff: the
median sample created a batch worker after caller submission, versus zero
for READ and TENT. Diagnostic WRITE medians were 137/118 µs (stable/MSRV),
versus TENT's 81/59 µs; they are not untraced S26 results. The
[small-file paired pass](#small-file-paired-results) now measures a bounded
caller CQ wait for the existing eligible request. Expiry still hands the
same owner to the worker and is never a release receipt. The multi-request
direct-I/O owner and the historical tail diagnosis remain unchanged.

### Direct-I/O issue paths and verification load

The [block-layer comparison](https://github.com/OrbitInfra/OrbiTE/actions/runs/42562245093)
uses production `82cf6cef` with diagnostic harness `2fba491d`. The
[counterfactual run](https://github.com/OrbitInfra/OrbiTE/actions/runs/42563852934)
uses `dca107d`, with the **same production tree**, against frozen TENT
`e1ea5d31`. Each diagnostic case has three warmups and 1,000 byte-checked
samples. The latter run uses four-vCPU Ubuntu 24.04 / Linux
7.0.0-1017-azure VMs: stable Rust 1.99.1 on EPYC 9V45, Rust 1.85 on
EPYC 7763. The initial MSRV trace and stable counterfactual expose
`MSFT NVMe Accelerator v1.0`, with a 256 KiB request limit; the counterfactual
MSRV job exposes `Msft Virtual Disk` (`sda`), with a 2 MiB limit. Both report
512-byte logical / 4 KiB physical sectors and the `none` scheduler. Each
within-job engine comparison shares its device; the jobs are distinct virtual
disks, not installed NVMe qualification.

Both implementations open the payload descriptor with `O_RDWR|O_DIRECT`,
without `O_DSYNC` or `O_SYNC`; OrbiTE also uses `O_CLOEXEC`. The separate
OrbiTE buffered descriptor's `O_NONBLOCK` is absent from the direct descriptor.
`strace` records opens and subsequent `fcntl` calls. Plain ring setup, SQEs,
file offsets and device requests agree. For each gapped 64-block WRITE:

| Boundary | OrbiTE | Frozen TENT |
| --- | --- | --- |
| SQE submission | Existing batch worker, one enter | Caller, one enter |
| Ring / SQEs | One / 64 `WRITEV` | One / 64 `WRITEV` |
| File offsets / lengths | 4 KiB + request index × 8 KiB / 4 KiB | Same |
| Kernel issue | 64 NOWAIT direct attempts, then 64 io-wq direct issues | Same |
| Bios / alignment | 64 × 4 KiB, 4 KiB-aligned sectors, stride 16 sectors | Same |
| Merges / unplug width | Zero merges; 64 unplugs of one request | Same |
| Median of per-batch maximum driver-issue depth, initial MSRV run | 9 | 10 |
| Cached folios invalidated after reset | 49: orders 0–2 | 14: orders 0–5 |
| Untimed verification reads, per sample | 32 × 4 KiB plus one large read | 5 × 4 KiB plus five larger reads |

The first repeatable divergence is **verification before the next sample**.
Rust reads the whole file; the earlier C++ harness used an `istreambuf_iterator`
with small buffered reads. Their readahead and cached-folio layouts differ even
though both reset in place and finish `fdatasync`. In the initial MSRV trace,
median between-window time is 1.72 / 2.81 ms (OrbiTE / TENT). All measured
WRITEs are issued before the long waits: OrbiTE's slowest-1% median last issue
is 0.82 ms, while the longest block wait is 29.68 ms. No delayed submit,
extra SQE, bounce allocation, flush/FUA or ownership wait explains that gap.
The trace's queue depth is this batch's outstanding driver requests, not
hardware-wide queue depth. Physical disk service is not visible to the guest.

Two controls leave production and the timed operation unchanged. A bulk-read
C++ verifier preserves every byte/guard check and reproduces the long tail in
TENT. An additional **untimed 10 ms pause after reset** reduces offered load.
Numbers below are medians of the slowest ten traced samples, in milliseconds;
they are **not** untraced benchmark p99s. Stable / Rust 1.85 stay separate.

| 64 × 4 KiB WRITE | Original verifier | TENT bulk verifier only | Untimed pause only |
| --- | ---: | ---: | ---: |
| OrbiTE gapped | 30.78 / 34.16 | — | 1.54 / 2.58 |
| TENT gapped | 3.15 / 3.35 | 29.83 / 28.79 | 2.83 / 3.29 |
| OrbiTE contiguous | 26.58 / 19.71 | — | 17.41 / 3.00 |
| TENT contiguous | 25.61 / 12.89 | 25.01 / 17.48 | 1.88 / 2.77 |

A [reversed-order run](https://github.com/OrbitInfra/OrbiTE/actions/runs/42567993578)
on source `fc3c2a5` runs bulk verification first and the legacy verifier later.
TENT gapped tail medians are 27.03 / 30.12 ms with bulk and 3.28 / 3.39 ms
with legacy verification. The effect persists in both orders.

The original TENT bulk-control longest block waits are 28.45 / 27.48 ms. Thus an
unchanged TENT owner exhibits the purported OrbiTE-specific gapped tail when
only verification changes. Lower offered load reduces it without replacing
OrbiTE's worker, ring or drain; it does **not** eliminate every outlier (notably
stable contiguous OrbiTE). Periodic stalls after driver issue are consistent
with virtual-disk rate limiting, but the host's quota/controller is not exposed:
that mechanism is an inference, not a measured throttle receipt. The evidence
supports a verification/load-dependent comparison and a limit below guest
issue, not a production ownership fix or a general storage speed claim.

Both normal harnesses now verify files with a bulk read. The legacy C++
verifier and pacing are diagnostic controls only; `benches/run.py` strips all
inherited `ORBITE_BENCH_*` settings and records explicit diagnostic overrides.
No pacing is added to the normal S26 sweep. The
[complete paired report](https://github.com/OrbitInfra/OrbiTE/pull/186) records
every row with the original trials, samples, warmup and byte checks. Use
`benches/compare.py BEFORE AFTER --changed-profile none --verification-comparison`:
the explicit mode permits the C++ verification transition only, keeps reset
and Rust-workload checks, and labels affected TENT file rows `V`. TENT CPU/TCP
rows remain noise references; changed TENT file rows do not. The clean-reset
numbers below remain evidence for the **earlier verification regime**, not
proof that OrbiTE issues intrinsically slower direct WRITEs. Current virtual-disk
tails remain a P11 limit for the installed NVMe batch. The bounded small-WRITE
wait is measured separately above; a missed window still needs worker drain.

### Reset-only paired comparison

The [untraced paired run](https://github.com/OrbitInfra/OrbiTE/actions/runs/42536943466)
passes all **480 old and 480 fixed rows on each toolchain**, `521861db` →
`19ec2009`. Each pair uses one four-vCPU EPYC 7763 VM, Ubuntu 24.04 / Linux
7.0.0-1017-azure, GCC 13.3.0, and stable Rust 1.99.1 or Rust 1.85.0. The
original S26 three trials, 100 samples, three warmups, two configured workers,
sizes/shapes and byte checks are unchanged. Production sources are identical.
The complete [stable table](https://github.com/OrbitInfra/OrbiTE/pull/184#issuecomment-6730046861)
and [1.85 table](https://github.com/OrbitInfra/OrbiTE/pull/184#issuecomment-6730078580)
include every ext4 row for both implementations, CPU/throughput and every
numerical regression. Raw results remain outside Git.

These are medians of the three untraced trial p99s, in milliseconds:

| Toolchain / 64 × 4 KiB shape | OrbiTE p99 old → fixed | TENT p99 old → fixed |
|---|---:|---:|
| Stable gapped READ | 15.60 → 10.41 | 2.04 → 1.74 |
| Stable gapped WRITE | 8.08 → 32.98 | 1.67 → 1.82 |
| Stable contiguous WRITE | 1.09 → 14.09 | 1.56 → 3.97 |
| Rust 1.85 gapped READ | 17.17 → 10.63 | 2.44 → 1.59 |
| Rust 1.85 gapped WRITE | 10.83 → 25.98 | 2.67 → 1.76 |
| Rust 1.85 contiguous WRITE | 1.02 → 18.00 | 1.39 → 6.18 |

The reset artifact is removed, but the tail and implementation gap do **not**
largely disappear. WRITE tails worsen, with nonoverlapping OrbiTE trial ranges
in both shapes/toolchains; contiguous TENT WRITE worsens too. These regressions
remain open. Small tmpfs WRITE p50 is 194→203 / 194→195 µs (stable/MSRV),
versus TENT after 81/79 µs. HP-TCP 1 MiB READ p50 moves 605→548 / 548→588 µs;
the MSRV increase has nonoverlapping ranges. No watch item is declared fixed.

Stable/MSRV have 110/114 of 160 cases with a worse statistic; 71 in each pair
have all four trial ranges overlapping. Non-file rows do not execute reset
and remain run/build-placement controls; their clock/cache/VM contributions
are unisolated. File preparation intentionally changes allocation/cache state
and I/O timing in **both** harnesses, so TENT file rows are not pure noise
controls. Neither those controls nor range overlap explain away the file gap;
range overlap is not a significance test. This is a measurement correction,
not an engine speedup.

Reproduce with `benches/compare.py BEFORE AFTER --changed-profile none --reset-comparison`.
The flag permits exactly the documented reset transition and changed harness
hashes; all other workload/host/CPU checks remain. Instantaneous clock readings
are reported separately, not normalized.

### Unchanged-code paired control

The [paired control run](https://github.com/OrbitInfra/OrbiTE/actions/runs/42522826931)
uses the **old harness**, `7e40c7b` → `23f6911`, on each toolchain: identical production sources and
workload hashes, with the ordinary S26 three trials, 100 samples and byte checks.
All **480 rows per side passed**. The complete
[stable](https://github.com/OrbitInfra/OrbiTE/pull/184#issuecomment-6728129888) and
[1.85](https://github.com/OrbitInfra/OrbiTE/pull/184#issuecomment-6728131269)
tables include every slower row. Stable uses four EPYC 7763 logical CPUs;
1.85 uses four Xeon 6973P-C logical CPUs. Each pair shares one VM with Ubuntu
24.04 / Linux 7.0.0-1017-azure. In the 1.85 pair, only the instantaneous
`lscpu` scaling reading differs (116% → 154%); model, topology, flags and
frequency limits match. Frequency was not locked or used to normalize times.
Reproduce them with `benches/compare.py BEFORE AFTER --changed-profile none`.

Stable medians of the three trial statistics; all ext4 rows below use 4 KiB blocks:

| Workload | OrbiTE p50 µs before → after | OrbiTE p99 µs before → after | TENT p99 µs before → after |
|---|---:|---:|---:|
| ext4, 64 gapped READ | 2,220.5 → 1,652.3 | 5,424.5 → 6,536.6 | 4,335.6 → 5,419.0 |
| ext4, 64 gapped WRITE | 853.6 → 842.5 | 2,517.7 → 4,486.7 | 1,762.1 → 2,851.6 |
| ext4, 8 gapped READ | — | 1,582.4 → 777.8 | 1,077.4 → 919.8 |
| ext4, 8 gapped WRITE | — | 643.4 → 681.9 | 891.2 → 1,443.6 |
| ext4, 8 contiguous READ | — | 833.8 → 803.1 | 986.0 → 819.1 |
| ext4, 8 contiguous WRITE | — | 757.7 → 673.7 | 593.9 → 755.4 |

Rust 1.85's gapped 64-block READ/WRITE p99 changes 16.36→15.28 / 4.54→4.41 ms;
TENT changes 1.35→3.40 / 2.15→2.28 ms. Stable tmpfs READ/WRITE p50 changes
52.5→48.2 / 198.9→190.8 µs, versus TENT after 83.9/83.5 µs: the WRITE
handoff gap remains. HP-TCP 1 MiB READ p50 changes 543.6→541.0 µs on stable
and 610.3→583.8 µs on 1.85; this does not establish a fix to its watch item.

All payloads are unchanged. Stable has 141/160 cases with at least one worse
statistic, 105 with overlapping ranges for all four statistics; 1.85 has
104/160 and 84. Not all changes track TENT: stable single-block ext4 READ p99
rises 157% while TENT falls 5%, with nonoverlapping OrbiTE trial ranges.
Conversely, contiguous 64-block READ p50 rises 45% while TENT rises 57%.
The unchanged 1.85 SHM contiguous 64-block WRITE p50 rises 56%, despite no file
I/O. These are run/build-placement variations, not optimization results;
their clock, cache and VM contributions are not isolated. Every regression
remains flagged in the tables. Range overlap is descriptive, not a significance
test. Raw results remain outside Git; no physical-SSD claim follows.

## CPU fault matrix

[tests/faults/matrix.tsv](https://github.com/OrbitInfra/OrbiTE/blob/main/tests/faults/matrix.tsv)
is the executable fixture list: backend, fault, exact production-owner control
and expected charge/release state. It reuses the existing socket, file, mapping
and fallible-table controls. New TCP and HP-TCP cases kill a real requester
process after a partial WRITE. TCP leaves the destination untouched; HP-TCP
keeps the received prefix. Both retain the registration during unregister and
close, then verify all bytes and guards after physical socket drain.
The restart fixture keeps its reserved endpoint and bounds only setup-time
`AddrInUse` retries: concurrent process creation can inherit a listener until
exec closes it. Transfer deadlines, epoch rejection and drain assertions stay
unchanged.

The matrix covers peer death, timeout, cancellation, partial submission,
unregister/close during work, and restart authority. It distinguishes native
socket failures from injected completion faults. io_uring uses real files/rings.
A missing required kernel facility fails, never silently skips.
SHM's synchronous memcpy has no native cancel or partial-post queue; queued
cancellation belongs to the common batch owner. Local files have no peer epoch:
path replacement is tested against the admitted inode. Those dispositions are
explicit, not counted as passes.

```bash
cargo test --locked --features io-uring --lib --no-run --message-format=json > "$OUT/build.json"
# BINARY is the compiler-artifact executable in build.json.
ORBITE_EXT4_DIR="$EXT4_DIR" ORBITE_TMPFS_DIR=/dev/shm \
  python3 tests/faults/run.py --binary "$BINARY" --output "$OUT/faults" --kill-repeats 50
```

CI executes the matrix on both toolchains with the full feature set. It repeats
each new process-kill control 50 times. Existing native fault/stability gates
remain in the matrix. This prepares P10; it does not close its installed
hardware acceptance matrix.

## Installed fault fixtures

The [verification queue](qualification.md#verification-queue) owns the exact
commands and hardware leases. Reuse the installed RDMA fault runner's
`installed-inflight-dma`, `installed-lost-completion`,
`installed-cleanup-failure` and restart modes: accepted posts remain charged
until matching retirement receipts, and failed cleanup retains the native
owner. The NVLink grouped requester-exit and graceful-close modes check
cooperative CUDA import/event lifetime and guarded bytes. Exporter crash and
physical device loss remain explicit limits, not simulated passes.

The additional `gds-p10` packaged consumer runs one real native READ while a
small, test-only cuFile observer withholds status receipts. `drain` checks a
wait timeout and concurrent unregister/file close, then releases real CQEs and
verifies the full payload and guards. `quarantine` withholds receipts past the
submission deadline plus destruction window and checks `Unquiesced`, retained
byte charges and GPU backing after every caller handle is dropped. It never
fabricates a completion or retries a
WRITE. The driver requires compatibility disabled, resource locks and matching
nvidia-fs READ-counter evidence. Both modes are written and compiled in CI,
**not run**. These are controlled receipt faults, not spontaneous device faults.