Single-node fault qualification

The deterministic process gate runs a real Cache Manager, retained engine tensors, independently imported shared payload arenas, CUDA IPC physical routes, and SSD reads. It uses a separate test-hooks build; default release binaries contain no fault barriers. Each test owns its Manager and a private barrier directory. Ordinary completion and model-output gates run separately.

Publish logs a warning after the configured ordinary call deadline, then at most once per minute. It never frees sources merely because a timer expired. A permanently stuck live Manager requires operational restart; this gate does not install an automatic process killer.

Channel tests additionally drop the submission ACK entirely, race cancellation against claim in separate mappings, and prove a delayed request cannot execute after cancellation. They distinguish UDS closure from actual peer process exit and cover shared-record capacity, generation reuse, and duplicate claim rejection.

Completion-evidence requalification

The September 28 completion-evidence increment uses bootstrap 7, channel ABI 11, cache schema 8 and lifecycle 4. Its matching test-hooks build passes 12 selected H20 process/GPU fault cases in 37.22 seconds. They cover the local Restore faults below plus lost completion notification/ACK, restart with stale clients, and test_local_completion_evidence_excludes_retirement_and_trains_once. That test pauses Manager retirement after native drain: the engine consumes correct GPU bytes while Manager source reservations remain held. On release, the report trains once with the native duration, excluding the delayed reap. Channel tests also reject stale/recycled or duplicate reports and discard invalid optional timings without blocking authoritative completion.

The final production build, including source/backend cost-key isolation (c1d8d43c), passes 7 native channel/client cases in 13.02 seconds. On Qwen3-8B with direct, unencoded DRAM, vLLM passes 6 / 1 skipped in 279.98 seconds and SGLang passes 1 / 1 deselected in 204.96 seconds. The vLLM skip is the dense model’s inapplicable recurrent-state case; the SGLang SSD case was not selected. Both use their existing deterministic restart/output contracts. This is single-GPU serving correctness, not multi-rank/P-D/RDMA qualification.

Logs and artifact identifiers are retained in .orbitkv/completion-evidence-20260928/. The diagnostic performance cohorts in communication measurements use the preceding Manager build; their hashes are recorded separately.

Engine-local raw Restore gates

The frozen compact-restore-final-fault bundle passed the selected real Manager/native-client integration and fault suite: 38 passed, 30 skipped in 97.05 seconds. The skipped cases require the cuFile configuration, which was not selected. All selected single-GPU local Restore lifecycle cases listed below passed with bootstrap 6/channel ABI 9/lifecycle 4.

The cases are implemented in test_cache_faults.py. Earlier Manager-owned Restore results remain separate from this evidence.

The CUDA unit test caller_context_survives_registration_and_readiness_success_and_failure in transfer/local.rs tests also passed, covering restoration of the caller’s CUDA context after both successful and rejected registration/readiness paths.

The final release workspace gate passed 488 tests / 38 ignored, excluding nested child-helper invocations from the pass count. The context test above was also run explicitly outside that default gate. Python unit tests passed 374 cases and benchmark-tool units passed 199. The matching production Manager and extension passed all 7 ordinary channel/client GPU integration cases. The same-host Mooncake TCP raw and encoded peer round trips also passed. These remote checks qualify data correctness through the new local executor, not cross-host RDMA performance.

The compaction unit gates cover permuted dense/sparse destinations, split K/V, adjacent distinct allocation IDs, destination overlap, a 32,768-page dense plan, and lease preservation for oversized fragmented plans. The existing 32,768-block GPU benchmark passes Criterion --test mode; it checks submission/drain, while the matched communication runs separately verify GPU bytes and copy counters.

The local_restore_dma barrier is reached after the first enqueue call. It proves that Manager death does not manufacture local completion, and that retained imports support subsequent drain and correct bytes. It does not prove that the copy was physically in flight at the exact instant of SIGKILL. The engine-death case pauses after claim; it verifies conservative source quarantine, not engine death during proven active hardware DMA.

The native worker captures ready_stream and owns tensors independently of Python waiters. Its local terminal result precedes asynchronous Manager source reaping. The channel suite separately covers claim/revoke races, plan-bank pressure, record generations, and retirement acknowledgement. Full serving, multiple-GPU, huge-page, prolonged allocator-pressure, and graph replay gates remain separate from these single-GPU process tests. See engine-local qualification.

The matching production bundle subsequently passed Qwen3-8B DRAM serving correctness on H20: vLLM 6 passed / 1 skipped (the skipped recurrent-state case does not apply to this dense model), and SGLang 1 passed / 1 deselected (DRAM selected, SSD excluded). Both exercise engine restart with a retained Manager. These are normal serving gates, not additional injected-fault tests; the exact configuration does not qualify arbitrary graph modes, hybrid models, or multiple GPUs.

Reproduce

Build before starting any native/GPU tests: Mooncake shared libraries are restaged by Cargo and must not be replaced under running processes.

PYO3_PYTHON=$PWD/.venv/sglang-release/bin/python \
  cargo build --release -p orbitkv-server -p orbitkv-py \
  --no-default-features --features cuda-13,mooncake,orbitkv-server/test-hooks
# Install the matching extension using the development/build instructions.
cd python
ORBITKV_CACHE_MANAGER_BINARY=../target/release/orbitkv-cache-manager-py \
ORBITKV_FAULT_TESTS=1 ../.venv/sglang-release/bin/python -m pytest -m integration \
  tests/integration/test_cache_faults.py tests/integration/test_session_watcher.py

The test fixture alone sets ORBITKV_TEST_FAULTS for its private process. Do not ship test-hooks binaries. Run the hybrid recovery gates with normal builds as well. The H20 container verifies functional SSD I/O on its mounted filesystem; it does not measure physical NVMe or RDMA behavior.

The cuFile cases additionally requires --ssd-backend cufile and a writable --basetemp on a cuFile-compatible mount. Use the environment settings in GPU storage recovery and record whether compatibility mode was enabled. Without this option, the ordinary fault gate skips the cuFile-only cases.

Concurrent Qwen3 serving

The explicit stress gate uses Qwen3-8B with deterministic inference in each pinned engine. It restarts the engine while retaining the Manager, pauses SSD reads and abandons a streaming request while an unrelated request progresses, drops completion notifications, and kills the Manager during a GPU restore. It checks reference outputs, positive restore bytes, cancellation observations and final ownership counters. A restarted Manager starts cold; this is not an SSD index persistence test.

cd python
ORBITKV_FAULT_TESTS=1 ORBITKV_PREPARE_REQUESTS=1 \
ORBITKV_CACHE_MANAGER_BINARY=/absolute/path/to/test-hooks-manager \
../.venv/vllm-release/bin/python -m pytest -m stress \
  tests/stress/test_recovery_faults.py -k vllm --model /workspace/models/qwen3-8b \
  --basetemp=/var/tmp/orbitkv-bench/serving-fault-vllm

Use the SGLang interpreter and -k sglang for its gate. Set ORBITKV_PREPARE_REQUESTS=0 for ordinary demand. Keep the two GPU runs sequential. The workspace test directory retains every engine/Manager incarnation log and fault-results.json. Manager sockets use short temporary paths independently of the evidence directory.

Multi-rank serving, long-running injected-fault traffic, hardware hangs and remote failover have separate qualification gates. Logical framework page-generation evidence remains separate from payload-allocation IDs and process/session fencing.