OrbitKV transport architecture

Prepared Save requests share the existing native pending owner with submitted requests. Metadata stays charged until its last owner drops, including caller-held completed handles; registration tensors are retained through submission or proven never-submitted cancellation. The same 1,024-operation/32-MiB process cap includes conservative merge scratch. This metadata limit is separate from registered allocation and payload budgets.

This document separates the inference-to-Cache-Manager channel, remote data movement, and replica discovery so none of them becomes an accidental second source of KV truth. “Same-host” describes where the client connects; the Cache Manager decides whether a requested block is resident in RAM, needs SSD prefetch, or can be fetched from a peer with Mooncake.

Decision

Both framework adapters require orbitkv-channel for hot data operations. Every inference process must connect to a Cache Manager on its own host; a missing Unix socket fails fast. Registration, health, session watching, and unregistration use the authenticated bootstrap UDS for both adapters.

Registration returns an opaque native GpuRegistration containing the existing InstanceContext generation, fixed client/rank/device identity and retained tensor exporters. Publish accepts that object instead of a mutable instance/rank/device lookup. Both adapters capture it when queuing their producing step. The native client validates the original client and generation after acquiring its worker lock, so a wait or late callback cannot bind a replacement registration. Each Publish chunk carries the captured generation; the Core owner rechecks the same retained instance before topology or GPU pointer binding. Matching lifecycle 9 and cache schema 10 artifacts remain required; retired schemas are rejected.

This is registration lifetime fencing, not allocation/page generation or a physical termination receipt. Holding the registration also holds its exporters after unregister; it does not reauthorize them. Workloads and adapters retire their own references after normal cleanup and retain them if cleanup fails. A separately queued reference keeps its exporters until that task retires. Restore retains its existing query-lease checks and API. The earlier 03cbcebd acceptance covers its snapshot and Core/lifecycle controls. The producer-bound authority has bounded installed registration acceptance; matching real-model regression and physical lifetime remain separate gates in the completion plan. CPU authority controls use an authenticated process fixture and an idle worker with no CUDA executor; they do not register, compute or submit device DMA.

The official vLLM 0.31.0 KVCacheBlock callback and SGLang 0.5.22 external-linker page descriptors expose reusable slot IDs without allocator-issued allocation generations. OrbitKV cannot repair that boundary with an adapter-local map: such a map misses allocations and frees performed by native prefix, scheduler and other connector paths. The upstream interface work is tracked in vLLM #62810 and SGLang #45493. Until a released interface is consumed, adapters retain unknown work and do not claim generation-safe page reuse.

Draft vLLM #62819, exact b37aa38c, has independent source/CPU acceptance for allocation and finish-time identity handoff. It remains an upstream candidate rather than a selected release; OrbitKV does not patch the pinned engine or enable a compatibility fallback around it.

Draft SGLang #45548, exact 3cc5121f, has independent source/CPU acceptance plus a scoped H20 allocator primitive pass for warmed steady-state asynchronous generation issuance, snapshot and cross-stream readiness. Its unwarmed first invocation still waits for preceding CUDA work. This does not qualify the current installed engine, its default Rust tree runtime under faults, serving performance, page reuse after cancellation or physical termination; the selected 0.5.22 package remains unchanged.

Explicit registration replaces the framework pickle payload with versioned protobuf storage/view metadata. The native client exports retained tensors directly; Manager validation precedes CUDA device discovery/import. Storage suballocation and tensor view offsets are separate; dtype/element size, shape/stride extent, handles, ref-counter and readiness event are checked. The selected PyTorch 2.13 ordinary storage export includes a version-3, cudaMalloc type envelope around the 64-byte CUDA allocation handle. Registration validates the complete 66-byte envelope and passes it unchanged to PyTorch’s importer. Bare legacy handles, unknown versions and expandable-segment exports are rejected; they do not share this allocation-owner contract. A native exporter-owner generation is distinct from the Manager registration and engine page-slot generations. The importer still uses the pinned PyTorch storage IPC primitive and holds the resulting tensor mapping until its owner retires. The importer creates a zero-sized typed tensor with torch.empty, then binds the validated storage/view with set_; a Rust byte vector becomes Python bytes and cannot serve as an empty torch.tensor input. The old wrapper-byte field and helper/API are removed. The client retains the exact storage objects used for initial geometry. It exports those objects, rather than reacquiring storage after a wait, and rechecks the tensor pointer, offset and layout before sending and after the import reply. A changed tensor binding is rejected; possibly imported work remains retained. Registered engine pools must keep their binding immutable. Native registration reserves its client/key before creating any export. Each returned credit is charged to the attempt owner immediately; a never-sent export can be repaid exactly once. Register replies bind the export generation and explicit credit disposition. Only NeverImported permits repayment; importer ownership is a handoff, and Unknown never proves import or device termination. The selected framework’s rollback helper is best effort and can return after silently skipping a decrement. The native attempt owner therefore holds the actual exported counter file, checks its identity and exclusive value before rollback, and removes a credit only after the same slot is observed to change from one to zero. Missing/replaced files, unchanged counters and call errors retain the credit without another repayment attempt. This is a reference-credit proof; it is not a device-operation terminal receipt. A failed, partial or lost reply retains the attempt’s client, tensors and credits, even after caller/client destruction. Unregister refuses a pending/retained attempt rather than reporting completed cleanup. The process limits these attempts to 64, each with at most 4,096 tensors/credits, and conservatively charges at most 1 TiB of tensor storage bytes. Within one validated single-device batch, exact (storage base, storage bytes) aliases are charged once; different extents and partially overlapping ranges remain conservatively additive. Each layer still retains its tensor/storage owners and its own export credit. Attempts remain independently charged, including failed attempts with uncertain credits. These limits are separate from ordinary payload admission and existing transfer descriptor limits. Retention counters and per-generation states are available from the native client. The original unaccepted implementation and its negative review are preserved. GpuRegistration.ipc_credit_evidence is an immutable snapshot from importer handoff. retained_credit_evidence(instance_id) observes the original held counter descriptors for failed attempts belonging to that client and instance. The versioned JSON reports original and current-path identity separately, the value observed in the held slot, and explicit read/path errors. A missing or replaced path cannot become a successful observation. These diagnostics add no repayment/release action, and a zero counter is not a device drain receipt. Successful handoff drops the extra descriptors after the snapshot; uncertain attempts retain them under the existing admission limits. Diagnostic output is bounded to 4,096 credits and 4 MiB per call and is consumed by the strict installed registration gate before the producer exits.

Independent installed prerequisite acceptance covers partial-export repayment, conservative throw retention, duplicate registration rejection, native Restore byte assertions, producer-stream fencing and bounded pool-retirement controls. Actual model recovery is recorded separately under S3/S5 in the completion plan. CUDA IPC warnings and an uncertain retained attempt remain explicit limits. These checks do not supply allocation/page generations or physical DMA termination.

Process channel

An isolated deployment can set the same absolute ORBITKV_CHANNEL_CONFIG file for client and Manager. The Rust channel owner reads at most 64 KiB once per process and passes that config explicitly to both Node builders. The file must be regular and not symlinked, with an existing canonical absolute root, a private prefix and node/service directories inside the root. Invalid input refuses Node creation. Precreate every node/service directory component; the owner rejects symlinks and retains their device/inode identities for a check before each Node. Keep that task-owned directory chain immutable from admission through cleanup. The dependency operates on pathnames after this check, so a privileged concurrent rename is outside this isolation contract and must be excluded by the resource owner/lock, rather than treated as atomically fenced by the library. Freeze the TOML bytes and domain resources for qualification; changing the file after first channel construction cannot rebind that process. This separates iceoryx2’s persistent counter, shared resources and stale-node cleanup from a retained historical domain; unique service names alone do not. The default deployment still uses the dependency’s standard configuration.

Native TENT failure retention

The pinned TENT C ABI’s tent_free_batch() accepts a deferred free request; its zero return is not an independent physical-termination acknowledgement. Native FAILED, CANCELED, TIMEOUT, INVALID or unknown task states do not authorize source release or destination reuse. OrbitKV retains the synchronous native call, batch descriptors and caller-owned buffers after observing one. It does not send source release from that call or recycle its admission slot. A wrapper deadline initiates best-effort cancellation and retains ownership while tasks remain pending; only later successful completion of every task permits return of the original logical failure. A native failure is not converted into success by a later poll or replacement transport.

Each TENT engine admits at most 1,024 batches and 32 MiB of C request descriptors across live and retained calls, rejecting excess before native allocation or submission. These limits bound descriptor memory, not the retained payload; physical memory remains owned by the existing pool/query/source budgets. Native transport failover on polling is disabled until the preceding attempt’s termination can be established. Successful normal TCP/RDMA/CUDA transfers keep the existing completion path; physical-failure recovery remains unqualified.

Mooncake #4816 requests a per-transfer physical termination receipt for this retained boundary. The independently accepted public reproducer is OrbitKV PR #237, merge 53e69620: it pauses an admitted TCP READ, observes successful logical unregister, mutates still-valid source bytes and then observes the old READ consume the replacement payload. The control proves why unregister is not a drain; it does not simulate freed memory or establish CUDA/RDMA behavior.

Retention is visible through orbitkv_native_transfer_batches, orbitkv_native_transfer_request_bytes, orbitkv_native_transfer_retained_batches, orbitkv_native_transfer_retained_descriptor_bytes and admission rejections. The retained descriptor-byte sum counts requested payload lengths, including overlapping slices; it is not a physical allocation-size gauge. Uncertain calls keep the owning process isolated and can prevent graceful shutdown. Retention does not authorize killing that process or reclaiming remote buffers. See S3 in the completion plan for the remaining native termination and installed gates.

The source shutdown candidate closes a whole native engine’s TCP admission before waiting for inbound pointer handlers. Handler acquisition precedes address resolution, and release follows synchronous CPU access. A closed gate never reopens; local Rust requester admission also closes, and active requester owners prevent a successful close. Only then does the Manager pool owner unregister and release its backing. Timeout, unsupported transport or error preserves engine, tokens and pool without an unregister/destructor retry. The process charges live and retained pools against 1,024 registrations of pools and 1 TiB registered backing. These are additional source-owner caps, not a change to the 1,024 batch/32 MiB request-descriptor limits or pool/query budgets.

The candidate TCP wire carries a random 128-bit native engine epoch and an explicit success marker on READ replies. Old epochs are denied before resolving pointers, including after source restart; a retry cannot upgrade the request to a different epoch. Peers require matching candidate packages. The pinned upstream ABI does not provide this extension; it is not released upstream support. The receipt covers CPU-only/TCP-only source shutdown. It does not prove device/RDMA termination, close an individual transfer ticket, or permit live allocation/page reuse. Those paths remain retained and unqualified. The original packet control captured its native epoch at the first segment lookup and proved packet/retry fencing only. The separately accepted grant-epoch code adds native_source_epoch to transfer authorization. Manager captures it from its actual registered TENT owner, not discovery. Rust rejects missing/malformed values, refreshes the existing TENT descriptor before native batch allocation, and passes the same expected epoch into each C request. TCP requires that expectation at its first attempt and preserves it through retry; request merging keeps different epochs apart. The same-endpoint/address CPU restart gate checks stale rejection with untouched sentinels and exact replacement bytes for a fresh grant. Clearing the expected epoch is a deliberate negative control. Matching peer/native artifacts are required. Independent ACCEPTED_CODE_CPU_SCOPE at 02ab7fe3 covers this source-grant/first-TCP-attempt code and CPU control boundary. The decisive restart control establishes a healthy replacement connection, then explicitly rejects the old expected epoch; clearing that expectation reads all replacement bytes. Earlier EOF-only race evidence remains rejected. Actual installed Manager/server RPC, GPU and RDMA qualification remain open. This does not qualify allocation/page reuse, RDMA endpoint fencing or a new physical-drain receipt; see S3 in the completion plan and the immutable grant-epoch artifact root for review and hashes. The CPU/TCP receipt profile rejects delegated raw-pointer transfers: that staging API lacks a source epoch and would bypass the guarded pointer ingress. Ordinary CPU/TCP READ/WRITE remains available. Other transport profiles still cannot produce a physical receipt. Delegate, Pin and Unpin use the same pointer admission owner. They are refused before parsing in the CPU/TCP receipt profile; other profiles count accepted callbacks and still return unsupported physical drain. RPC readiness is published only after all engine initialization. Per-agent dispatch ownership is acquired before a handler is queued on YLT’s global executor and released after actual callback completion. Source close shares its five-second deadline with dispatch drain. Native teardown joins those owned callbacks before stopping I/O and resetting service/transport fields; it does not stop the shared global executor. A queued callback keeps the source owner retained.

orbitkv-channel uses iceoryx2 0.10.0 and a fixed 64-byte command carrying protocol version, opcode, request identity, Manager epoch, descriptor offset, length/generation, and two opcode-specific scalar fields. Variable-length hashes, leases, and page arrays use the descriptor arena. KV payload bytes do not travel through either command or descriptor memory.

Each native cache client has a query/restore endpoint and opens a separate Publish endpoint on its first save. The Manager owns the thread-safe iceoryx2 service on a dedicated control thread. Clients enqueue before signaling the required request event; the Manager briefly spins and then waits for an event or maintenance deadline. Fixed 50 us idle polling is removed. Notifications are hints, so a failed wake after enqueue never permits early page reuse.

Bootstrap, registration, and versions

A mode-0600 UDS authenticates the Manager’s uid with SO_PEERCRED and assigns an exclusive descriptor slot and session token. Bootstrap version 7 passes five FDs: descriptor memfd, grant memfd, Manager-to-engine Restore eventfd, engine-to-Manager retirement eventfd, and Publish reply eventfd. Memfds are size-sealed. Descriptor request/response generations advance monotonically, including when a slot is reused by a replacement session.

After bootstrap, the persistent UDS carries epoch-checked lifecycle version 9 frames, with a 64 MiB metadata limit. Registration reuses protobuf metadata without HTTP/2 or gRPC. A successful GPU registration reply also attaches payload arena FDs, identities, and sizes with SCM_RIGHTS. The engine validates the seals and independently maps and CUDA-registers each backing. This setup happens once per arena and GPU binding, not per restored block.

Channel ABI 11 rejects older clients. Client, native extension, and Manager must be rebuilt together; there is no old-wire decoder or alternate runtime protocol. Registration requires actual tensor/exporter objects through register_context_batch(..., tensors=...), keeping them alive with the local CUDA binding. Repeated registration of the same instance/rank/device on one client is rejected. Unregister and close drain accepted operations before releasing the bindings.

Every Manager incarnation advertises a unique iceoryx2 service name behind the stable UDS address; --channel-service is a prefix. Old handles cannot be adopted by a new client. Reconnect establishes new mappings and tensor bindings. See fault qualification.

Native peer_alive() reads the pidfd of the Manager authenticated at bootstrap. It remains independent of UDS health and descriptor/lifecycle waits. The restore completion owner uses it for pending peer-exit diagnosis and to wake readiness checks after process exit, while Active/ActiveMore local DMA stays pending until the engine reports its drain. It neither reconnects nor changes generations.

ObserveCompletion carries bounded P/D completion and decode-resource evidence, without request IDs or state keys. Cache schema 10 combines these observations with the engine-local Restore grant protocol; older schemas are rejected.

Restore and source ownership

Restore reserves a shared operation identity before descriptor submission. The Manager authenticates and claims it before consuming leases. A lost or malformed ACK retains the known handle once preparation was claimed; the request is not retransmitted. Preparation failures are terminal results on that handle. UDS closure stops descriptor admission but does not invalidate retained completion mappings or prove DMA completion.

For unencoded DRAM sources, the Manager compiles a bounded plan containing arena/allocation identities, source allocation bounds and subranges, registered layer names, and destination-relative offsets. It installs the selected source and query-reservation owners before publishing Granted. The engine wins Granted → Active before reading the plan and uses its own pointers and CUDA context. start_restore(..., ready_stream=...) supplies the actual engine stream whose previous use of the destination pages must finish. The first slice waits for that dependency and uses a single copy stream with a whole-operation fence.

After the final local drain, the native worker publishes Drained and makes the local result available through poll_restore/wait_restore. An engine-local eventfd wakes framework completion handling. The Manager’s separate retirement task then releases source owners and publishes Reaped; engine acknowledgement allows record reuse. Source reaping is not on the successful page-consumption critical path. There is no terminal Poll RPC.

The grant mapping contains 1024 records of 192 bytes and a 1 MiB plan bank. Error text is limited to 88 bytes per shared record. Plan consumption releases plan-bank capacity independently of DMA completion, and a bounded dirty bitset plus eventfd drives Manager retirement. Full shared plan-bank capacity defers prepared grants. Larger raw plans are automatically partitioned under one operation ID; nonfinal PartDrained records keep the result pending and retain all source/target owners. The Manager acknowledges each part once before publishing its successor. Operation metadata above 32 MiB is rejected before lease consumption, and each session reserves at most 64 MiB of prepared metadata. Shared-grant schema 5 requires matched Manager and native client builds.

At most 64 session mappings can be live or retained. An engine that dies after claim without drain evidence leaves its sources, byte reservations, and session credits quarantined. Neither TTL, UDS closure, nor pidfd exit authorizes source reuse. If the Manager dies during a claimed local copy, the engine retains its own imported mapping, CUDA registration, and tensors until local drain. A managed Restore without a terminal publication also remains pending after Manager process exit. An unreadable/unknown completion keeps the native worker job and registered tensors owned; it does not produce RestoreStatus.done. poll_restore can expose a bounded retention reason while wait_restore still respects its deadline. Pending operation admission remains charged until a known completion or a proven never-submitted drain. This candidate does not provide a device/RDMA failure fence or reclaim orphaned pages. Generic claim errors cannot publish drain merely by observing Active or ActiveMore: another executor may own that transition. Only the completion owner that won the CAS can establish never-submitted work when it rejects a plan before returning its bytes. CAS loss and unknown failures remain pending. Independent acceptance at d9f779c9 covers code/CPU owner controls and the matching installed wheel’s import/ELF checks. It does not qualify live Manager endpoint/GPU bytes, device/RDMA termination, page reuse or adapter faults.

SSD, encoded, and mixed-source restores use a Managed execution state and retain Manager workers for their materialization, decode, and I/O. This follows the prepared physical route, not a retry after local failure. CUDA IPC remains necessary for those routes and Publish. The obsolete Manager raw-descriptor worker branch is removed. See engine-local Restore for full ownership, limits, and pending qualification gates.

Both native and Manager executors drain accepted copy work after partial enqueue failure. The SaveTask owner candidate retains its source, destination and admission owners after reply loss or an execution unwind. Failed Manager stream or codec synchronization retains the call instead of using process exit as completion evidence. Engine-local Restore failure handling remains separate. A wait timeout or dropped Python handle does not cancel accepted work; the connector must retain logical GPU-page assignments until a terminal result. A proven drained failure can use vLLM’s single-cache-group recomputation path; hybrid failures retain their existing fail-closed engine semantics.

A missing GPU save worker reply leaves the admitted Publish future pending. The caller retains its instance and imported GPU context; no consumed error is published from reply loss alone. The SaveTask execution boundary borrows its resources across raw/codec operations and contains an ordinary Rust unwind. SaveTask also owns the registered instance through caller-future cancellation. The shared process admission limits accepted Save operations to 1,024 and retained request metadata to 32 MiB, including vector spare capacity. Worker completion and proven pre-submit rejection repay the charge; reply loss and loss of logical admission owners do not. Active operation, metadata and logical-byte gauges remain charged on parked sync/unwind paths, and reply/owner loss additionally increments the retained-operation gauge. Logical bytes describe requested GPU ranges and are distinct from physical pool/SSD/GPU budgets. Arbitrary SSD worker unwind has not gained a physical termination proof. This preserves software ownership; it does not prove that a failed device access ended. Actual GPU fault gates and bounded operational reporting remain separate.

Publish and query control

Publish holds its iceoryx2 reply while the Manager save runs asynchronously. The dispatcher serves other requests during D2H, but the caller waits until copies complete and host publication is queued. Queries see blocks after the write pipeline seals them. After bounded spinning, Publish waits on its own reply eventfd and Manager pidfd. A 10 ms response-queue recheck recovers lost wakes. Publish and Restore do not consume each other’s notification counter.

Publish requires a Manager pidfd before submission. An ordinary deadline logs a warning, then at most once per minute; it does not release source pages. The request service permits exactly one Manager recipient. Clients reject a service configured for more recipients; restart the Manager and clients together when upgrading from the earlier service configuration. The native request-send error remains typed. Only request-loan, pre-delivery connection and active-request admission failures prove no submission. Other send errors may follow delivery and cannot release source pages, even with one configured recipient. Ambiguous/malformed replies and post-submit transport errors retain the source owner even after confirmed Manager exit. Process liveness cannot prove CUDA completion. Before parking on these errors, the client closes the session and releases its descriptor lock. Other descriptor callers on that same client fail without reusing the slot; the original source owner remains held. Only a matched valid result following the Manager copy drain, or a proven never-submitted send failure, permits return. This conservative retention still blocks the caller; bounded operational responses and adapter page-generation consumers remain separate pending work. The independently accepted SaveTask correction keeps a lost worker reply pending and retains admission instead of publishing a consumed error. Its code/CPU and package-load acceptance does not establish installed GPU or physical fault completion. Publish batches split to the negotiated descriptor slot capacity, retaining matching page ranges across layers until all chunks finish. The local Publish activity candidate reports active/retained callers, the elapsed interval since activity last became nonempty, overdue status and authenticated Manager liveness. It uses immediate local lock attempts and pidfd polling, without waiting for the blocked descriptor/lifecycle exchange; contention returns an explicit unavailable error. The interval can include several overlapping calls and is not a per-request latency. vLLM/SGLang completion polls use it for rate limited diagnostics only. Pending, overdue, retained and dead-peer observations never mark engine pages complete. At most 1,024 callers enter one channel’s Publish activity; the existing Save worker budget separately bounds accepted Manager work. The original synchronous caller remains held until its established completion path; an asynchronous operational return requires a later owner API. Independent 26884461 code/CPU and candidate-package API acceptance covers this diagnostic scope. Per-poll native/pidfd overhead and debug logging overhead are unmeasured; neither package loading nor the controlled SGLang method qualifies Manager/GPU operation, asynchronous completion or physical fault reclamation.

The native Save handle candidate admits a request under a process-wide 1,024-operation/32-MiB request-metadata budget, including retained vector capacity. It retains the native caller, registration worker and tensor exporters on a Rust thread. Dropping a Python handle does not cancel accepted work or release pages. wait_save(timeout=...) returns pending when its wait expires; no timeout is a terminal receipt. Without a timeout, the existing Rust owner waits through late completion and retains unknown outcomes. Client close/unregister blocks new admission while waiting for accepted Saves, then follows the existing worker and Manager registry drain. The adapters retain task/page completion state until a matched receipt and reject reset/unregister after an unknown wait response. This candidate has no page-slot or physical-fault reclamation qualification; request metadata and tensor/allocation bytes have separate owners and bounds. The accepted d49f6deb correction exposes SaveNotSubmittedError only for an explicit rejection before native worker launch. It proves nothing about older Saves; unknown allocation, attachment and wait outcomes still retain pages. SGLang also keeps already queued tasks behind an unknown offload from advancing its completion FIFO. Code/CPU and package review, installed normal DRAM/io_uring byte gates, both official-release model gates and same-host TCP shared-cache restart gates pass for the frozen cb19ce82 wheel. See the qualification scope. Prepared Save handles remain charged while held in Python queues, retained adapter tasks or user-held completed handles, until their last native owner is dropped. This metadata cap is separate from physical payload budgets. Unknown-owner reconciliation, GPU/model fault gates, page generations and physical termination remain separate requirements. This Publish partitioning does not yet partition oversized raw Restore plans.

Rust CacheClient, exposed directly by PyO3 as CacheManagerClient, owns query tickets for both adapters. Python constructs an immutable BlockHashes batch once per lookup; views share its allocation and native pending-query identity checks avoid repeated per-page conversion. Submission, revision, polling, and cancellation remain native, and blocking calls release the GIL.

Cache schema 10 distinguishes metadata-only candidates from leased reads. Discovery returns Candidates, never a Restore lease, and does not reserve payload bytes. read_recovery translates compiled demand into exact hash views and validates complete leased coverage. The warmup flag prepares pages without a lease, skips warmup-budget pressure, and retires without polling. See queued warming. Consumer-owned preparation and exact revision claims retain their contract in request preparation.

Pending query operations are bounded to 128 per session and 1024 globally, with a 60-second reply lifetime. Revision changes can replace hashes or wait policy while preserving instance/request/group identity. Old polls and cancels cannot consume replacements; unknown polls do not submit work. Submitted reads retain ownership through cancellation or expiry, and undelivered results release their leases without another poll. Delivered query reservations pass into the Restore grant or Manager worker and survive until its actual terminal fence.

Host and lifecycle boundary

The adapter derives /tmp/orbitkv-<addr-port>.sock unless orbitkv.bootstrap_socket is set. A scheduler querying several TP shards uses local sockets in orbitkv.tp_shard_bootstrap_sockets. The current centralized vLLM scheduler requires all configured query shards on its host; cross-host TP query fan-out remains future work. orbitkv.wait_for_full_prefix keeps its existing local pending-query semantics.

Standalone mode starts no gRPC listener. Distributed placement configuration enables the peer transfer-control listener. UDS and HTTP cleanup serialize instance lifecycle, close Manager GPU queues, drain their submitted work, and then release imported CUDA IPC mappings. Imported contexts use exact instance, TP/PP rank and GPU UUID keys; punctuation in a legal instance name cannot make cleanup select another instance that shares its prefix. Independent f8cb87bb acceptance covers the selector/actor CPU controls and matched installed import; actual CUDA mapping/lifetime gates remain open. The native client separately drains its engine-local operations. Manager cleanup cannot discard active local grant owners merely because the instance or UDS session has closed.

Manager shutdown first closes one atomic Publish admission/count state. A Publish that registers before that close is an accepted operation and its full continuation is drained; a control iteration that was prechecked but has not yet registered is rejected after close and cannot enter during resource drain. No mutex is held while an accepted continuation completes.

Lifecycle close is durable rather than an edge-triggered notification. A full header and payload establish the accepted-operation boundary. Shutdown may close a partial header or payload without dispatch, but an accepted operation keeps its dispatch and bounded response write; the connection then observes the latched close even when the client stays open and idle. Connections accepted around the boundary either join the tracked pre-close set or are rejected by the same closeable connection gate. Session-owner cleanup remains part of connection drain.

Measured process-channel baseline

The current communication measurements compare request events, Publish reply notification and encoding changes against the preceding revision, including CPU cost and raw-ping regressions.

Measurements were collected on one H20 node with two Linux processes and a 64-byte request/response descriptor, before bootstrap version 2. They are engineering evidence for the transport choice, not measurements of this revision or end-to-end serving results.

In that historical run, the real local path was about 4.58x faster than gRPC by both mean RTT and sequential throughput. Its roughly 107 us RTT was far above the 4 us iceoryx2 substrate. The current request event removes the fixed 50 us idle poll, and request encoding removes repeated allocation and speculative Publish chunk copies. The benchmark is sequential because the scheduler needs one answer before committing a recovery boundary.

The iceoryx2 result can be reproduced with the two binaries documented in crates/orbitkv-channel/README.md.

Earlier local validation

The following serving results predate the engine-local raw Restore cutover; its new process-fault and serving gates are tracked separately in engine-local Restore.

With the pinned vLLM 0.29.0, a single-node H20 run passed the applicable pure-attention E2E recovery gates after an inference-process restart, comparing ordered generated text to vLLM native prefix caching. SGLang 0.5.20 passed its direct GPU-page restore gate after a radix-cache flush. These are local correctness checks, not multi-host reliability or throughput measurements. The SGLang multi-rank and cross-host TP paths still need qualification. See Python test gates and the completion plan.

Mooncake findings

Mooncake separates three responsibilities:

1. TransferEngine exposes Segment, Buffer, and BatchTransfer abstractions.
2. Transfer metadata publishes protocol, topology, registered regions, and connection endpoints.
3. Mooncake Store Master owns generic object allocation and the PROCESSING -> COMPLETE replica lifecycle.

The useful transport mechanisms are:

• topology-aware path selection across CPU, GPU, and multiple NICs;
• request slicing and multi-rail bandwidth aggregation;
• on-demand endpoint creation with a bounded SIEVE endpoint pool;
• two-phase endpoint retirement after outstanding work requests drain;
• retry on alternate local or peer rails;
• SHM, TCP, RDMA, EFA, NVMe-oF, and accelerator-specific backends behind one BatchTransfer interface.

P2PHANDSHAKE does not mean that Mooncake is control-plane-free. It starts a custom TCP socket daemon, exchanges JSON metadata and QP/MR information, and then uses the selected data transport. Larger deployments may instead publish Segment metadata through etcd, Redis, or HTTP.

OrbitKV pins OrbitInfra/Mooncake integration commit 12cbf549b6351c800de0f94a480276d216b9a43b, based on upstream v0.3.13.post1, and builds the TENT tent_shared target in orbitkv-mooncake-sys without applying build-time patches. Native loading first checks ORBITKV_MOONCAKE_LIB_DIR, then the executable or Python extension directory, then the local .orbitkv/mooncake/{cuda|cpu}/lib build cache. Wheels bundle libtent_shared.so, libmooncake_common.so, and libasio.so; the legacy libtransfer_engine.so is removed from staged runtimes. System RDMA/CUDA libraries remain deployment prerequisites.

Experimental OrbitTE allocation bridge

The opt-in orbitkv-core/orbitte feature pins OrbitTE 02d64e55. PinnedAllocation::orbitte_memory returns a canonical Arc<HostMemory> retaining that exact allocation. A pool owner alone would keep the mapping alive while allowing a slot to be reused. The allocation holds only a weak cached view, so concurrent aliases share one access lock without creating a reference cycle. The unsafe caller must coordinate CPU, GPU and other process access; this lock cannot synchronize unrelated CUDA streams or another process’s mapping.

Register these views through OrbitTE’s host registration/batch API and retain its returned handles. Accepted requests retain the allocation through logical cancellation, wait timeout, unregister and a detached batch until the backend proves local drain. This first bridge registers allocation ranges; it does not add a whole-pool MR cache or switch the Manager peer path. OrbitKV still owns source grants, allocation generations and completion-mailbox matching. The peer profiles below consume source grants and generation-scoped gRPC completion. Transfer-notification mailbox integration and installed serving qualification remain open.

Source CI checks the optional feature with the normal CUDA 13 build. Run the hardware gate explicitly on an isolated GPU with a compatible CUDA runtime:

cargo test --locked -p orbitkv-core --features orbitte --lib \
  memory::pool::orbitte_tests -- --ignored --test-threads=1

The two tests use the production PinnedAllocator (memfd, CUDA host registration and real slot reclamation), actual OrbitTE engines and TCP bytes. They cover concurrent aliases, duplicate/subrange registrations, READ/WRITE integrity, blocked transfer cancellation and detach, failed early free, unregister, slot exhaustion, allocator drop and eventual owner release. A TCP relay delays the actual data connection without implementing a fake transfer engine. They are ignored by ordinary tests because a real GPU is required; an unavailable GPU leaves this gate unqualified.

Historical allocator pilots used a private CUDA 12.2 runtime. Their immutable source/artifact records are retained, but current OrbitKV builds require CUDA 13; CUDA 12 is unsupported. Run the current allocator gate on a CUDA 13-capable machine. Keep exact source/binary/runtime hashes, device identity, resource locks, bytes and cleanup outside the checkout; do not infer RDMA, GPU DMA or crash recovery from a TCP ownership gate.

Experimental OrbitTE peer TCP profile

An opt-in orbitte build can select --peer-transfer-backend orbitte-tcp. The default remains mooncake. Build the standalone Rust payload path with:

cargo build --locked -p orbitkv-server --no-default-features --features cuda-13,orbitte

Shared discovery, planning, authorization and completion compile when either mooncake or orbitte selects a payload transport. The standalone profile does not compile or load Mooncake. It still needs OrbitKV’s ordinary CUDA/Python prerequisites; this is an experimental source profile, not an installed wheel or serving qualification. Select one provider across the tested cluster.

Manager still authorizes residency and holds generation-scoped gRPC tickets. The OrbitTE source registers the actual allocations of each accepted grant and keeps those explicit references with its window/slot/generation. Concurrent grants share canonical allocation views and registration references. Expiry retains the source. A completion for an older generation cannot retire a newer grant. In-flight OrbitTE handlers retain the backing even after logical unregister.

The requester opens the orbitte+tcp://IP:port endpoint with the epoch returned by the source grant and uses it unchanged for READ. A refreshed descriptor cannot replace that authority. Provider mismatches fail before payload I/O. A blocking READ owner retains source tickets and actual destination allocations after caller cancellation. A locally unquiesced outcome quarantines both the ticket and buffers, keeping completion admission charged instead of sending a release RPC. Source shutdown fences admission even if requester drain fails; failed cleanup retains its engine and process pool budget.

This first profile requires a concrete numeric advertised IP, has no RDMA NIC filter, and limits one READ to 64 MiB and 16,384 split descriptors (8 MiB each). An engine retains at most 4,096 unique allocation regions and at most its pool capacity. Providers share the existing process admission of 1,024 pools/1 TiB. The planner is not automatically resized for these limits: a source registration or READ exceeding them is rejected; a READ budget rejection occurs before payload I/O. Use bounded grants for this profile. Whole-pool MR caching, multirail, RDMA/device DMA, GDS and broader faults remain open.

Transfer notifications and their generation mailbox remain a separate transfer API contract. Manager releases source grants through its existing gRPC completion owner. This profile does not turn a notification receipt into a physical drain proof.

CPU CI runs real TCP epoch/restart and protocol controls, the existing ticket loss/retry/cancellation controls, and the unquiesced quarantine control in its own exact-filter process. Run the GPU gates explicitly with a compatible runtime:

cargo test --locked -p orbitkv-core --features orbitte --lib \
  peer::export::tests::orbitte_grant_registrations_share_backing_and_fence_old_completion_generations \
  -- --exact --ignored --test-threads=1
ETCD_BIN=/path/to/etcd ORBITKV_METADATA_ARTIFACT_DIR=/var/tmp/orbitkv-peer-gate \
  cargo test --locked -p orbitkv-server --features orbitte --lib \
  cluster::tests::p2p_mooncake::p2p_orbitte_remote_fetch_roundtrip \
  -- --exact --ignored --test-threads=1

The core gate uses the real pinned allocator, duplicate grants, slot generation reuse, eviction and actual TCP bytes. The server gate reuses the maintained workload with actual Manager gRPC services, etcd/inventory discovery, source budget/replay/fencing controls and GPU Save/Restore byte comparison. It awaits both service tasks and records test-owned etcd wait results outside the checkout. At implementation 5a50d092c651d28ff3f79f6de661c9da8120b2b3, both H20 source gates passed and were independently rerun with a private CUDA 12.2 runtime. Source checks, protocol/ticket controls, exact artifact hashes, mapped libraries, GPU leases and owned native/supervisor/etcd reaping were checked. The retained failed first attempt and corrected runs are outside the checkout at /root/orbitte-artifacts/peer-read-20261021/; see PR #304. This is a same-host embedded Manager source pilot. It does not qualify an installed Manager, serving, CUDA 13 runtime, physical cross-host Manager traffic, RDMA or device DMA.

Experimental OrbitTE peer RDMA profile

Build the explicit single-rail host-staging candidate with GNU C/C++ compilers. CI fixes Ubuntu 24.04 with matching Clang/libclang 18 for ibverbs-sys binding generation; use CLANG_PATH=/usr/bin/clang-18 and LIBCLANG_PATH=/usr/lib/llvm-18/lib for that parser profile.

CC=gcc CXX=g++ cargo build --locked -p orbitkv-server --no-default-features \
  --features cuda-13,orbitte-rdma

Select --peer-transfer-backend orbitte-rdma --orbitte-rdma-device mlx5_0 --orbitte-rdma-port 1 --orbitte-rdma-gid-index 3 alongside the ordinary membership/peer flags. Device/port/GID must match the actual RNIC and route; these values describe the supplied RoCE test profile, not device discovery. The device is required. The Mooncake --nics filter is rejected. Rust users set EngineConfig.orbitte_rdma with the re-exported OrbitTeRdmaConfig. The default provider remains TENT; the orbitte feature alone does not enable RDMA.

The provider advertises orbitte+rdma://IP:port and fixes every payload request to Transport::Rdma. Metadata, descriptor bootstrap and DONE/ACK use TCP; payload bytes use RC verbs and private host staging. An unavailable device, source or build feature returns an error without selecting TCP or TENT payloads. Device availability is checked when the first native endpoint opens, not by successful metadata lookup. A TCP namespace cannot authorize this provider. The pinned allocation, grant epoch, registration references, cancellation owner and generation-scoped completion remain those described above.

This is the existing OrbitTE staging path: a PD/CQ/QP/MR per slice, up to 64 endpoints and 64 MiB staging per engine by default. These charges are separate from source tickets and the process pool budget. The TCP profile’s 64 MiB READ, 16,384-descriptor and 4,096-allocation limits still apply. READ commits after local CQ completion, successful local QP/MR destruction and the source ACK. No application/GPU pointer is registered with the RNIC; this is not GDR or zero-copy host registration. Endpoint reuse, multirail and pipelining remain open.

Local locally_drained describes requester resources. A source native cleanup failure currently arrives as an ordinary remote backend error; it does not supply a source termination receipt. The source transfer engine independently retains its actual registered backing, admission, failed staging and budget, poisons new native admission and prevents a successful source close. A gRPC ticket release cannot override those owners. Local Unquiesced still retains the requester ticket and destination allocations. Native calls can exceed a Rust deadline; neither timeout nor watchdog action is physical drain evidence.

The explicit host component gate builds with CUDA 13 but never initializes CUDA:

CC=gcc CXX=g++ cargo test --locked -p orbitkv-core --no-default-features \
  --features cuda-13,orbitte-rdma --lib --no-run
python3 /path/to/OrbitTE/tests/hardware/run.py \
  --profile /external/orbitkv-host-profile.json \
  --binary /external/core-test-binary --output /external/host-run-001

Use the maintained OrbitTE runner with runner: "orbitkv-peer-host", explicit RNIC/port/GID, identity-pinned SSH nodes, both kinds cpu and transport rdma. It invokes the ignored exact test peer::orbitte::tests::rdma_host::registered_host_rdma_peer. Native assertions check fragmented registered READ bytes, all guards, old epochs, protocol mismatch, actual requester CQ counts, zero endpoint/staging/registration charges, mapped verbs libraries, absent CUDA/TENT libraries and owned shutdown. The controller holds RNIC leases until native wait/reap. This exercises the production registered-read execution with ordinary host backing; it does not test the PinnedAllocator or Manager GPU path.

Independent source/CPU review and physical host reruns pass at consumer 660f893d: A-to-B with a 4 KiB payload region and B-to-A with a 1 MiB payload region, each with 64-byte guards at both ends and three fragmented READ CQ completions per requester. Guards and unrequested bytes remain unchanged. Normal close returns all registrations/endpoints/staging to zero. Both processes map libibverbs/libmlx5 and no CUDA/TENT; native children and supervisors exit and release their RNIC leases. The initial readiness-only failure is retained with successful runs. Frozen binary/source provenance, raw runs and independent review locations are linked from OrbitKV #305 and OrbitTE #12.

The embedded Manager GPU gate is separate and requires a working CUDA 13 runtime:

ETCD_BIN=/path/to/etcd ORBITKV_METADATA_ARTIFACT_DIR=/external/manager-rdma \
ORBITKV_TEST_ORBITTE_RDMA_DEVICE=mlx5_0 ORBITKV_TEST_ORBITTE_RDMA_PORT=1 \
ORBITKV_TEST_ORBITTE_RDMA_GID_INDEX=3 \
  cargo test --locked -p orbitkv-server --no-default-features \
  --features cuda-13,orbitte-rdma --lib \
  cluster::tests::p2p_mooncake::p2p_orbitte_rdma_remote_fetch_roundtrip \
  -- --exact --ignored --test-threads=1

The supplied H20 nodes combine kernel driver 535.161.08 with cuda-compat-13-1 user-mode driver libraries 590.48.01. Kernel driver R535 alone does not rule out CUDA 13: NVIDIA’s forward-compatibility matrix allows this combination on supported data-center GPUs. Record actual mapped libraries and run each required contract; a CUDA compute probe alone does not qualify CUDA IPC, Manager operation or GDR. Do not substitute CUDA 12.

At consumer 660f893d, the A-node CUDA 13.1 source profile passes real pinned source-grant controls and embedded Manager GPU Save/Restore with both TCP and same-RNIC RDMA staging. Two embedded engines exercise actual etcd, inventory, gRPC source grants, exact epochs and payload bytes. Maps contain the installed libcudart.so.13.1.80, compatibility libcuda.so.590.48.01 and, for RDMA, libibverbs/libmlx5, with no TENT. Source/service workers drain and owned native, supervisor and coordinator processes are reaped; the fixture kills its private etcd coordinator, so this is not graceful-etcd qualification.

Reproduce that compatibility profile by prefixing the gate command with LD_LIBRARY_PATH=/usr/local/cuda-13.1/compat:/usr/local/cuda-13.1/targets/x86_64-linux/lib and selecting an exclusively leased GPU/RNIC. Library paths are deployment inputs, not a fallback selected by OrbitKV. Keep the external source/binary, loader, resource and independent-review evidence linked from OrbitKV #306. B-node GPU computation remains untested while its canonical GPU leases are busy. That embedded profile does not qualify installed CUDA IPC. The separate installed profile follows below. Installed serving, physical cross-host Manager/GPU, RNIC faults, GDR and full TENT parity remain open.

Installed client and Manager profile

PR #307, source eb1cc90f, adds complete optional wheels and a maintained two-Manager installed gate. Independent H20 reruns pass both TCP and same-RNIC RDMA using the complete CPython 3.11 wheel 36b491e1a0daa97d, built from identical package inputs at 487beb85. This profile uses kernel driver 535.161.08, compatibility driver 590.48.01, Torch 2.13.0+cu130 and one explicitly selected CUDA 13.0.96 runtime. The compiler is CUDA 13.1; that does not make the loaded runtime CUDA 13.1.

Each provider registers two clients through cross-process CUDA IPC, publishes source GPU pages, awaits the inventory fence and fetches through the selected OrbitTE payload provider. It restores 16,384 GPU bytes and checks every untouched page, exact Save/remote/H2D byte counters and source completion. Four original held IPC slots per case change from one to zero before normal Manager exit. All client/Manager runtime files match the frozen inputs, RDMA maps real verbs/mlx5, and neither provider maps TENT. The client, Managers, supervisors and outer waiters exit normally, canonical leases are released and GPU memory returns to zero. The private etcd fixture is killed and reaped; graceful-etcd shutdown is outside this acceptance. Producer-exit warnings reject the qualification.

Runtime selection is an explicit deployment input. Frozen cudarc 0.19.8 tries libcudart.so before libcudart.so.13. Torch supplies the latter; a system unversioned library can therefore load a second runtime even with Torch’s versioned file preloaded. This profile freezes a private directory containing a standard unversioned alias to the actual Torch runtime and places it first:

mkdir -m 700 /absolute/private/runtime
ln -s /absolute/torch-site/nvidia/cu13/lib/libcudart.so.13 \
  /absolute/private/runtime/libcudart.so
export LD_LIBRARY_PATH=/absolute/private/runtime:/usr/local/cuda-13.1/compat${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}
export LD_PRELOAD=/absolute/torch-site/nvidia/cu13/lib/libcudart.so.13

Freeze that directory, alias identity and target before launching. The gate requires exactly the selected CUDART file in every client/Manager mapping; its version observer uses RTLD_NOLOAD and checks that observation preserves the mapping set. Full loader-order CPU controls preserve the mixed-runtime negative cases. Failed startup, duplicate-evidence and mixed-runtime attempts remain archived with their verdicts. Raw runs, immutable wheel/source/loader manifests and independent review are outside the checkout at /root/orbitte-artifacts/installed-rdma-20261022/, indexed from PR #307.

DRAM Restore uses the client’s local H2D executor; CUDA IPC is exercised by registration and Publish. This scoped gate does not qualify Manager IPC Restore, inference serving, physical cross-host Manager/GPU, RNIC faults, GDR, multirail or complete TENT parity. A uniform installed CUDA 13.1 runtime remains open; the separately accepted CUDA 13.1 embedded/source profile above is unchanged.

What OrbitKV reuses

The default orbitkv-transfer::TransferEngine is a narrow wrapper over the pinned upstream Mooncake TENT C ABI. There is no legacy runtime selector and no OrbitKV-owned verbs implementation. Authorized plans are lowered directly to Mooncake Segment addresses, BatchTransfer operations, and notifications.

Memory registration returns a MemoryRegistration token. Its nonconsuming try_unregister keeps the retry handle and TransferEngine after an error; a successful call marks the token inactive. Drop the inactive token when its engine reference is no longer needed. Token destruction still attempts cleanup on a best-effort basis and cannot report failure or prove native drain. The token does not own the allocation itself: retain the actual backing and token while explicit cleanup is unresolved. The current host pool remains beside its tokens in MooncakeTransport, whose owner requires a source-close receipt before unregistering and retains the pool and engine on failure. The experimental Python TENT API uses register_memory([memoryview(buffer)]) for nonempty writable contiguous byte views of built-in bytearray storage. It retains an actual buffer export, preventing view release and backing resize, and returns the registered addresses. Registration attempts become owned before entering native code, including a failed call that returns no token. Local READ/WRITE slices must fit a successfully registered range; cleanup serializes with synchronous transfers. Caller writes must still be ordered after transfers.

close() retires the whole source, requires a successful source-drain receipt, then unregisters every attempt before releasing exports. An error preserves the successful prefix, failed range, engine and charged backing for an explicit retry. GC does not retry a preceding failure; it retains that owner until process exit. The current native receipt profile supports CPU-only TENT/TCP. CUDA builds or other loaded transports reject close, so this API does not qualify physical GPU/RDMA termination or reclamation.

The separately named register_memory_borrowed(regions) accepts raw CPU/GPU pointers for existing experiments. It does not retain allocations: callers must keep their actual backing through registration errors, unknown completion, cleanup failure and engine GC. Its explicit unregister_memory(addresses) is logical registration cleanup, not a physical drain receipt. Owned and borrowed registrations cannot share an engine. The installed TCP gate isolates borrowed calls in a child that exits without Python teardown on an unknown failure; this does not establish safe device reuse after process exit.

Python live and retained owners share a process cap of 1,024 regions and 1 TiB. Owned views charge each complete bytearray allocation, including capacity; aliases are conservatively charged again. Borrowed pointers can charge only their declared ranges. MooncakeTransferEngine.memory_usage() returns region count, charged bytes and retained-engine count. These diagnostics are not completion receipts and are separate from the Core source-pool budget. A future engine-HBM export must similarly keep its page-generation grant beside the token, using Mooncake’s cuda:N location, until native completion drains. The current C ABI does not report the data transport actually selected for a batch, so OrbitKV does not infer TCP/RDMA/GPUDirect from --nics or success; qualification records Mooncake logs and NIC/device counters externally.

TENT task cancellation is best effort. OrbitKV requests cancellation after a deadline or partial submit and continues polling. It calls tent_free_batch only after every task reports COMPLETED; a successful free request is not a completion fence. A native non-completed terminal or unknown status retains the call and its source/destination buffers, as described in native failure retention. After every task completes, the call returns the first recorded submission, polling or cancellation error. It returns Timeout only when no such error was recorded and the logical deadline expired.

OrbitKV also exposes TENT’s bounded NIC load snapshot: device name, in-flight bytes and EWMA bandwidth. This is live rail-pressure evidence, not a per-batch transport receipt. P/D may use the aggregate for diagnostics, while shared-cache qualification still checks external NIC counters before claiming RDMA or GPUDirect.

The runtime loader resolves only tent_* symbols from libtent_shared.so. CPU and CUDA variants are built separately from the same pinned source; the CUDA variant includes CUDA/GDS/NVLink-capable TENT components. Missing TENT artifacts fail startup and never fall back to libtransfer_engine.so.

TENT versus the legacy Transfer Engine

The important change is the execution contract, not just a renamed shared library. TENT exposes one segment and batch model across CPU, CUDA, file and network-capable transports; extended registration carries memory location and permission, while topology and rail selection remain inside Mooncake. Its C ABI also exposes per-task terminal status, cancellation, notifications and NIC-load snapshots. OrbitKV retains each allocation while the exact batch lacks completion evidence and can feed live resource pressure into planning without owning an RDMA implementation. Native failure statuses do not establish physical drain.

The legacy wrapper primarily exposed submit/poll/free around the older TransferEngine object. In particular, freeing its batch handle was too easy to misread as a completion fence, and its Python integrations commonly held only raw addresses rather than registration owners. OrbitKV’s TENT path instead uses Rust RAII registration tokens and cancellation with completion or retained ownership. This does not make every transfer automatically faster: HBM-to-HBM speed still depends on the selected transport, topology, registration cost and chunking. It does make the ownership and observability needed for safe high-performance selection explicit. Because the current C ABI has no per-batch transport receipt, RDMA or GPUDirect must still be proven with runtime logs and hardware counters rather than inferred from a successful call.

TENT implements a real peer probe internally, but the pinned stable C ABI does not export it. tent_available reports only local-engine health, and segment metadata can remain thread-locally cached for an hour, so neither is a valid substitute. OrbitKV therefore rejects SGLang’s optional failed-session probe mode instead of allowing a false-positive unblacklist. A failed native batch keeps its caller and source-completion owner held; it does not return to invalidate a cached peer segment while physical termination remains unproven.

OrbitKV does not adopt Mooncake Store Master as its semantic authority. Today the cache engine handles versioned model/storage keys and leases above Mooncake. The common recovery plan additionally includes:

• validated StateDescriptor, StateBundle, and RecoveryContract evidence;
• replica selection and restore-versus-recompute planning;
• query leases and generation validation;
• semantic and execution frontiers;
• publication only after every required component is complete.

The current Catalog advertises candidate owners, not durable KV data or permanent raw addresses/rkeys. A selected source authorizes and pins current bytes before transfer. Fixed shards and owner replay are implemented; directory replication is future work.

Remote operation choice

• current demand-driven remote cache reuse uses Mooncake READ into destination pinned memory before restoring into framework HBM;
• vLLM live P/D uses the official NIXL connector and its native ownership;
• SGLang live P/D uses the official disaggregation backend. OrbitKV’s external linker is independent and disables historical restores on D;
• proactive replica placement, retry across replicas, and bundle-aware publication are planning targets, not current guarantees.

The target recovery contract must reject incomplete or incompatible bundles; the Manager wire protocol does not yet enforce StateBundle completeness. SGLang validates prefix/window/checkpoint requirements and vLLM validates attention/recurrent requirements in their adapters before restore, using the shared Rust contract with absolute token coverage and leased group positions; see hybrid recovery. Logical framework page generations remain separate from the local source allocation IDs used by raw grants.

TCP copy error propagation candidate

The pinned integration branch carries the scoped backports of merged Mooncake #3476 and #3973; see native source provenance. TCP client and server copy errors propagate to native FAILED. A READ error response whose length equals the requested payload is padded, preventing error text from being consumed as valid bytes. CUDA host-only copies use memcpy, while device copies use the allocation-owning device and the driver’s pointer classification. Unexpected pointer-query failures do not select host memcpy, and error exits after device selection restore the caller’s device when one was previously active. Controlled CUDA API checks cover these failure paths; healthy-GPU execution remains unqualified. After a CUDA memcpy attempt reports an error, the native owner retains the stream, current frame and RPC staging buffers; errors are not a DMA termination proof. Its separate live/retained device-copy admission is 64 calls and 1 GiB logical copy bytes; RPC strings allocated before that boundary are separate transient allocations. Existing Rust descriptor and physical-memory owners keep their budgets. Unusable CUDA bundle admission fails before transport startup. No timeout-based reclamation is introduced.

These changes preserve the existing conservative failed-batch retention. A copy error or FAILED status does not prove physical quiescence, and the wrapper does not free retained source or destination owners. Deterministic CPU controls and installed host-buffer byte tests have a separate trigger from healthy-GPU, GPUDirect RDMA, Manager SSD and adapter-fault qualification. The fixed pin is not replaced by an unreviewed upstream-main bundle.

Remaining work

The local raw payload arena protocol and native executor are implemented. Bounded large-plan partitioning and native layer readiness are implemented; broader graph/topology, page-generation and deployment gates remain separate. CUDA IPC metadata still serves Publish and Manager SSD/codec routes. Peer metadata currently uses gRPC. The local global index is synchronized through bounded Manager inventory snapshots/deltas, with etcd limited to membership, and has no foreground discovery RPC. Source grants/completions retain gRPC; TENT retains payload READ/WRITE. The custom native metadata bus is outside the delivery plan. See the communication plan and peer-control boundary.

The next TCP worker candidate owns its callback and batch ID before terminal publication, then avoids task access after the state becomes visible. A staging exception is converted to FAILED and remains subject to OrbitKV’s permanent failure retention. CPU callback/exception controls are separate from delayed batch notification, address reuse and physical transport termination gates.

Native registration error control

Run crates/orbitkv-transfer/tests/native/run_unregister_error.py with --native-build pointing to the matching native CMake build, --runtime to its frozen libraries and --output to a new external evidence directory. The runner uses the build’s exact include paths and definitions, hashes the compiler inputs and actual mapped libraries, and records actual child waits. It does not rebuild or overwrite the runtime libraries. Compilation is bounded to 60 seconds and each CPU scene to 20 seconds; any forced timeout cleanup is recorded separately and cannot establish drain.

The actual engine receives a deterministic test-only removeMemoryBuffer failure through its existing transport test hook. Stable host backing remains alive; no payload transfer or device registration is submitted. The default fixed-runtime gate requires successful strict refusal/retry, partial-transport, reference-count, wildcard, publication-tombstone and batch controls. The historical-defect reproduction scene must fail against the fixed library. Batch controls include failed publication after multiple final removals, same-base wildcard suffix retry, and an older publication completing after newer withdrawals. Completed prefixes remain protected until a successful publication covers their actual withdrawal revision. Use --expect-defect for an archived historical runtime: its strict refusal must fail, and its explicit defect reproduction must succeed. Neither negative control is a qualification pass.

Use --control rdma to call the actual native MR and local buffer owners with an injected verbs refusal. These controls require retention of failed MRs and retry of unfinished rails only. -fno-access-control is limited to the test executable; it does not expose new production APIs or replace production method implementations. Use a complete build with matching headers, not a library combined with independently patched transport translation units.

Add --build-rust-control to the engine gate to emit unregister-control.so. With that matching CPU runtime, run the ignored Rust owner boundary explicitly:

MC_FORCE_TCP=1 ORBITKV_MOONCAKE_LIB_DIR=/path/to/frozen/cpu/lib \
ORBITKV_UNREGISTER_CONTROL_LIBRARY=/external/control/unregister-control.so \
cargo test -p orbitkv-transfer --no-default-features --lib \
  failed_native_unregister_preserves_the_rust_registration_until_retry \
  -- --ignored --nocapture --test-threads=1

The production unregister subset adapts existing upstream #4573 and #4249; see the original ownership notes and the current native pin. It preserves unfinished transport cleanup and failed-publication tombstones, rejects same-base registration during retirement and returns failures through the C ABI to Rust try_unregister. Partial registration rollback, consuming unregister/Drop, terminal destruction and physical-failure reclamation remain unqualified. Test-only owner reconciliation is not a production drain receipt. These controls are distinct from the earlier real TCP admitted-READ/unregister counterexample and do not qualify RNIC failure, device drain or allocation/page reuse.