CUDA staging

runtime/proxy_manager.rs::ProxyManager owns an engine-wide pool of reusable CUDA-pinned host chunks. TCP and staged RDMA share that pool for device-to-host and host-to-device copies. Each payload transport still owns its network buffers, MRs, grants and completion receipts.

Configuration and lifetime

With the cuda feature, EngineConfig.staging_chunk_size defaults to 4 MiB and staging_chunk_count to 64. Allocation is lazy: one local logical GPU copy checks out one chunk and processes its full range, including the final short chunk. The actual CudaMemory allocation lock remains held across all chunks. Public CudaMemory::read/write retain their existing one-shot staging behavior.

The pool bounds pinned copy bytes across source and requester operations and all local CUDA contexts. Size/count must be positive, size at most 64 MiB, count at most 4096, and their product at most 1 GiB. Active, constructing, closing and retained chunks count against the bound. Capacity exhaustion returns ResourceExhausted; it does not create an uncharged allocation or wait for a nested request. TCP payload buffers and RDMA staging/MR quotas remain separate.

Idle chunks reuse their retained native CUDA context. A different context may replace an idle chunk only after its native release and caller-context restore both succeed. Native allocation, copy and cleanup never run under the pool map lock. The retained context prevents address reuse while a chunk is cached.

A lease becomes unsafe to return before native submission. Only event completion (or recovery stream drain) and caller-context restoration permit return. Failed completion retains the actual chunk, native context, GPU owner and parent admission. Partial native allocation and failed native cleanup also keep their slot charged. Successful free followed by a failed context pop is recorded separately; another close or Drop never repeats the free.

Deadlines are checked at each drained chunk boundary. They cannot preempt a native CUDA call. A partially copied logical request fails with no completed request bytes; partial destination effects are possible and must not be replayed automatically. Cancellation keeps the existing task boundary: queued requests may be cancelled; a running logical copy continues through drain.

Directional close_source and close_requester leave the shared pool available to the other direction. Full TransferEngine::close fences and drains both, then closes idle chunks. A timeout permits another drain wait; uncertain native cleanup remains retained. EngineStats.staging exposes reserved/idle/active/ retained chunk counts and allocation, reuse, completed-chunk and byte counters. Chunk byte counters describe local copy progress, including a completed prefix of a failed logical request; batch completion remains a separate contract.

TENT correspondence

The frozen TENT ProxyManager owns location-specific StageBuffers, chunk Pin/Unpin and staged work. Its implementation separates PRE/CROSS/POST work, remote staging operations and deferred cleanup.

P5.5a implements the local chunk owner and consumes it in TCP/RDMA CUDA copies. It uses one global engine bound with retained-context chunks rather than an independent budget per location. cuda/staging.rs owns the local copy loop; cuda/stream.rs retains the allocation’s stream/event completion authority. There is no extra Proxy worker pool in this substage.

P5.5b adds an opt-in remote operation with serial and bounded pipeline modes. The names and phases follow TENT’s pinStageBuffer, delegateAsync, submitCrossStage and RemoteStageOperation: the latter distinguishes an observed remote result from an unconfirmed operation before deferred cleanup. OrbitTE uses scoped receipts and an inline source handler to preserve that distinction with bounded resources. The new bounded remote mode follows transferEventLoop’s ring of at most 16 stage buffers (id % kStageBuffers). source_task.rs owns source Chunk scheduling; requester_task.rs reserves the local ring and full GPU range loan. requester_task.rs also owns local PRE/POST scheduling and bounded CROSS queues. staging_task.rs::StagingTask owns the connection and ordered remote phases; worker_shard.rs queues these sessions on shared WorkerShards. StageBufferCache retains the request’s private registrations and buffers, following the resource role of TENT’s cache of the same name. stage_registration.rs::PinnedStage owns the actual pinned lease, private registration, pending copy and ordered retirement on either side. remote_stage_operation.rs owns scoped messages and receipts. Per-slot CUDA completion supplies the PRE/POST completion boundary, while TCP/RDMA still own CROSS.

The current implementation can keep both source and requester CUDA copies pending during CROSS. Independent local and remote rings follow TENT’s local_locked / remote_locked ownership: local WRITE slots become reusable after CROSS, and local READ slots after POST. The requester event loop tracks at most L chunks and submits CROSS commands in logical order, while querying each local PRE/POST independently. A ready slot immediately starts its next chunk; there is no whole-window refill barrier. Local progress continues while a CROSS or Delegate receipt is pending. The worker still serializes CROSS and scoped receipts on one connection. Frozen TENT creates eight shared worker shards; each dequeues one task and runs its chunk event loop to completion. OrbitTE starts at most eight shards on demand, reuses an idle shard and otherwise selects the smallest queue with rotating ties. Each shard still executes one session at a time. TENT’s transferEventLoop polls PRE/CROSS/POST batches and remote operations, releasing each local buffer at its own phase boundary. OrbitTE uses the same chunk phases for requester progress, with CopyStream querying the recorded CUDA event and the connection worker publishing CROSS receipts. Idle sweeps follow TENT’s 128 yields followed by exponential delays from one to 200 microseconds, bounded by the deadline. Concurrent CROSS on the wire and general remote-operation/location scheduling remain in the existing P5/P6 queue. CPU controls establish scheduling and owner lifetime; physical DMA overlap and throughput require hardware measurement.

Serial remote staging

Set EngineConfig.remote_staging = true on both peers, built with cuda. This explicitly selects remote GPU staging for TCP/RDMA submissions; the remote registered range must be CUDA. Requester ranges may be CPU or CUDA. NVLink keeps its own path. Remote host ranges, an opted-out source and a config combining this flag with RdmaCudaMemory::Direct are rejected. No automatic fallback or new payload transport is introduced. The flag defaults to false.

Phase	READ	WRITE
Local preparation	Reserve one bounded private slot and borrow the full local GPU range	Same; no full-request snapshot
Pin	Validate source epoch/region/range/permission; borrow its GPU range and pin its slots	Same
PRE	Source GPU → pinned source slot, with event/context completion before CROSS	Local range → requester slot, with event/context completion before CROSS
CROSS	TCP/RDMA source slot → requester slot	Requester slot → TCP/RDMA source slot
POST	After the scoped CROSS receipt, requester slot → local range	Delegate source slot → source GPU
Unpin	Drain accepted copies, retire private registrations/MRs and release source slots/range, then acknowledge	Same
Request completion	Require final Unpin receipt and requester slot/MR retirement	Same

One control connection and one source admission own the entire session. At depth one, CROSS commands are sequential, including the short tail. The local next PRE or current POST can start while the worker awaits a remote Delegate receipt. Nested CROSS calls the existing transport directly; it never recursively submits an ordinary batch or waits for a second handler. One handler, one parent batch slot and one pinned source chunk per engine are sufficient. A GPU requester also needs one local pinned chunk; self-peer and opposite-direction sessions share their engine pool. With pipelining enabled, each engine needs its outgoing local ring plus its incoming source ring; self-peer capacity is L + R below. Insufficient capacity fails immediately, without waiting for a slot held by the same session. Native RDMA grant/CQ/MR budgets still apply. RDMA uses its slices protocol even for a single chunk, retaining its bounded same-connection QP-cache-miss retries and actual verbs payload path.

Each CROSS uses local offset zero in its selected private host slot. At the default depth of one, CUDA requesters reserve one physical pool slot; larger depths reserve a bounded local ring before Pin. Each physical slot retains its stream/event across sessions. CPU requesters still allocate at most min(request.length, staging_chunk_size) host bytes under the parent admission; they do not spend CUDA pool slots. No full-request snapshot or commit payload is allocated. TCP/RDMA retain their independent bounded transport resources.

Pin negotiates min(source chunk size, requester chunk size) bytes per logical chunk. Each side still reserves and reports its actual configured physical slot size. A larger source allocation does not enlarge the negotiated CROSS grant. The tail uses its exact remaining byte count.

cuda::StagingCopy retains the actual Arc<CudaMemory> and an exclusive logical range loan. Its allocation mutex protects only metadata, not network I/O or slot event waits. Nonoverlapping source/requester ranges on the same allocation can progress concurrently. Overlapping staging, CPU or RDMA access fails before new work; whole-allocation IPC export conflicts with every staging range. Self-peer overlap is rejected rather than providing snapshot-copy semantics. A busy metadata mutex fails admission with ResourceExhausted. Pool exhaustion also fails explicitly; no hidden pool or silent reduction of source depth is used.

Wire version 17 introduced PinRead/PinWrite, Delegate, CROSS and Unpin; version 18 added pipeline width. Current version 19 also carries the requester chunk cap in Pin. Both peers must use version 19; older headers are rejected. Each frame binds a fresh nonce, original registration, phase and logical range; success receipts echo that full scope. A repeated prior receipt cannot stand in for Unpin. Private chunk descriptors never appear in Describe and cannot be used outside their session. These identities fence reuse; they are not authentication.

CROSS receipts follow actual transport grant cleanup. Unpin first unregisters the private view, retires every native MR and verifies no private registration borrower remains, then explicitly releases the chunk and GPU range loan. A transport ACK alone never completes a remote GPU WRITE. Lost POST/Unpin replies leave WRITE with RemoteEffect::Unknown. READ commits each completed chunk after its scoped CROSS receipt; a lost Unpin receipt still fails the request even when all chunks are visible. A failed request contributes zero completed requests/bytes, but may have changed bytes. Independent local POST streams can complete out of order; a native failure does not establish an exact visible prefix. locally_drained proves physical local completion, not rollback, and failed groups do not send attached notifications. Successful local native drain allows cleanup after EOF, while uncertain CUDA or RDMA cleanup retains actual backing and the responsible parent/source admission. Ordinary confirmed errors remain errors even when locally drained. There is no automatic retry of a possibly executed WRITE.

EngineStats.staging.remote_pins, remote_delegates, remote_crosses and remote_unpins count source-side completed operations. Unpin returns the drained chunk to the pool before its reply; native host unpin normally occurs when the pool closes. A lost reply can coexist with a completed source Unpin. These observations do not replace requester completion receipts.

Bounded remote pipeline

Set EngineConfig.staging_pipeline_depth to 2–16 on both peers to opt in. It defaults to 1 and cannot exceed staging_chunk_count. For a request of N bytes, physical chunk sizes C_local / C_source and configured depth caps D_local / D_source, the rings are:

Quantity	Bound
Logical chunk size C	min(C_local, C_source)
CUDA requester width L, reserved before Pin	min(D_local, ceil(N / C_local))
Source width R, negotiated by Pin	min(D_local, D_source, ceil(N / C))
CPU requester width	One ordinary host buffer, no CUDA pool charge

The requester reserves its entire local ring and full GPU range before opening the control connection. The source checks out its entire negotiated ring before acknowledging Pin. Insufficient capacity fails with ResourceExhausted; neither side silently narrows its ring. A large local physical chunk can conservatively limit L even when the smaller negotiated chunks would permit more concurrency. There is no late expansion after Pin. The requester uses asynchronous copies and queries even at width one. The source keeps its existing synchronous copy at width one and accepted-copy pipeline at larger widths.

Logical chunk i uses local slot i % L and source slot i % R. Each CUDA slot owns its actual pinned allocation, a persistent stream/event and a private registration for the current session. Scoped CROSS grants bind the source slot. Source registrations retire at Unpin; requester registrations retire after its receipt or after draining an errored session. Drained native allocations and stream/events remain in the pool for later sessions. Full engine close destroys those handles before freeing pins. Failed construction or destruction remains charged without a Drop retry.

Direction	Scheduling and completion
READ	Delegate prefills R source PREs. CROSS waits for its source slot and any previous POST in its local slot. Its scoped receipt permits local POST to begin; the freed source slot queues the next PRE at i + R. Other local POSTs can remain pending until local slot reuse.
WRITE	Prefill L local PREs. CROSS waits for its local PRE and any previous POST in the source slot. After CROSS’s scoped receipt and the Delegate send, the freed local slot queues PRE at i + L without waiting for that Delegate receipt; other PREs and source POSTs can remain pending.
Final Unpin	Drain all local POSTs before sending Unpin. The source drains every accepted PRE/POST, retires private registrations/MRs and releases its range before replying. Only the matching receipt and local retirement complete the request.

A pipelined Delegate receipt confirms acceptance, not CUDA completion. remote_delegates advances after the source event wait and context restoration. Requester completed-chunk counters advance once after event query (or error-drain wait) and successful context restoration. remote_crosses still follows actual transport cleanup, and remote_unpins follows full session cleanup. WRITE becomes RemoteEffect::Unknown before sending Delegate and is never automatically replayed after an ambiguous reply.

Session progress and completion

The requester reserves its physical ring and GPU range loan before submitting a session to the shared pool. Existing parent admission remains charged through queueing and completion. At most eight orbitte-staging-{id} threads exist per ProxyManager, with at most 64 waiting sessions per shard. Full queues return ResourceExhausted; no nested public batch is admitted. Queue time consumes the original request deadline. A cancelled/expired queued session is removed and its actual captures are destroyed before terminal publication, without Pin, payload I/O or waiting for an unrelated active native call. An active session negotiates Pin on its shard, then publishes the negotiated chunk size to the caller. Command and CROSS completion queues each hold at most L entries. The event loop retains one chunk state per slot and references at most L registrations. The shard owns the control connection; local CUDA owners remain in the caller. TCP still transports TCP payload; RDMA uses its existing shared verbs workers/CQ lanes and uses TCP only for control.

The caller sweeps Pre, Cross, Post and their in-flight states without blocking on an individual CUDA event or CROSS completion. A WRITE’s PRE must complete before its ordered CROSS; a READ’s POST must complete before its slot can receive another CROSS. Other slots continue to progress. CROSS results are consumed individually and start the corresponding local POST or next PRE. A CROSS result is published only after actual transport grant cleanup and the scoped source receipt; the worker drops its local registration clone first. It sends the required Delegate before publishing CROSS completion, records WRITE uncertainty before that send, then waits for the remote receipt independently of local CUDA progress. CROSS ready releases the local slot, not the remote operation or the whole request.

cuEventQuery returning CUDA_ERROR_NOT_READY retains the exact pending length, event generation, GPU range loan, pinned lease and private MR. Only query success followed by successful context restoration clears that pending copy and advances completion counters. A query error or failed context restoration quarantines the actual slot and makes range failure sticky. Error cleanup still waits for every other accepted copy; success on a sibling cannot clear uncertainty. Ordinary healthy requester progress performs no event synchronization. The CUDA Driver API event contract defines the query result; event destruction is not completion evidence.

After consuming all CROSS results, the caller drains every accepted local copy and queues Unpin. The worker processes Unpin only after every preceding Delegate receipt succeeds. A missing final Delegate receipt therefore fails the request even when every CROSS result was received. On failure, the caller stops queue admission and cancels commands that have not started. The active CROSS continues through the existing transport drain protocol under its original deadline; cancellation does not interrupt its control channel. The caller drops its queue handles and waits for a separate task terminal receipt before private MR retirement. That receipt is sent only after the task closure, command receiver, registration clones and connection have been destroyed. It cannot be blocked by a full CROSS completion queue. Caller unwind drops both chunk channel handles before its task guard cancels and waits. Task panic becomes Unquiesced while the shard remains available; uncertain native errors take precedence over an ordinary local failure and keep actual owners and admission charged.

EngineStats.staging.worker_shards counts live threads, including idle ones; opened_worker_shards records total starts. active_tasks counts executing sessions and queued_tasks counts waiting sessions. The active count can reach zero before terminal delivery; it is not a drain receipt. cancelled_tasks counts local scheduling exits that stop queued work. Counters are independently sampled. close_requester fences admission and waits for accepted calls, leaving idle shards available to the engine owner. close_source does not stop outgoing shards. Successful whole-engine close joins all shards before releasing pooled buffers. If requester/source drain times out, the pool remains available for already accepted work; retry observes the same owners. Native uncertainty can keep that engine and its idle shards retained. At the pool shutdown boundary, admission and lazy creation are fenced together, queued tasks are cancelled and active tasks retain their original drain obligation. A pool close timeout can be waited again without restarting threads. Idle workers hold only their shard state, not a strong engine reference. Timeout never substitutes for CUDA/DMA drain.

EOF, a protocol error or deadline expiry stops new work but drains every accepted copy once, including siblings of a failed copy. PinnedStage takes its pending length exactly once; an absent pending copy performs no native wait. The range owner tracks aggregate accepted copies, while each physical slot owns its stream/event and refuses retirement while a copy is still pending. Ordinary pending work does not fence the entire allocation. Native uncertainty is published as a sticky allocation flag after private host/lease/stream locks are released. A successful sibling never clears it. An uncertain CUDA copy retains the actual GPU owner and range loan. Private MR retirement failure retains the host stage and responsible admission even when CUDA has drained and its GPU range loan can be released. A recovered submission error can release resources after all other slots drain. Unproven completion retains the actual native buffers, registrations and responsible admission. No extra handler or nested batch slot is required.

This source implementation has deterministic CPU controls for multiple pending copies during network progress, depth negotiation, short tails, stream reuse, partial admission, wrong slot/scope, EOF, lost final receipts and failed completion/construction/destruction. CROSS setup and PRE drain errors use the nested transport’s source epoch; Delegate and Unpin use the staging nonce. An error after a partial native wire exchange may still require closing the connection with an unknown remote effect. A requester still writing CROSS may observe a socket error before it can read the source’s error receipt. Long RDMA sessions rotate exhausted MW key indices through the PD-owned key domain. The earlier v18 pipeline profile passes 24 two-host cases on stable/MSRV plus four independent repeats. That exact source and healthy profile are accepted; see qualification. Scheduling overlap and native completion receipts do not establish GPU execution overlap or a performance improvement.

Validation

CPU native-call shims cover TCP/RDMA bytes and guards, chunk tails, context reuse, capacity, cancellation, deadline prefixes, close races and allocation/copy/free failures. Combined CUDA and ordinary CQ failures must preserve Unquiesced so source/requester admission cannot be returned early.

The installed gate extends the existing hardware_peer consumer. Build with python3 tests/hardware/build_installed.py --consumer gpu --toolchain stable --output /external/installed. A fresh two-host profile specifies TCP or staged RDMA, CPU/GPU memory kinds, and staging_chunk_size smaller than each logical request with a nonzero tail. Run tests/hardware/run.py with --installation, --binary, --staging-observer (the manifest’s cuda_observer), --profile and an external --output. The packaged passive CUDA observer records actual pins, chunk copies, event waits, context pops and frees; it never fabricates CUDA completion. Both peers compare full buffers and guards, close all owners, and run under inventoried GPU/RNIC locks and process supervision.

The installed stable/MSRV profile passes 24 two-host H20 cases and four fresh independent repeats, with native chunk allocation, copy and free receipts. Source and this bounded healthy profile are independently accepted. See Qualification for the exact artifacts and limits. CPU failure controls do not qualify physical GPU faults or remote proxy staging.

The serial remote path has CPU controls through the public engine for TCP and RDMA, both memory directions, private MR cleanup failure, CUDA uncertainty, source authorization, wrong scope/order, abandoning a staged WRITE, lost final receipts, self-peer and opposite-direction sessions. These native-call shims do not qualify real GPU/RNIC failures. The installed consumer accepts remote_staging: true in the existing staging profile, requiring a CUDA source, one RDMA slice per chunk and the same passive observer. It additionally checks real remote-operation counters and private MR retirement. The consumer waits for SubmissionStatus.locally_drained before its next incremental submit reuses the one-slot parent admission; task byte completion alone does not release that slot. The CUDA ledger checks all asynchronous copies, including setup/readback, against actual live allocations and completion before cross-thread reuse/free.

Stable/MSRV installed consumers pass 16 two-host H20 cases for this healthy serial profile, with four fresh independent repeats. Source and this bounded profile are independently accepted. The profile and artifact binding are in Qualification. The pipeline has separate evidence below; physical faults and full ProxyManager parity remain open.

Installed pipeline validation

Set staging_pipeline_depth in the hardware profile, or use source_pipeline_depth / requester_pipeline_depth for asymmetric depths. source_chunk_size / requester_chunk_size override the shared staging_chunk_size for physical slots; native copy/grant checks use the negotiated minimum while pinned-byte accounting uses each physical size. The consumer reports its actual configuration; the harness requires the independent widths R and L to match the physical pool slots and persistent stream/event owners on each side. Source private MR counts scale by R; requester private MR counts scale by L (one for CPU requesters). Six sessions therefore create 6 * R and 6 * L private MRs, respectively. Physical pinned bytes use each side’s own chunk size and width. Byte and chunk counts do not scale with width.

The staging-only observer adds context push and stream/event create/destroy receipts plus paired entry/return records for copy, query, wait and cleanup calls. staging_native.py joins each event record generation to its actual stream, thread/context and live GPU/pinned allocations. Completion requires both the matching successful query/wait return and context restoration. NOT_READY keeps the same live generation; stale or repeated terminal receipts fail the gate. Distinct slots may submit and finish in different orders; reusing a slot or entering cleanup on any thread before its completion fails the gate. Pool streams/events must be destroyed before their pinned allocation is freed. The observer forwards all native arguments, outputs and return codes without inserting driver waits.

The frozen bounded profile covers TCP/RDMA, CPU/GPU requesters, both host orientations, asymmetric caps, widths 1/2/3/4/16 and shortening a requested width to the chunk count. Main pipeline cases turn the ring more than three times and include a tail. All six sessions reuse physical slots; setup/readback copies are also checked. Native slot copies are matched by exact address/length multiplicity within each payload phase; setup/update/readback ordering and per-slot generations remain strict. Tests distinguish unconfirmed submissions from physical DMA execution: native receipts do not establish execution overlap or throughput. The historical 24-case installed matrix uses wire v18 and the generation-aware observer. Four fresh independent repeats and their cleanup audit also pass; earlier serial evidence remains bound to its original observer and wire version. The current requester ring requires its own v19 artifact and evidence. Its controller supports independent widths and conservative local capacity, but hardware execution is deferred while new resource requests are paused. The shared worker shards and requester query event loop also require fresh installed evidence; the CPU controls prove ownership and scheduling only. Concurrent CROSS on the wire and location selection remain in P5.5b.