# RDMA and CUDA

This guide describes the current source. Start with [Build](build.md) for
provider dependencies and [Architecture](architecture.md#source-reading-order)
for the code layout. Executed profiles and their limits live in
[Qualification](qualification.md); the [completion plan](completion-plan.md#p4-remaining-delivery)
is the only P4 execution queue.

## Build and select a path

The default crate supports CPU/TCP without a native SDK. The independent
`rdma` and `cuda` Cargo features enable libibverbs and CUDA Driver API bindings.
Runtime use also requires the actual driver, provider and accessible devices.
OrbitTE's engine, connection protocol, progress and resource owners are Rust;
there is no Mooncake runtime dependency or C bridge in the production crate.

Each `TransferRequest` explicitly selects `Transport::Tcp` or `Transport::Rdma`.
An unavailable RDMA path fails preflight. Host policy selects the backing used
by RDMA:

| Memory and policy | Payload path |
| --- | --- |
| Host, `RdmaHostMemory::Direct` (default) | Application host range ↔ RC RDMA READ/WRITE ↔ peer registered range |
| Host, `RdmaHostMemory::Staged` | Application bytes ↔ private registered host staging ↔ RDMA |
| NVIDIA device, `RdmaCudaMemory::Staged` (default) | Device ↔ owned host staging ↔ RDMA |
| NVIDIA device, `RdmaCudaMemory::Direct` | Device MR ↔ RC RDMA READ/WRITE ↔ peer registered range |
| NVIDIA device over TCP | Device ↔ owned host staging ↔ socket |
| Host over TCP | Application bytes ↔ owned host buffers ↔ socket |

Each peer selects its host and device policies independently. Failed direct MR
registration returns an error; it never changes the request to staging. CUDA
IPC and NVLink use a separate [cooperative transport](cuda-ipc.md). Direct GPU support requires actual
PeerDirect-capable GPU/RNIC registration and the prerequisites below.

Opt-in [remote CUDA staging](staging.md#serial-remote-staging) coordinates
Pin/Delegate/Unpin over the same control channel, while TCP and RDMA retain
payload ownership. The serial profile passes 16 installed H20 cases; the bounded
up-to-16-slot pipeline passes 24 fresh cases on stable and Rust 1.85. See
[qualification](qualification.md#bounded-remote-cuda-pipeline) for exact artifacts,
review status and limits.

TCP and staged RDMA use the shared [local CUDA staging pool](staging.md) for
GPU copies. The pool bounds reusable pinned chunks; existing payload buffers,
MRs and RDMA quotas keep their own lifetimes.

## Device, port, GID and host topology discovery

Set `EngineConfig.rdma` to a configured `RdmaConfig`:

| Configuration | Behavior |
| --- | --- |
| `RdmaConfig::default()` | Discover permitted devices, ports and valid GIDs |
| `RdmaConfig::new("mlx5_0")` | Strictly pin that device, port 1 and GID index 0 |
| `device`, `port`, `gid_index` | `Some(...)` is a strict pin; `None` enables selection for that field |
| `rdma_whitelist`, `rdma_blacklist` | Filter devices before opening; a nonempty whitelist takes precedence |
| `device_mask` | Restrict dispatch by stable topology NIC IDs, up to 64 devices |
| `preferred_numa_node` | Caller-supplied memory locality hint |

Discovery prefers active routes and retains otherwise usable paused ports.
Each selected context owns its actual device, protection domain (PD), key
history and address. A failed slot never renumbers the other NIC IDs. A partial
construction whose cleanup fails retains its owners and fences admission.

The selector intersects the configured mask, active local contexts and
compatible remote paths. It prefers the best topology tier, then fewer charged
in-flight bytes, rotating ties. Compatible same-name NIC pairs take precedence;
other compatible paths use stable NIC order. A matching name or address family
does not prove physical reachability. Allocation page placement, PCIe distance
and bandwidth estimates are not inferred from the NUMA hint.

The control protocol publishes bounded NIC/PD path metadata without memory
authority. Actual connection descriptors must match the selected path. Explicit
pins never silently fall back to another device, port or GID. See
[initially paused contexts](#initially-paused-contexts-and-gid-reselection) for
the recovery rules.

## Eager publication and cross-rail slice completion

With host Direct policy, registration prepares physical MRs on every usable
context, including paused ports, before exposing the registered range. A
local-only view uses one MR per context; a remotely accessible view uses two:
a local-only MR and a separate `LOCAL_WRITE|MW_BIND` parent. Dispatch masks do
not reduce this registration fan-out. Neither parent key is exported.

Batch visibility and duplicate-reference increments commit only after every
required MR is ready. Failure retires the new copies while preserving previously
published registrations. Failed rollback retains its actual MR, backing, PD
and budget. Published active copies cannot be evicted. Final unregister hides
lookup visibility and retires every copy; already admitted work retains its
exact allocation and MR borrow through native drain.

The MR cache is keyed by actual context/PD, registered view identity and
capability. Two roles may therefore charge two full-view MRs for the same
allocation. An idle MR holds no DMA exclusion lease. Each grant separately
locks its exact allocation-relative range, so disjoint ranges may proceed and
conflicting CPU/DMA access through the same owner is rejected across aliases.

Requests use TENT's bounded slice geometry: a 64 KiB base block by default,
at most 64 slices for host-local READ and 32 for WRITE or device-local READ.
The block adapts for large requests; a tail shorter than one quarter of the
chosen block folds into its predecessor. CUDA-pinned host memory counts as
host memory. Peer limits may reduce the effective slice count.

`DeviceSelector` selects a compatible rail for each slice. The source validates
exact ordered coverage and reserves the whole parent's grant and receive
capacity before publishing authority. Every child grant must be ready before
payload submission. The parent reports success and its byte count only after
all slices complete, including their source receipts. Cleanup visits every
slice even after one rail fails; an unquiesced child prevents a parent drain
claim. Earlier slices may already have changed bytes, so a multi-rail WRITE
is not atomic.

## Connections, grants and completion

Host Direct peers negotiate a fixed reusable QP width equal to the smaller
`num_lanes` value. `EndpointStore` provides a canonical peer array, and each
cold builder reserves a slot before creating a native QP. A slot stays occupied
through construction, use and unresolved native destruction. Waiters use their
own deadlines without cancelling the builder or holding store locks across I/O.

Arrays are scoped by local context, remote process/PD/NIC, role and READ/WRITE
capability. Concurrent grants share published QPs; successful records can
retire independently. The physical QP becomes idle only after all affiliated
records finish. A direct GPU requester can reuse a direct host source QP. If either peer uses
staging, or the source uses direct GPU memory, connections remain per-grant and
require physical close barriers.

The current OrbitTE protocol is v19. Every phase binds both process and
connection incarnations, the exact request and an independent grant generation.
This protocol is separate from native TENT API/wire compatibility.

For host/staged source memory, the source binds a one-use Type-2B memory window (MW) to the exact range and
direction, consumes its successful BIND completion, and posts a real completion
receive before publishing the grant. Type-2B MW support is required; a remotely
enabled parent MR is not a fallback. The PD retains a bounded, non-evicting key
history so an old grant key cannot regain authority within that domain.
When a native MW index has no fresh low-byte tag, the context keeps that newly
allocated, never-bound MW alive and allocates a replacement from the same PD.
Keeping the exhausted index allocated prevents immediate provider reuse without
clearing key history. One parked MW per index and the historical index count
are both bounded by `max_window_key_indices`; duplicate live indices, exhausted
capacity or allocation failure reject admission. `exhausted_windows` counts
these PD-owned MWs inside `active_windows`. Close destroys them before the PD;
failed destruction retains the remaining MWs and PD without a Drop retry.

The requester submits bounded, signaled READ/WRITE work. After every payload
CQE succeeds, it posts a fenced `SEND_WITH_INV` on the same QP. The source must
receive the exact 141-byte binding and invalidated key through its actual RQ
CQE. A SEND CQE or control-plane DONE alone cannot establish source completion.
Only after the bound DONE and native receipt agree does the source clean its
MW and grant, send ACK, and publish READY for reuse.

`RdmaQp` owns SQ depth, RQ buffers and exact outstanding WR identities.
Independent grant records borrow queue capacity. A received buffer identifies
the native slot; the message binding identifies the completed grant, which may
be a different record. Checked WR namespaces and CQ/QPN validation prevent
old or duplicate completions from paying current debt.

Only an explicit pre-authority cache miss permits bounded connection bootstrap
retry on the selected PDs. An ambiguous WRITE or payload failure is never
replayed automatically.

## Workers, CQ lanes and budgets

One transport-level `Workers` pool progresses all contexts. `num_lanes`
controls the data worker count and the SQ/RQ CQ pairs per context. New slices
use worker load; a reused QP stays on its original CQ lane. Each CQ has one
polling owner. Data workers bind to allowed NUMA CPU sets; synchronous native
calls can still delay other work on that worker.

`RdmaProgressMode::Polling` is the default. `EventDriven` arms both CQs before
polling and uses native events to wake the existing data worker. One transport
`Monitor` consumes and acknowledges device/CQ events and scans contexts each
second. Native data work and fallible destruction stay with the data owners.

| Budget | Default | Charged resource |
| --- | --- | --- |
| `max_endpoints` | 64 | Aggregate physical QPs, including retained connections |
| `max_grants` | 64 | Whole-parent grant reservations, independent of QP count |
| `max_send_wr` | 16 | Unresolved signaled sends per physical QP |
| `max_cqes` | 1024 | Each CQ's capacity; aggregate SQ and RQ debt are bounded separately |
| `max_direct_bytes`, `max_staging_bytes` | 64 MiB each | Active direct loans and private staging respectively |
| `max_cached_mrs`, `max_cached_mr_bytes` | 64 / 64 MiB | All full-view physical MR copies across contexts and capabilities |
| `max_window_key_indices` | 65,536 | Historical MW key indices and parked exhausted MWs in each PD domain |

Creating, closing and retained resources remain charged. Native CQ completion
repays its exact WR debt; receive admission remains charged until the actual
message MR is reaped. Returning aggregate capacity wakes the owning lanes.
Adding a context does not multiply the aggregate posted-work budget. These
limits describe admission, not a throughput or fairness guarantee.

## Failure, drain and resource ownership

Post failure releases only a proven unaccepted suffix. An unknown `bad_wr`,
invalid CQE, timeout or CQ error leaves unresolved work charged and fences
affected admission. Physical cancellation includes all grants affiliated with
the failed QP; uncertain debt requires the native QP/CQ drain barriers.
Healthy sibling owners can finish their own cleanup.

Parent cleanup has one shared thirty-second destruction window, separate from
the data deadline. A timeout does not prove physical termination. Native
teardown failure retains the remaining resources, backing and charges and is
not retried by another close or Drop. A fully drained and detached progress
owner can be replaced while preserving its context's PD and persistent MRs;
quarantined native failures cannot be healed this way.

`TransferEngine::close(timeout)` fences source and requester admission, waits
for accepted work, retires connections and MRs, drains CQs and workers, then
destroys PDs and contexts. Its shared deadline can expire while the native
owner remains live; a later close can continue waiting. `close_source` and
`close_requester` retain their directional contracts and do not supply a
whole-engine PD/context teardown receipt.

`Error::Unquiesced` means physical completion or destruction is unproven.
Logical failure can therefore coexist with `locally_drained == false`; batch
capacity, registration owners and DMA loans remain retained. Successful engine
close does not free caller-held batch handles or unregister caller-held memory.
Native provider calls are synchronous and cannot be forcibly interrupted by an
API deadline. See [native cleanup](native-cleanup.md) for tested failure owners
and [async batches](async-batches.md) for caller-visible completion.

## Periodic monitor recovery

The monitor queries selected routes even without new submissions, covering
missed port events. A changed or inactive address fences route admission. Old
builders, QPs and CQs must physically drain and detach before a validated route
is published under the unchanged fault generation. Recovery preserves PD
identity, key history and persistent MRs.

`monitor_ticks`, `route_queries` and `route_recoveries` distinguish scans,
native queries and resumed routes. `progress_recoveries` counts reclaimed,
fully drained lane owners. Busy owners can defer a tick; the one-second scan
interval is not a real-time recovery guarantee. Fatal device events, lost event
consumers and retained cleanup failures remain fenced.

### Initially paused contexts and GID reselection

An initially inactive port with valid provider/GID metadata retains a paused
context and PD. Host Direct registration prepares its MR copies even when all
selected ports are paused, while path selection and QP creation reject that
route. A healthy sibling rail remains eligible. Stable NIC IDs and MR fan-out
do not change during recovery.

The context retains configured `gid_index` separately from its selected address.
Healthy ticks query the selected index. Paused automatic recovery (`None`)
reranks valid GIDs on the same device and port; `Some(index)` remains strict.
This follows frozen TENT's `RdmaContext::enable` and `refreshAddress`.
OrbitTE's ranking also includes real IPv6 RoCEv2. New-device hotplug and general
cross-rail retry policy remain outside this behavior.

CPU controls cover these transitions. The focused native gate passes with
stable 1.99.1 and Rust 1.85.0 on one mlx5 RoCE RNIC: automatic recovery
selects GID 3 from initial GID 0, while the explicit pin stays at 0. It uses
real PD/MR/QP/CQ operations and native GID queries with a process-local
override of the observed port state. It sends no payload and does not
change a physical port. See [the frozen evidence](qualification.md#focused-native-recovery)
for source equivalence, provider maps, supervision and remaining acceptance.

On an authorized RNIC, inspect its GID table and choose a valid
`ORBITTE_INITIAL_GID` different from the automatically preferred index. Supply
the existing exclusive resource-lock file and an external log destination:

```sh
ORBITTE_RDMA_LOCK=/path/to/authorized-rnic.lock \
ORBITTE_SHARED_DEVICE=mlx5_0 ORBITTE_SHARED_PORT=1 ORBITTE_INITIAL_GID=0 \
CC=gcc CXX=g++ cargo +stable test --locked --features rdma \
  transport::rdma::context::tests::native_context_recovery::native_context_recovery_preserves_pd_mrs_and_strict_gid_selection \
  -- --exact --ignored --nocapture --test-threads=1 \
  > /path/to/external-artifacts/context-recovery.log 2>&1
```

Both automatic and pinned cases register two persistent MRs while paused,
resume through monitor queries, drain a native QP/CQ on the next pause, and
resume with a fresh connection incarnation. Receipts require unchanged native
context/PD/domain and MR counts, then zero live or retained native owners at
close. This gate cannot replace two-host payload or physical-loss evidence.

## NVIDIA memory ownership

`CudaMemory::allocate` owns device memory and its primary-context reference.
Unsafe `CudaMemory::from_raw_parts` retains the supplied allocation and context
owner, validates device type/context/ordinal/extent, and rejects host or managed
pointers. Aliases must share one `Arc<CudaMemory>`. CUDA batch registration uses
the same transactional bounds, permission and retired-owner accounting as host
registration.

`cuda::allocate_pinned_host` returns an owned `HostMemory` allocation with its
actual CUDA deallocator and context. Ordinary host registration does not pin
memory through CUDA. The feature uses generated Driver API signatures and
selective fallible loading from `libcuda.so.1`; it does not require nvcc,
libcudart, cuBLAS or NVRTC to build.

Callers finish external producers and exclude conflicting external access before
handoff, then coordinate consumers through their own stream/lifecycle contract.
These APIs do not infer framework readiness. Failed copy synchronization,
context restore or unpin retains the actual owner and is observable through
`cuda::retained_resources`.

### Copy completion and reuse

Each `CudaMemory` lazily owns one nonblocking CUDA stream and completion event.
READ/WRITE staging uses async Driver copies, records the event after the copy,
and waits for that event before exposing host bytes or releasing staging. The
allocation mutex serializes aliases and event reuse. Other streams are not part
of that completion receipt; caller producer handoff remains explicit.

Failed copy submission or event recording attempts to drain this same stream.
If drain cannot be proven, the operation returns `Unquiesced` and retains its
actual allocation, pinned staging and stream/event owners. Later access cannot
reuse them. `cuda::retained_resources` includes `streams` and `events`; cleanup
failure preserves the corresponding context and unresolved handles. Initial
allocation zeroing still uses context synchronization.

This follows CUDA's [event capture and completion semantics](https://docs.nvidia.com/cuda/archive/13.0.2/cuda-driver-api/group__CUDA__EVENT.html):
only an event successfully recorded after the copy proves its completion.
Destroying an event or stream is cleanup, not a DMA completion receipt.

Run the explicit byte/guard gate only on allocated, locked GPUs:

```sh
cargo build --locked --release --features cuda --example cuda_copy
# Ordinals refer to the selected process-visible devices.
target/release/examples/cuda_copy 0 1
```

The runner checks three unaligned sizes, eight concurrent callers and four
rounds per caller, then releases all owners. It qualifies local staging only;
GPUDirect and interprocess GPU sharing use their own separate installed gates.

## GPUDirect RDMA ownership

Build with `--features rdma,cuda`, set `MLX5_SCATTER_TO_CQE=0` before process
startup, and select `RdmaConfig.cuda_memory = RdmaCudaMemory::Direct`. Keep the
provider environment unchanged while any engine is alive. Initialization rejects
an absent or different setting: mlx5 scatter-to-CQE can otherwise copy small
RDMA READ results into the SGE from the CPU. This is incompatible with ordinary
device pointers ([provider guidance](https://github.com/linux-rdma/perftest#readme)).
No process-wide environment mutation or automatic staging fallback occurs.

`CudaMemory` validates the device allocation and context. Before native
registration it sets `CU_POINTER_ATTRIBUTE_SYNC_MEMOPS` once and checks device
GPUDirect write-ordering/host-flush capabilities. `LocalBufferManager` publishes
one local-only GPU MR per selected context transactionally, under the owning
CUDA context. Requester slices borrow these persistent MRs. Allocation-wide
`DmaLease` ranges reject overlapping copies or grants across registration views
and engines; they never expose a GPU pointer as a CPU byte slice.

NVIDIA [PeerDirect does not support Memory Windows](https://docs.nvidia.com/networking/display/mlnxofedv23102131201lts/optimized%2Bmemory%2Baccess).
A GPU source grant therefore owns a fresh PD, exact-range MR with only the
requested remote permission, and a fresh QP. Its RQ message MR uses that same
PD. The device context, CQ lanes and transport Workers remain shared. Protocol
profile 2 binds the separate private-PD connection identity and uses fenced
ordinary SEND completion; it never pretends that a normal MR supports remote
invalidation. The public NIC path still selects and validates the device route.

Source ACK follows actual RQ completion and matching DONE, QP destruction,
payload/message MR deregistration, private PD deallocation, and GPU write
visibility. Requester completion likewise establishes visibility before
releasing its DMA range. Hardware owner ordering is used when advertised;
otherwise `cuFlushGPUDirectRDMAWrites` targets the current owning context.
Any uncertain native cleanup, flush or context restoration retains the real
allocation, range loan and associated admission. External producer/consumer
stream handoff remains the caller's responsibility; SYNC_MEMOPS does not wait
for arbitrary kernels ([CUDA ordering contract](https://docs.nvidia.com/cuda/gpudirect-rdma/#synchronization-and-memory-ordering)).

This follows frozen TENT's `LocalBufferManager` → `RdmaContext::registerMemReg`
registration and `Workers`/slice progress boundaries. TENT publishes native MR
keys; OrbitTE's exact per-request authorization additionally needs private PDs
for GPU source grants. GPU source MRs/QPs are currently per-grant and are not
cached. Host source MW/QP reuse is retained; a GPU requester can reuse a pair
when the source is direct host memory. Registration failure on any context fails
the batch publication. The [bounded installed H20/RoCE profile](qualification.md#gpudirect-rdma)
covers two contexts and preserves an unqualified host-wide context inventory
limit. Other GPU/provider, native GPU fault and performance claims remain open.

The explicit `hardware_peer` gate accepts `ORBITTE_RDMA_CUDA_MEMORY=direct` and
optional `ORBITTE_RDMA_SLICE_BYTES` for 1/17/32/64-byte controls. The existing
supervisor owns GPU/RNIC locks and process reaping. Its passive native observer
records real MR type/flags, QP PD and payload WR addresses; `gdr.py` checks those
against declared allocations and full-buffer/guard results. Evidence belongs
outside the checkout; see [qualification](qualification.md).

Build the installed GPU consumer from a clean commit, then pass both its
manifest and passive observer to the controller:

```sh
CC=gcc CXX=g++ python3 tests/hardware/build_installed.py --consumer gpu \
  --toolchain stable --output /external/installed-gpu-stable
python3 -B tests/hardware/run.py --profile /external/gpu-profile.json \
  --binary /external/installed-gpu-stable/installed/bin/orbitte-hardware-peer \
  --installation /external/installed-gpu-stable/manifest.json \
  --gdr-observer /external/installed-gpu-stable/liborbitte_cq_fault.so \
  --output /external/runs/gpu-direct-001
```

Use `--toolchain 1.85.0` and a new output directory for the MSRV consumer. Each
GPU case declares `cuda_memory: "direct"`, `transport: "rdma"`, and memory
kinds `cpu`/`cuda:0`, with at least one GPU. The controller sets the required
scatter environment before spawning the native process. Nodes must have
distinct pinned hostnames and boot IDs. Small cases use `payload: 4096` and
`slice_bytes` of 1, 17, 32 or 64. The two-rail folded-tail profile adds `rails`
and both original NIC locks, `payload: 8388768`, `rdma_block_size: 1048576`,
and `expected_grants: 12`. Each native process runs three READ requests followed
by three WRITE requests. Run both physical directions and both values of
`rdma_events`; preserve failed runs. The observer does not simulate transfers
or change provider return values. This profile proves correctness and ownership,
not bandwidth or GPU-native fault recovery.

## Run a native profile

Resolve the current platform attempt and collect fresh host/provider/device
inventory before choosing execution nodes. Use an external JSON profile with
[`tests/hardware/run.py`](../tests/hardware/run.py):

```sh
python3 tests/hardware/run.py --profile /external/profile.json \
  --binary /external/build/release/examples/hardware_peer \
  --output /external/runs/run-001
```

The basic profile contains `ssh_config`, `rnic`, `gid_index`, `nodes` and
`cases`. Each node supplies an existing SSH `alias`, exact `hostname` and
`boot_id`, a `bind` address and the original `nic_lock`. GPU cases additionally
supply `gpu_lock` and `gpu_uuid`; only that UUID is exposed, so the memory kind
uses ordinal 0. Optional node `env` freezes native library paths.

Each basic case selects `source`, `requester`, `source_kind`, `requester_kind`,
`transport`, `payload` and `seed`. The `hardware_peer` example supports `cpu`,
`pinned:0` and `cuda:0`, compares complete bytes and guards, and checks drain
after noncontiguous READ/WRITE operations. Optional `host_memory` and per-peer
overrides select Direct or Staged. This correctness pilot is separate from
the installed and fault profiles.

For `orbitte-multirail` and `orbitte-installed-host`, profiles supply exactly
two distinct physical `rails` and each node's `nic_locks` map covering both.
The runner validates each mode's fields and exact native test selector before
deployment. Use the maintained [installed host recipe](installed-host.md) for
packaged public consumers and [native cleanup recipe](native-cleanup.md) for
instrumented libtests. Their artifacts and proof obligations are separate.

The controller verifies host/boot identity and binary hashes, deploys into a
unique `/var/tmp/orbitte-runs` directory, and collects actual provider mappings.
Supervisors hold nonblocking resource locks until `waitpid` confirms their
owned children's exit. A missed reap, broken supervision or missing receipt
fails cleanup qualification. Only owned children can be terminated; the runner
does not change physical NIC configuration or platform resource allocation.
Keep raw runs, manifests and failed controls outside the source checkout.

## Reference and historical records

The frozen TENT reference remains
`e2d6e8e0f2df113d84f9e39379f98c4b67b199ab`. See the
[owner mapping](architecture.md#rdma-resource-owners) for `RdmaTransport`,
contexts, buffer managers, endpoints, workers and monitor responsibilities.
Exact-range MW authorization, source receipts and conservative failed-cleanup
retention are explicit Rust contracts; architectural correspondence does not
establish TENT wire compatibility.

The [pre-cleanup guide](https://github.com/OrbitInfra/OrbitTE/blob/9e1fdeaa9f7cd6c03e30aab2dff19a00f0a44578/docs/rdma-cuda.md)
preserves all earlier stage-specific recipes, source revisions and evidence
locations, including historical OrbitKV experiments. Those records describe
their original revisions. [Qualification](qualification.md) indexes the current
evidence without treating old filenames, protocols or open-item lists as current
contracts.