# RDMA and CUDA

This guide describes the current source. See [Build](build.md) for provider
dependencies, [Architecture](architecture.md#source-reading-order) for the code
layout and [Qualification](qualification.md) for executed hardware profiles.

Set `RdmaConfig.disable_gpu_direct_rdma` and `log_slice_affinity` in the
immutable typed or JSON configuration. JSON switches require booleans.
Disabling GPU Direct while explicitly requiring `RdmaCudaMemory::Direct`
returns a configuration error. Affinity diagnostics log each selected NIC pair
and slice length through the `log` facade at debug level.

## Build and select a path

The default crate supports CPU/TCP without a native SDK. The independent `rdma`
and `cuda` features enable runtime-loaded libibverbs and CUDA Driver API bindings.
RDMA install resolves `libibverbs.so.1` once and retains it for the process lifetime;
a CPU-only engine in that build does not load it. A missing library or required
export returns `Unavailable` naming `libibverbs.so.1`. Runtime RDMA still needs
the host provider and devices. All engine, protocol,
progress and resource owners are Rust. A small C shim only queries port rates
from the installed libibverbs headers.

A request names `Transport::Tcp`, `Transport::Rdma` or `Transport::Nvlink`, or
uses `Transport::Unspec` for [transport selection](transport-selection.md). Each
peer's memory policy then picks the backing:

| Memory and policy | Payload path |
| --- | --- |
| Host, `RdmaHostMemory::Direct` (default) | Application range ↔ RC RDMA ↔ peer registered range |
| Host, `RdmaHostMemory::Staged` | Application bytes ↔ private registered host staging ↔ RDMA |
| NVIDIA device, `RdmaCudaMemory::Staged` (default) | Device ↔ owned host staging ↔ RDMA |
| NVIDIA device, `RdmaCudaMemory::Direct` | Device MR ↔ RC RDMA ↔ peer registered range |
| NVIDIA device over TCP | Device ↔ owned host staging ↔ socket |
| Host over TCP | Application bytes ↔ owned host buffers ↔ socket |

A failed direct MR registration returns an error; it never switches to
staging. Same-host GPU sharing uses the [CUDA IPC transport](cuda-ipc.md).
TCP and staged RDMA share the [CUDA staging pool](staging.md), and
[remote staging](staging.md#serial-remote-staging) adds Pin/Delegate/Unpin over
the same control channel.

## GPU Direct registration routes

With `cuda_memory: "direct"`, set `rdma.gpu_direct_registration` to `auto`
(default), `dmabuf` or `peermem`. The typed field takes
`GpuDirectRegistration::{Auto, Dmabuf, Peermem}`. `auto` prefers dma-buf when
CUDA advertises `DMA_BUF_SUPPORTED` and both export and verbs symbols exist;
otherwise it uses peermem only when a supported kernel module is loaded. An explicit
route with missing prerequisites returns `Unavailable`. The default CUDA memory
policy remains `staged`; a failed explicit Direct registration does not change
that policy or retry a device pointer as host memory. An attempted export or
native registration failure is returned without trying another route.

Peermem detection follows [NCCL's module probe](https://github.com/NVIDIA/nccl/blob/3e46c089f4ae4732722a419eb6096aead7e47a30/src/transport/net_ib/gdr.cc):
any one of `/sys/kernel/mm/memory_peers/nv_mem/version`,
`/sys/kernel/mm/memory_peers/nv_mem_nc/version` or
`/sys/module/nvidia_peermem/version` must exist. This covers MLNX_OFED's
`nv_peer_mem` and NVIDIA's `nvidia_peermem`; a module directory alone is not
evidence of support.

`cuMemGetHandleForAddressRange` exports the containing allocation. Its base and
export size are host-page aligned; a legacy allocation's length rounds up to a
host page. VMM uses the owned, fully mapped extent and keeps its existing
`gpuDirectRDMACapable` flag. The MR retains the exact requested length, with
`offset = address - allocation_base` and `iova = address`. This meets the
[CUDA export contract](https://docs.nvidia.com/cuda/archive/13.0.0/cuda-driver-api/group__CUDA__MEM.html)
and the [verbs offset/IOVA rule](https://github.com/linux-rdma/rdma-core/blob/v59.0/libibverbs/man/ibv_reg_mr.3).
It follows [NCCL's host-page registration geometry](https://github.com/NVIDIA/nccl/blob/3e46c089f4ae4732722a419eb6096aead7e47a30/src/transport/net_ib/reg.cc),
while keeping OrbiTE's exact-range grant authority.

Export flags are zero on every driver and platform, preserving CUDA's default
mapping. [NCCL's mapping flag](https://github.com/NVIDIA/nccl/blob/3e46c089f4ae4732722a419eb6096aead7e47a30/src/transport/net.cc)
forces PCIe only for its `ncclTopoGdrModePci` mode:
[topology selection](https://github.com/NVIDIA/nccl/blob/3e46c089f4ae4732722a419eb6096aead7e47a30/src/graph/paths.cc)
requires a local GPU→CPU `PATH_C2C` and a GPU→NIC distance other than `PATH_P2C`.
x86 PCIe and Grace CPU-attached NIC paths use zero. OrbiTE lacks this C2C/PCI
classification, so forced PCIe mapping is **not implemented**. Grace mapping
and route verification remain an installed-gate limit; an export error fails
closed rather than trying peermem.

The optional `ibv_reg_dmabuf_mr` symbol is resolved from loaded libibverbs;
older rdma-core needs no new link-time symbol. Linux
[takes its own dma-buf reference](https://github.com/torvalds/linux/blob/v6.17/drivers/infiniband/core/umem_dmabuf.c)
for the MR. OrbiTE conservatively retains the fd too, under the existing bounded
MR/grant budgets, until deregistration succeeds. It then closes the fd exactly
once. A failed deregistration keeps the MR, fd, allocation, context and charges;
a failed fd close is sticky and retains the remaining owners. Neither destroy
is retried. Shared MRs, private grant PDs and opt-in GPU-source reuse all use
this same registration boundary.

Debug diagnostics report registration ID, length and route, without keys or
addresses. `RdmaStats::{dmabuf_registrations, peermem_registrations}` counts
successful native registrations. `RemoteRegion::gpu_direct_routes()` reports
routes observed for that published region through Describe; it is evidence,
not a release receipt or proof that a future route is usable. The optional
metrics exporter exposes both counters. Both [installed route gates](qualification.md#verification-queue)
remain **Implemented; installed profile unqualified**.

## Device, port, GID and memory topology discovery

Set `EngineConfig.rdma` to an `RdmaConfig`:

| Configuration | Behavior |
| --- | --- |
| `RdmaConfig::default()` | Discover permitted devices, ports and valid GIDs |
| `RdmaConfig::new("mlx5_0")` | Strictly pin that device, port 1 and GID index 0 |
| `device`, `port`, `gid_index` | `Some(...)` pins the field; `None` lets discovery choose |
| `rdma_whitelist`, `rdma_blacklist` | Filter devices before opening; a nonempty whitelist wins |
| `device_mask` | Restrict dispatch to stable NIC IDs (up to 64) |
| `preferred_numa_node` | CPU NUMA hint overriding the owner's location |
| `strict_local_numa` | Reject known cross-NUMA NICs (default `false`); unknown locality stays eligible |
| `priority_matrix` | Inline TENT matrix JSON; wins over `custom_json_path` |
| `custom_json_path` | TENT matrix or native `nics`/`mems` file, read at initialization |
| `scheduling` | TENT `SchedulingParams` for bandwidth-aware selection |

Discovery prefers active routes and keeps usable paused ports. Each context owns
its device, protection domain (PD), key history and address; a failed slot never
renumbers other NIC IDs. Explicit pins never fall back to another device, port
or GID.

With `cuda` as well, discovery lists CUDA-visible devices without creating
contexts. Each `cuda:N` entry carries its PCI bus ID, optional NUMA node and
three GPU-to-RNIC tiers: nearest known PCI bridge, other same-NUMA NICs, then the
rest. Missing sysfs distance never counts as nearest. No CUDA driver leaves the
host inventory usable.

`DeviceSelector` compiles location priorities once. Each request supplies its
owner's location (`cpu:N`, `cuda:N` or unknown `*`). Selection intersects the
mask, healthy contexts, compatible peer paths and strict-NUMA eligibility, then
picks the lowest estimated service time. A matching NIC name or address family
does not prove physical reachability, and strict locality cannot be overridden
by a mask. See [qualification](qualification.md#cuda-topology-and-rdma-selection).

## Bandwidth-aware RDMA selection

Smart selection, following TENT's `DeviceSelector`, is on by default. For each
eligible NIC:

```text
score = (charged_inflight_bytes + slice_bytes) / selection_bandwidth_bytes_per_second
        * numa_tier_weight
```

A single slice takes the lowest score, with rotating ties. A multi-slice parent
follows TENT's `selectMultiPath`: each NIC gets `floor(weight / total_weight *
slice_count)` slices with `weight = 1 / (score + 1e-12)`, and leftovers go to the
best NIC. Tier weights default to `[1, 5, 10]`. Every 100th multi-slice parent
probes eligible NICs round-robin; probes never reach failed rails. Set
`scheduling.smart_selection = false` for plain round-robin within the first tier.

All of a parent's byte charges commit under one selector lock and move to each
grant's `DeviceLease`; abandonment returns only unconsumed charges. Eligibility
rules stay hard limits: weights can favor an idle lower-ranked NIC but never
admit an excluded one.

Port rates come from `ibv_query_port` at initialization and on each monitor
scan, preferring the extended speed encoding (including XDR) and the optional
`ibv_query_port_speed` LAG rate. A failed query keeps the last effective value.
Unknown or out-of-range rates use `default_bandwidth_gbps` (400). Topology
attributes `rdma.link_speed_mbps` and `rdma.effective_speed_mbps` expose the
result. See [qualification](qualification.md#rdma-effective-link-rate).

Each NIC keeps two independent estimates, both clamped to 0.1–10× the seed:

- **Selection EWMA:** successful payload bytes over the time from first post to
  last payload CQE (old-value weight 0.01).
- **Transmit bandwidth:** completion bytes over time with posted payload debt
  outstanding (old-value weight 0.9, 10 ms updates, 50 ms window).

Failed or uncertain posts invalidate their samples; possible native debt stays
charged until completion or drain. `TransferEngine::nic_load_stats()` returns
these estimates (bytes/s), link rates, charged, posted and completed bytes for
initialized NICs without probing devices. They are local requester observations.
Selection adds no driver query or heap allocation per slice. See
[bandwidth](qualification.md#rdma-bandwidth-selection) and
[multipath](qualification.md#rdma-weighted-multipath) evidence.

## Completion and transport retry

RDMA reports a replayable READ failure only after all affected QPs and WRs
drain and their loans are released. A CQE error followed by failed retirement
retains the backing and admission. A partially posted WRITE keeps its unknown
effect even after local QP retirement. A transport switch consumes the same
physical owner’s retry budget. See [transport attempts](async-batches.md#transport-attempts).

## Request priority and bandwidth arbitration

`request.with_priority(...)` selects `High` (the default, as in TENT), `Medium`
or `Low`. TCP and NVLink accept every priority without scheduling by it. On
RDMA, the effective priority follows ordinary grants, multirail slices and
remote staging CROSS work into the worker queues.

Each RDMA worker collects ready payloads from all its contexts and posts High,
then Medium, then Low, FIFO within a tier. Requests within a batch keep their
order, posted DMA is never preempted, and completion, receive, MW and cleanup
work bypass the priority queue. A full QP does not block another ready QP.

With `RdmaConfig::deadline_bw_arbitration = true` (default `false`), candidates
sharing a NIC and priority are ordered by
`(posted_bytes + unposted_grant_bytes) / transmit_bandwidth / remaining_deadline`.
As in TENT, the first 64 picks are greedy and the tail uses a fixed backlog with
FIFO ties; unknown bandwidth keeps FIFO. This orders work; it is not a rate cap
or a deadline guarantee. OrbiTE groups by local NIC across peers because its
workers share one CQ budget, whereas TENT also groups by remote segment. Sorting
buffers are reused, with no per-dispatch allocation. See
[qualification](qualification.md#rdma-request-priority).

Admission uses the same `DeadlineMlu` formula before posting. It counts
dispatched, unfinished RDMA payload bytes and uses the sum of current transmit
estimates for healthy NICs allowed by the physical owner’s mask. An effective
link rate supplies the initial estimate. Measured transmit bandwidth can
subsequently replace it. If an eligible NIC has no rate or transmit sample,
or recovery owns the route snapshot, admission leaves the owner eligible.
The RDMA worker continues to arbitrate its own posted backlog. A pre-dispatch
drop creates no grant, connection or WR. See
[queue configuration](async-batches.md#runtime-admission-and-dispatch-window).

## Priority promotion and shared slots

`priority_promotion_timeout` defaults to 10 ms, as in TENT. Workers check
lower-priority ready payloads at most once per millisecond and promote one tier
per pass, keeping the original enqueue age. By default
(`priority_promotion_per_entry = false`, TENT's behavior) an aged head promotes
its whole tier; set it to `true` to promote only entries that individually aged
out. Aging lets old work progress against new High arrivals; it does not
guarantee deadlines or isolate tenants.

Cooperating processes on one host can share priority slots:

```rust
use orbite::transport::rdma::RdmaConfig;
use std::time::Duration;

let config = {
    let mut config = RdmaConfig::default();
    config.priority_promotion_per_entry = true;
    config.shared_quota_shm_path = Some("/orbite-rdma-slots".into());
    config.slot_rotation_interval = Duration::from_millis(2);
    config
};
```

The 2 ms slots repeat High-only, High+Medium, then all priorities, matching
TENT's `SharedSlotManager::canSend`. Only payload posts are gated. This is a
worker-pass gate, not a bandwidth cap or exclusive turn. All participants need
the same name, interval, effective UID, `/dev/shm` mount and time namespace.
An empty or absent path leaves scheduling local. With a name configured, one
publisher per RDMA worker pool updates the mapped slot from the monotonic
clock every millisecond. A nonblocking kernel lock releases when its owner
exits. Readers also check the clock boundary so a stalled publisher cannot
extend a Low slot. The fixed OrbiTE header validates the interval, clock
domain and format before mapping. A foreign or damaged object fails closed.

NIC bytes live in each process's selector. Smart selection charges the exact
slice length through retirement, including short and folded tails. Round-robin
keeps inflight selection bytes at zero. Both modes count lifetime assigned
bytes in `NicLoadStats::allocated_bytes`. A process exiting cannot return
another process's charge. Stop **all** participants before removing
`/dev/shm/<name-without-leading-slash>`. See
[qualification](qualification.md#rdma-promotion-and-shared-slots).

## Custom topology and priority matrices

`priority_matrix` accepts TENT's classic matrix. Each row is
`[preferred NIC names, available NIC names]` (rank 0, rank 1):

```rust
let rdma = {
    let mut config = orbite::transport::rdma::RdmaConfig::default();
    config.priority_matrix = Some(
        r#"{
    "cpu:0": [["mlx5_0"], ["mlx5_1"]],
    "cuda:0": [["mlx5_1"], ["mlx5_0"]]
}"#
        .into(),
    );
    config
};
```

Names must exist in the discovered inventory after filters; `cuda:N` uses the
CUDA-visible ordinal. Only listed NICs are eligible for a row, an empty rank
disables a location, and masks or strict NUMA can only narrow the candidates. An
explicit `*` row covers unlisted locations; otherwise the classic matrix
synthesizes `*` from all named NICs.

`custom_json_path` also accepts TENT's native format:

```json
{
  "nics": [{"name": "mlx5_1", "type": 0}, {"name": "mlx5_0", "type": 0}],
  "mems": [{"name": "cuda:0", "type": 1,
            "device_list": {"rank0": [0], "rank1": [1], "rank2": []}}]
}
```

Indices refer to the file's `nics` order. Memory type, when given, must match
discovery (`0` host, `1` CUDA, `4` for `*`). JSON never replaces discovered
PCI/NUMA metadata or capabilities. Both inputs are limited to 1 MiB, and invalid
explicit configuration fails initialization (TENT warns and falls back to
autodiscovery). Ranks are frozen after initialization. See
[qualification](qualification.md#custom-topology-input).

## Eager publication and cross-rail slice completion

With host Direct policy, registration prepares MRs on every usable context,
including paused ports, before the range becomes visible: one local-only MR, or
for remote access a local-only MR plus a separate `LOCAL_WRITE|MW_BIND` parent.
Neither parent key is exported. Visibility commits only after every MR is ready;
failure retires the new copies, and failed rollback retains its MR, backing and
budget. Final unregister hides the range while admitted work keeps its MR borrow
through drain. Each grant locks its exact range, so disjoint ranges proceed and
conflicting CPU or DMA access is rejected across aliases.

Slicing follows TENT: a 64 KiB base block, at most 64 slices for host-local READ
and 32 for WRITE or device-local READ, adaptive blocks for large requests and a
short tail folded into its predecessor. Each slice selects its own rail. The
source reserves the whole parent's grant and receive capacity before publishing
authority, and the parent succeeds only after every slice and its source receipt
complete. Earlier slices may already have changed bytes, so a multirail WRITE is
not atomic.

## Connections, grants and completion

Peers share a fixed-width reusable QP array per local context, remote
process/PD/NIC, role and capability; width is the smaller `num_lanes`. Each cold
builder reserves a slot before creating a QP, and waiters never hold store locks
across I/O. Concurrent grants share published QPs. If either peer stages, or the
source uses direct GPU memory, connections are per grant. Every RDMA phase binds
process and connection incarnations, the exact request and a grant generation.

For host or staged source memory, the source binds a one-use Type-2B memory
window to the exact range and direction and posts a completion receive before
publishing the grant; Type-2B support is required. The PD keeps a bounded key
history so an old key cannot regain authority, parking exhausted MW indices up
to `max_window_key_indices`.

The requester posts READ/WRITE work and sends a fenced `SEND_WITH_INV` after
all payload WRs complete. `signal_interval` defaults to 1. A larger interval
signals every Nth WR and the actual post tail, capped at half the SQ depth
(minimum one). Each possible flush still needs one CQ slot per outstanding WR.
`RdmaStats::payload_send_cqes` counts physical payload receipts, while
`completed_reads` and `completed_writes` count successful WRs. The source accepts completion only when its
RQ receives the exact 141-byte binding and invalidated key, then cleans the MW
and grant, sends ACK and publishes READY. `RdmaQp` tracks exact outstanding WR
identities, so old or duplicate completions cannot repay current debt. Only an
explicit pre-authority cache miss permits a reconnect; ambiguous writes are never
replayed. This protocol is OrbiTE's own, not TENT wire compatibility.

## Workers, CQ lanes and budgets

One transport-level `Workers` pool serves all contexts. `num_lanes` sets the data
worker count and SQ/RQ CQ pairs per context; a reused QP stays on its lane.
Workers bind to allowed NUMA CPUs. `RdmaProgressMode::Polling` is the default;
`EventDriven` arms both CQs and wakes on native events. One `Monitor` consumes
device and CQ events and scans contexts each second.

| Budget | Default | Charged resource |
| --- | --- | --- |
| `max_endpoints` | 64 | Physical QPs, including retained connections |
| `max_grants` | 64 | Whole-parent grant reservations |
| `max_send_wr` | 16 | Unresolved send WRs per QP |
| `max_cqes` | 1024 | Each CQ's capacity |
| `max_direct_bytes`, `max_staging_bytes` | 64 MiB each | Active direct loans and private staging |
| `max_cached_mrs`, `max_cached_mr_bytes` | 64 / 64 MiB | Full-view MR copies across contexts |
| `max_window_key_indices` | 65,536 | MW key history per PD |

Creating, closing and retained resources stay charged. These limits govern
admission; they are not throughput or fairness guarantees.

The HCA supplies initiator and responder READ limits for each new RC QP.
`max_rd_atomic` and `max_dest_rd_atomic` default to 16. A requested zero
stays zero. A positive request uses at most the reported limit, or one when
the HCA reports zero. Data grants authorize only the requested READ/WRITE
direction. `last_connected_qp` records the local HCA limits, atomic support
and applied depths after both QP transitions succeed.

For wildcard memory the requester prefers the configured healthy direct peer
even if the peer lists it after another NIC. Slice planning refreshes local
paths after reading the peer’s table and preserves the request’s device mask.

## Failure, drain and resource ownership

A failed post releases only a proven unaccepted suffix. An unknown `bad_wr`,
invalid CQE, timeout or CQ error keeps the work charged and fences admission;
healthy siblings still finish their own cleanup. Parent cleanup has one
30-second destruction window, separate from the data deadline, and a timeout
proves nothing. Failed native teardown keeps its resources, backing and charges
and is never retried by another close or Drop. A fully drained progress owner can
be replaced while keeping the context's PD and MRs.

`TransferEngine::close(timeout)` fences admission, waits for accepted work,
retires connections and MRs, drains CQs and workers, then destroys PDs and
contexts; a later close can keep waiting. `close_source` and `close_requester`
are directional and do not tear down PDs or contexts. `Error::Unquiesced` means
physical completion or destruction is unproven, so batch capacity, registrations
and DMA loans stay retained. See [native cleanup](native-cleanup.md) and
[async batches](async-batches.md).

## Rail health

The requester tracks errors per peer epoch and local/remote NIC pair. The
registered memory's path table, static `rail_topology` (`all` and `direct`),
the policy device mask and current health determine which NIC can carry a
slice. An empty allowed set fails before payload. Port generations from the
existing monitor fence old observations. QP retirement and grant drain still
belong to the original CQ lane.

Defaults are three failures inside ten seconds, a 30-second cooldown and a
one-second exploratory probe interval. Set `RdmaConfig::rail_error_threshold`,
`rail_error_window`, `rail_cooldown` and `rail_probe_interval` to change them.
Zero probe interval allows an immediate trial. Looking up a mask never arms a
probe. Only one admitted trial can run on a paused pair. Early failure leaves
the cooldown intact. Expiry admits one half-open trial. A failed half-open
trial doubles the cooldown up to five minutes. Only a current trial’s positive
receipt reopens the pair. Cancelling an unposted trial returns its slot.

`RdmaStats` exposes errors, open pairs, probes and recoveries. NIC load stats
include each local NIC’s rail errors, probes and open-pair count. A process
retains at most `max_rail_pairs` pair identities. An exhausted table preserves
active and unproven owners and returns `ResourceExhausted`. See the
[installed verification queue](qualification.md#verification-queue).

## Periodic monitor recovery

The monitor queries selected routes even without traffic. A changed or inactive
address fences the route; old builders, QPs and CQs must drain before a validated
route is published again. Recovery keeps the PD, key history and MRs.
`monitor_ticks`, `route_queries`, `route_recoveries` and `progress_recoveries`
count scans, queries, resumed routes and reclaimed lanes. Fatal device events and
retained cleanup failures stay fenced.

### Initially paused contexts and GID reselection

An initially inactive port with valid GID metadata keeps a paused context and
PD. Host Direct registration still prepares its MRs, while path selection
rejects that route and healthy rails stay eligible. On recovery, automatic GID
configuration (`gid_index: None`) reranks valid GIDs on the same device and port;
`Some(index)` stays pinned. This follows TENT's `RdmaContext::enable` and
`refreshAddress`; hotplug and cross-rail retry are out of scope.

The focused native gate runs on one authorized mlx5 RoCE RNIC with real
PD/MR/QP/CQ operations and an overridden port state; it changes no physical port
and is not a two-host or physical-loss test. Choose an `ORBITE_INITIAL_GID`
different from the automatically preferred index:

```sh
ORBITE_RDMA_LOCK=/path/to/authorized-rnic.lock \
ORBITE_SHARED_DEVICE=mlx5_0 ORBITE_SHARED_PORT=1 ORBITE_INITIAL_GID=0 \
CC=gcc CXX=g++ cargo +stable test --locked --features rdma \
  transport::rdma::context::tests::native_context_recovery::native_context_recovery_preserves_pd_mrs_and_strict_gid_selection \
  -- --exact --ignored --nocapture --test-threads=1 \
  > /path/to/external-artifacts/context-recovery.log 2>&1
```

See [qualification](qualification.md#focused-native-recovery).

## NVIDIA memory ownership

`CudaMemory::allocate` owns device memory and its primary-context reference.
Unsafe `CudaMemory::from_raw_parts` adopts an existing allocation after
validating type, context, ordinal and extent; host and managed pointers are
rejected, and aliases must share one `Arc<CudaMemory>`. CUDA batch registration
uses the same transactional rules as host registration.
`cuda::allocate_pinned_host` returns owned pinned `HostMemory`. The feature
loads Driver API symbols from `libcuda.so.1` at runtime and needs no nvcc,
libcudart or other CUDA libraries to build.

Callers exclude conflicting external access before handoff. GPU producers either
finish their work first or use the readiness handoff below. Failed copy
synchronization, context restore or unpin keeps the owner, visible through
`cuda::retained_resources`.

### Producer-stream readiness

Unsafe `CudaMemory::record_producer_stream(stream, owner)` captures work already
queued on the producer's stream. Call it on the producer thread before handing
the allocation to a transfer. `owner` must keep the stream and all producer
inputs alive until completion, without an ownership cycle. The stream must
belong to the allocation's native context; default-stream handles resolve on the
calling thread, and graph-capturing streams are rejected.

`cuda/readiness.rs::CallerSyncEvent` belongs to the allocation and is shared by
its aliases. The first local copy, staging loan, GPUDirect DMA loan or IPC access
waits for the recorded generation once, with `cuEventSynchronize`. That wait
blocks the CPU and cannot be interrupted by a request deadline; it is not a
device-side stream dependency. An unproven record, wait or context restore fences
the allocation and keeps its producer inputs. TENT's `CallerSyncEventPool` /
`syncWithCallerStream` waits on `cudaStreamPerThread`; OrbiTE accepts the
caller's explicit stream and carries the generation across threads and
transports. Consumers start after the engine's completion receipt; consumer
stream chaining is not exposed. See
[qualification](qualification.md#cuda-producer-readiness).

### Pinned host producer handoff

With `cuda`, unsafe `HostMemory::record_producer_stream(stream, owner)` accepts
memory from `cuda::allocate_pinned_host` under the same contract; ordinary or
external host memory is rejected. The stream must belong to the allocation's own
context. CPU read/write, staging, host RDMA and local pinned IPC access consume
readiness before touching payload, and final `cuMemFreeHost` or
`cuMemHostUnregister` waits for any pending generation. Internal staging buffers
without a producer create no event. See
[qualification](qualification.md#pinned-host-producer-readiness).

### Copy completion and reuse

Each `CudaMemory` lazily owns one nonblocking stream and completion event.
Staging copies are asynchronous, record the event after the copy and wait for it
before exposing host bytes or releasing staging. If submission or recording
fails, OrbiTE drains that stream; if drain is unproven, it returns `Unquiesced`
and keeps the allocation, staging and handles. Destroying an event or stream is
cleanup, not completion evidence.

Run the byte/guard gate only on allocated, locked GPUs:

```sh
cargo build --locked --release --features cuda --example cuda_copy
# Ordinals refer to the selected process-visible devices.
target/release/examples/cuda_copy 0 1
```

It checks three unaligned sizes with eight concurrent callers for four rounds
each. It covers local staging only.

## GPUDirect RDMA ownership

Build with `--features rdma,cuda`, set `MLX5_SCATTER_TO_CQE=0` before the process
starts, and select `RdmaConfig.cuda_memory = RdmaCudaMemory::Direct`.
Initialization rejects any other setting, because mlx5 scatter-to-CQE can copy
small READ results with the CPU, which is incompatible with device pointers
([provider guidance](https://github.com/linux-rdma/perftest#readme)). There is no
automatic fallback to staging.

Before registration, OrbiTE sets `CU_POINTER_ATTRIBUTE_SYNC_MEMOPS` and checks
GPUDirect write-ordering support. `LocalBufferManager` publishes one local-only
GPU MR per context under the owning CUDA context, and requester slices borrow
these persistent MRs. Allocation-wide `DmaLease` ranges reject overlapping copies
or grants and never expose GPU pointers as CPU slices.

A GPU source can reuse a connected QP and its local-only GPU MR when every
advertised rail can create a hardware-revocable key over that MR.
Set `RdmaConfig::gpu_source_reuse` to `true` to opt in. The default uses the
private grant path qualified on H20. Changing that default requires the
[installed reuse gate](qualification.md#verification-queue) to pass. On mlx5, OrbiTE probes the
actual parent using DevX, indirect KLM translation and named key indices. Each
grant binds a fresh key to its QPN, exact byte range and READ or WRITE right.
The source verifies the device's key context and KLM before publishing it.
Firmware that requires the wildcard QPN fails the registration probe. A full key
is never reused on that PD. Key history shares the bounded window-key budget.

The source releases the range after the matching receive-with-invalidate CQE,
bound DONE, key destruction and CUDA write visibility. Uncertain revocation,
destruction or visibility retains the owners and admission. With reuse
disabled or an unsupported parent/provider, a grant has a private PD, exact MR
and QP. Its ordinary fenced SEND is followed by QP/MR/PD teardown before ACK.
Both paths avoid binding ordinary memory windows to GPU MRs.
[Provider memory window limits](https://docs.nvidia.com/networking/display/mlnxofedv23102131201lts/optimized%2Bmemory%2Baccess),
[mlx5 programming reference](https://network.nvidia.com/files/doc-2020/ethernet-adapters-programming-manual.pdf).

The `hardware_peer` gate accepts `ORBITE_RDMA_CUDA_MEMORY=direct` and optional
`ORBITE_RDMA_SLICE_BYTES` (1, 17, 32 or 64). Its passive observer records MR
flags, QP PDs and payload WR addresses, which `gdr.py` checks against the
declared allocations and full buffers:

```sh
CC=gcc CXX=g++ python3 tests/hardware/build_installed.py --consumer gpu \
  --toolchain stable --output /external/installed-gpu-stable
python3 -B tests/hardware/run.py --profile /external/gpu-profile.json \
  --binary /external/installed-gpu-stable/installed/bin/orbite-hardware-peer \
  --installation /external/installed-gpu-stable/manifest.json \
  --gdr-observer /external/installed-gpu-stable/liborbite_cq_fault.so \
  --output /external/runs/gpu-direct-001
```

Use `--toolchain 1.85.0` and a new output directory for the MSRV build. Each case
declares `cuda_memory: "direct"`, `transport: "rdma"` and `cpu`/`cuda:0` memory
kinds with at least one GPU. Small cases use `payload: 4096`; the two-rail
folded-tail case adds `rails`, both NIC locks, `payload: 8388768`,
`rdma_block_size: 1048576` and `expected_grants: 12`. Run both directions and
both `rdma_events` values, and keep failed runs. Set `gpu_source_reuse: true`
on GPU-source cases to require key and QP reuse. The verifier matches each
payload rkey to its source receive CQE on the parent PD and checks both peers'
full buffers and guards. See the [verification queue](qualification.md#verification-queue).

## Run a native profile

Collect fresh host, provider and device inventory first, then pass an external
JSON profile to [`tests/hardware/run.py`](../tests/hardware/run.py):

```sh
python3 tests/hardware/run.py --profile /external/profile.json \
  --binary /external/build/release/examples/hardware_peer \
  --output /external/runs/run-001
```

A profile contains `ssh_config`, `rnic`, `gid_index`, `nodes` and `cases`. Each
node has an SSH `alias`, exact `hostname` and `boot_id`, a `bind` address and its
`nic_lock`; GPU cases add `gpu_lock` and `gpu_uuid`, and optional `env` freezes
library paths. Each case sets `source`, `requester`, `source_kind`,
`requester_kind` (`cpu`, `pinned:0` or `cuda:0`), `transport`, `payload` and
`seed`, with optional `host_memory` overrides. Multirail and installed-host modes
need two physical `rails` and a `nic_locks` map; see the
[installed host recipe](installed-host.md) and
[native cleanup recipe](native-cleanup.md).

The controller checks host identity and binary hashes, deploys to a unique
`/var/tmp/orbite-runs` directory and records provider mappings. Supervisors hold
resource locks until `waitpid` confirms their children exited; a missed reap or
missing receipt fails cleanup. The runner never changes NIC configuration. Keep
runs and failed controls outside the checkout.

## Reference

The frozen TENT reference is `e1ea5d31bb5e46ff2df9f90e7c2fbaf52fcdee8c`; see the
[owner mapping](architecture.md#rdma-resource-owners). Exact-range MW
authorization, source receipts and failed-cleanup retention are OrbiTE
contracts. The
[pre-cleanup guide](https://github.com/OrbitInfra/OrbiTE/blob/9e1fdeaa9f7cd6c03e30aab2dff19a00f0a44578/docs/rdma-cuda.md)
keeps the earlier stage-specific recipes and evidence locations.