Architecture#
OrbiTE follows frozen Mooncake TENT’s engine, transport, context, registered
buffer, endpoint and worker boundaries, implemented as Rust resource owners.
The consumer scope defines
the consumer and non-goal boundary. The NIXL backend design
proposes an upstream C++ plugin over a purpose-built native boundary to these
owners, with no second coordinator. It follows upstream’s explicit-cancel
direction; terminal status requires physical retirement. Permanent quarantine
and borrowed-allocation lifetime still need upstream agreement. There is no
plugin implementation or qualification yet.
The current work order defines the plugin
prerequisites. Installed GPU/RDMA results on the current artifact precede the
formal backend RFC; this local design review adds no qualification.
The beta.1 Rust surface review and
Python surface define the API freeze
boundary. Public enums and field records are non-exhaustive except binary
TransferOp; configs use defaults and field assignment. Documented primary
helpers are stable candidates. Hidden owner helpers are release-private, so the
Python packaging crate pins the exact core version. The hidden adaptive core
has no stability promise. Remaining missing docs close before 0.1.0; source
readiness does not qualify a native device.
API, configuration and wire compatibility are out of scope. Provenance lives
in the owner and settings map; public
setting docs describe OrbiTE’s units, defaults, bounds and ownership effects.
Architectural alignment alone establishes no behavior or performance result. See capabilities for support and the
completion plan for open work.
Scope and non-goals#
OrbiTE is a transfer engine for LLM inference on NVIDIA.
Non-goals:
Cluster discovery and metadata, which belong to the consumer.
Collectives.
Non-NVIDIA accelerators.
TENT wire interoperability.
General-purpose RPC.
Python boundary#
The separate orbite-python crate builds only orbite._native. This is a real
wheel-packaging boundary: it keeps PyO3 out of the primary Rust crate and owns
the existing bounded memory, segment and batch tables directly. The removed
C/C++ exports, raw argument/output adapters and external plugin loader have no
Python callers. No Python dependency or interpreter is loaded by an ordinary
Rust build. See the consumer boundary. Shared PyO3 has a
compatible minimum of 0.27.2 and the lockfile fixes its MSRV-tested graph. Maturin
builds one wheel per CPython version because the 3.10 stable ABI lacks the buffer
API needed to retain exporters. Mooncake’s mooncake.tent and standalone tent
imports remain untouched; consumers explicitly import orbite.
Python argument conversion captures a writable, contiguous memoryview or CUDA Array Interface/DLPack exporter. The CUDA path verifies the full export against driver allocation bounds and device attributes, consumes producer readiness in the exact allocation context, and retains each Python/capsule lease through aliases, worker loans, unregister and native quarantine. The capsule follows the published DLPack C layout without a new dependency; CPU wheels never request GPU capsules. External integer addresses without an exporter are rejected before dereference. The installed negative controls reject bare pointers, resize while registered and out-of-exporter ranges; a held TCP reply keeps the actual exporter alive after timeout, cancel and early guard drop. A memory guard whose allocation is registered defers free until unregister drains it, or engine close drains the table. A batch guard moves its receipt to the existing deferred retirement owner. Neither guard destruction waits for an unbounded transfer.
Every native call detaches from the GIL and catches unwinding. Wait uses the
existing bounded backoff, dropping the per-batch mutex between observations;
physical drain is required independently of task state. A Python thread releases
a held real TCP reply while another is blocked in wait. No Python objects enter
the payload worker other than the retained exporter owner. Decrementing that
owner uses PyO3’s deferred reference pool; it never acquires the GIL beneath a
native lock. A cleanup-result fault keeps a real Python exporter and its closed
engine slot. Installed-wheel CUDA controls drive native range checks, exact context
restoration, a blocked copy and failed pinned cleanup through a fallible driver
shim. Export leases are bounded by the registration budget, including duplicates.
trim_idle_staging_buffers reclaims only idle ProxyManager slots, with a bounded
single pass and one-shot cleanup; active slots are untouched. The Python adapter
uses it together with CudaMemory::trim_idle_copy_resources before releasing
foreign registrations, avoiding an idle context cache pinning a Python tensor
indefinitely. Checked event/stream destruction reports Unquiesced before Drop;
failed cleanup keeps the registration, slot charge and exporter (test_failed_staging_cleanup_keeps_exporter_and_never_retries).
This retained interchange contract is an extension beyond TENT’s raw integers.
If interpreter finalization prevents a safe detach, the bounded
engine slot remains retained. The guide records these deliberate
lifetime differences from frozen src/python/pybind.cpp and its guard test.
Peer retirement#
runtime/segment_manager.rs owns a bounded table of endpoint/epoch identities.
Each prepared memory submission retains a peer permit in its existing batch
admission owner, through queued work, retries, aliases and quarantine. Notification
hooks retain every target until their final attempt. Closing a peer fences the
entry before waiting; native cleanup runs outside table and peer locks. A fresh
epoch gets a different entry. Each requester-owned RemoteSegment and clone
holds an identity token; permits and cleanup hold the same entry. Under the table
lock, a sweep reclaims only entries with no outside references, either still open
or successfully cleaned up. Admission requires that token and never inserts by
key. An explicit open_segment can recreate a reclaimed identity only after
checking the authoritative epoch through Describe. Another requester must open
its own descriptor. This closes the borrowed-snapshot resurrection hole without
charging historical peer incarnations. There is no engine descriptor cache to
erase: snapshots lose admission authority immediately.
Timed-out drain entries still owe cleanup, and failed native cleanup entries remain charged even after every handle disappears. Live/retired/unquiesced and cumulative reclaimed-entry counters expose this distinction. The churn control moves guarded TCP bytes through ten times the table capacity; pressure, last-clone, foreign-requester, cleanup-held and concurrent reclaim/open/admit/close controls check that only an explicit authoritative open can recreate admission.
After drain, each transport removes only matching requester resources. TCP leases an epoch-keyed cached stream to the drained call; retirement then closes that stream. HP-TCP caches key lanes by control endpoint and epoch. SHM imports include that pair in their mapping key. RDMA fences its matching endpoint arrays and uses the existing CQ retirement receipt; registration MRs, contexts and workers remain shared. NVLink and MNNVL close matching relocation entries through their existing one-shot native cleanup. Failed cleanup is sticky and retains its actual native owners and charges. Global requester close waits for selective cleanup and preserves its error. The existing fail-stop RDMA fault domain still applies if a provider teardown error quarantines a shared CQ; retirement does not weaken that safety boundary.
Frozen segment_manager.cpp::closeRemote, segment_tracker.cpp and
transfer_engine_impl.cpp::closeSegment invalidate handles/descriptors without
this reference-retained membership fence or cross-transport drain. Rust retirement is a
stronger extension. Python preserves reversible TENT handle close. peer_retirement_* controls distinguish
retirement from timeout, exercise submit/close races, keep a held TCP reply and
CUDA event charged, reject stale snapshots after a same-endpoint restart, and
prove native destroy is attempted once. The installed-wheel reopen control distinguishes handle close from peer retirement.
Configuration ownership#
The wire and consumer boundary separates the design reference from OrbiTE’s own public interfaces. Consumers publish endpoint and epoch; no metastore types, URI schemes or credentials are configuration inputs.
EngineConfig::from_json, from_file and from_env resolve one bounded,
strict document into typed owners. Only from_env reads ORBITE_CONF.
Explicit input takes precedence by choosing that entry point; there are no
field-level environment overrides. Both typed and JSON constructors capture
custom RDMA topology files before engine construction; hardware discovery
validates physical capabilities when the owner starts. No process environment
is changed. config::json::tests rejects compatibility keys, unknown fields,
wrong types and duplicate keys, and uses children to prove snapshot isolation.
The Python crate retains the same native memory and batch owners behind
Memory, RegisteredMemory, RemoteSegment, FileSegment and Batch.
Blocking calls detach the GIL inside the existing panic barrier. Native errors
retain their Rust variants instead of collapsing to TENT integer status codes.
The installed-wheel held-reply and CUDA-export tests check early drop, timeout,
cancel and sticky cleanup; these interface changes supply no new release receipt.
Metrics use orbite_ names with the same recording cells and histogram buckets.
Engine and transport boundary#
The CPU oracle and three frozen TENT RPC request prefixes remain test inputs.
Compatibility inventories and unused serializer captures are removed. The Rust
control plane retains its own codec and in-source fixtures.
The wire and consumer boundary
excludes S23. The native acceptor has one listener and no profile dispatch.
It rejects a foreign magic prefix before waiting for the rest of the header;
frozen_rpc_prefix_is_rejected_before_a_native_header_or_payload_is_read checks
that boundary, unchanged guards and continued native service. The frozen oracle
also runs frozen_tent_client_is_rejected_without_admission_or_payload with
actual TENT descriptor, READ and WRITE clients. The native handler reports
Error::Protocol before source admission. YLT remains test-only.
Callers build a TransferRequest with a transport (or Transport::Unspec) and
two absolute addresses; the operation is chosen at submission, and an optional
engine-resolved QoS handle sets priority and caps. Public request, transport and
error types are non-exhaustive.
TransferEngine owns registration, batches, admission and directional shutdown.
RegisteredMemory retains the actual backing, and aliases share allocation-level
access coordination. runtime::control_plane::ControlService owns peer lookup,
request framing and source handlers, and dispatches payload work to the sibling
TCP, RDMA and NVLink transports. RDMA negotiates paths, connections and exact
memory grants over the control connection and moves bytes through verbs; see
wire boundaries.
Ordinary TCP follows HP-TCP’s exclusive clean/dirty stream ownership, using the
existing caller/batch and control workers rather than adding a coordinator.
Both requester and source idle caches charge a bounded max_peer_segments
slot per entry. Direct requests block on deadline-bounded socket readiness;
staged requests keep bounded progress. Only complete frames permit reuse, and
the native frame layout is unchanged. This is deliberately stricter than
TENT’s RPC retry pool: no inner replay, no cross-epoch reuse, and failed shutdown
is sticky Unquiesced. transport::tcp::pool::tests covers epoch separation,
borrowed-capacity pressure, idle eviction, EOF before posting and one-shot failed
cleanup. runtime::control_plane::tests::tcp_reuse checks ordered scattered
bytes, dirty-frame retirement without replay and idle source admission. The
held-reply peer-retirement control proves a timeout cannot release backing or
close the active stream before the drain receipt.
HP-TCP is built into the CPU transport set and is disabled by default. When enabled, it replaces the ordinary TCP payload and retains the control service. Each configured worker owns a single-thread Tokio reactor, the Rust counterpart to TENT’s per-worker Asio context. Persistent lanes keep their worker affinity and use readiness-driven sockets with TCP_NODELAY. Physical owners retain registration and task/byte admission through the last slice receipt. The typed configuration, frame layout, static rail limits and generation fence are described in HP-TCP. Shutdown controls observe retired reactor tasks and the original accepted sockets; a forced port rebind proves that a later connection cannot identify the old listener. TCP staging teardown similarly waits for its own acceptor-join receipt before checking that an abandoned native delegate still retains its pin.
SHM follows the shared allocator and synchronous copy in frozen TENT. TENT
can also copy a caller-supplied path from MemoryOptions. OrbiTE publishes
only allocations whose mapping and fd it owns, so an ordinary host buffer
cannot grant access to an unrelated inode. OrbiTE also publishes the inode,
allocation offset and registration grant. The source holds Linux OFD locks for
the permitted directions, and every requester holds a separate activity lock
through its copy. Those receipts fence a cached mapping across unregister or
process death. Local/native loans and peer copies coordinate on inode byte
ranges. Mapping failure precedes payload I/O, and failed unmap quarantines the
mapping and its charge. POSIX and directory-backed paths use the same owners.
See shared memory.
Native verbs loading#
transport/rdma/sys/native.rs resolves the host’s libibverbs.so.1 on RDMA
install, before any native owner exists. A process-lifetime table retains the
library through every context and cleanup callback. Private header-generated
layouts and the port inline bridge carry no link-time verbs dependency. The
existing fallible Verbs table still owns operation injection, and post/poll
still use provider context operations. This follows frozen TENT’s
transport/rdma/ibv_loader.cpp. Unlike its static destructor, OrbiTE never unloads
a successfully installed table, so retained cleanup owners cannot outlive its
code. optional_exports_and_loaded_code_follow_the_retained_library checks that
the loaded code remains callable after its file is removed. This changes no
transfer, registration or retirement contract.
missing_library_and_each_required_export_are_unavailable_before_a_native_call
rejects each incomplete table, and the clean-container gate runs the full
feature suite and an installed wheel without any verbs/CUDA/cuFile runtime.
Its ELF negative control rejects a library linked to libibverbs. Explicit GDS
selection preserves the lazy loader’s error; automatic selection still skips
an unavailable provider.
Local file segment ownership#
TENT’s SegmentManager maps a name to one handle, makeFileRemote describes one
regular file with base zero, and io_uring passes the request offset straight to
positioned I/O. segment/file.rs follows that split: a bounded per-engine table
shares one entry per name, the size is cached for TENT’s one hour, and refresh and
range admission read the size from the open fd, so replacing the pathname cannot
change the file an admitted operation uses. A lease keeps the fd until its I/O
finishes; close fences all aliases before waiting, and a timeout keeps the entry
charged until close is retried. See file segments.
An idle engine retains no storage thread or ring after the setup probe.
For DRAM/file payloads the caller or existing batch worker owns the io_uring,
iovecs, host range loans and bounce storage through CQE drain. File and peer
targets share admission and batch progress. One small contiguous host-file
request can submit on an idle batch’s caller, peek once, then block for a CQE
for up to 200 µs, capped by its request deadline. At most three EINTR/EAGAIN
attempts share the same window. Unlike TENT’s nonblocking caller status polling,
this brief wait can finish a late CQE without a thread handoff. A timeout or
enter error transfers the unchanged ring to the batch worker.
late_small_cqe_waits_on_caller_without_a_worker_on_tmpfs_and_direct_ext4 holds
a real CQE past the initial peek and counts zero worker starts.
caller_wait_interruptions_errors_and_short_cqe_never_resubmit checks the shared
window, bounded retries and exact short prefix;
expired_caller_window_drop_and_engine_close_keep_the_drain_owner checks that
expiry, drop and close leave charges and loans retained until the actual CQE.
Queued, background-progress, fallback and CUDA paths keep
their original worker. pending_small_file_handoff_keeps_loans_through_cancel_timeout_and_close
holds a real CQE across cancellation, timeout, unregister and close;
failed_worker_handoff_quarantines_the_accepted_sqe_without_replay proves that
a failed handoff retains the ring and admission instead of replaying a WRITE.
worker_exit_retains_a_ready_file_handoff_and_publishes_unquiesced forces
early return and unwind before the worker takes a submitted ready job. Unlike
frozen TENT’s unbounded in-flight wait at sub-batch free, this lost drain owner
is quarantined with sticky Unquiesced, including its admission and bounce
charges; the native ManuallyDrop owner alone is not an accounting receipt.
gds_non_file_group_keeps_common_retirement_and_metrics checks that the GDS
target guard skips only file-specific lease cleanup, never common finalization.
The plain ring keeps default completion interrupts;
it does not defer completion work to the submitting thread’s next ring call.
pending_native_io_completes_on_worker_while_caller_only_computes forces
asynchronous kernel I/O and checks reaping on the worker without caller polling.
The same preflight, partial-submit drain and prefix aggregation serve both paths.
Aligned gapped and contiguous host batches keep one ring for the whole group,
without bounce allocation, as checked by
aligned_file_batches_use_one_ring_without_bounce_for_gaps_or_contiguous_ranges.
The block-layer diagnosis
finds a verification/load-dependent comparison; it changes no ring, submission
or drain owner. The long tail is reproducible with unchanged TENT by altering
only its untimed verifier. Retention and short-prefix controls remain intact.
CUDA/file submissions use the same
worker and ring for successive rounds. The optional file pool lends one
aligned slot per eligible request. A miss uses the charged pageable or proxy
fallback. Unlike TENT’s eager allocation, the slab is created on first use.
r1c_aligned_slots_and_pageable_fallback_are_bounded_before_ring_creation
checks capacity and preflight rollback. A mapped-CQ poll can recover a
failed wait, while the destruction deadline still quarantines missing receipts.
The whole GPU range stays loaned through file receipts and CUDA copy events.
A file vector has one file lease, admission owner and logical receipt. Overlapping
spans share an exclusive loan within their registration. The issuer retains
the original fragment order for kernel iovecs, staged copies and cuFile cookies.
Explicit file padding extends TENT’s contiguous buffer contract. Host vectors
lease private charged scratch, and CUDA vectors use their pinned staging chunks.
Each lease stays exclusive until its completion receipt, so a READ that dirties
scratch cannot race a WRITE that needs zeros. Padding makes cuFile ineligible.
GDS discovery and CUDA buffer registration start at first eligible file use.
GDS binds a cuFile handle to the retained file entry and pins the CUDA
registration across queued and running batches. Its caller owns status polling
and quarantine. See the file guide for completion and retention.
Benchmark and fault boundaries#
The S26 harness uses public engine calls; it adds no runtime fast path or benchmark API. The frozen TENT build remains an isolated oracle and performance baseline. Measurements require full-byte and guard checks; logical status alone is not evidence. Fault fixtures reuse the executing owners and record outstanding charges before the release receipt. A killed socket requester, an injected CQ error and physical device loss are distinct cases. GPU/RDMA/GDS installed fixtures stay unqualified until run on their hardware.
Transport selector ownership#
runtime/transport_selector.rs owns compiled first-match selection and
transport_selector/config.rs compiles bounded typed or JSON policies.
segment/metadata.rs owns the placement and provider metadata published in
Describe, plus the process identity shared with CUDA IPC. TransferEngine::prepare
resolves the rule index, transport, NIC mask and QP range before admission.
Every network request, local file operation and staged CROSS child carries that
decision. RDMA filters paths before allocating slices. Its endpoint owns QP
indices, native SL/TC and cache claims independently of CQ worker placement.
The pool layout is immutable across admitted work. See
transport selection.
Library diagnostics use the shared log facade. OrbiTE installs no logger.
Consumers can enable selector warnings and errors through their own logger.
Batch progress and notification ownership#
The batch executor runs each admitted transport to its receipt. Submitting
work and completing an execution signal the same per-batch state that
progress_batch advances. TENT uses progress-lock shards. OrbiTE uses a lock
per live batch and a bounded weak-ID registry for admission and deferred
retirement. The optional engine worker owns a bounded wake set and joins on
drop. The typed progress_fallback_interval corresponds to TENT’s
runtime_queue/progress_fallback_interval_us, and notification_rpc_fallback
corresponds to notification/rpc_fallback. A sweep probes retired owners independently and skips permanently
quarantined IDs. A source staging session continues to own its chunk and CUDA
loan after its socket disappears. Only a confirmed event and restored context
return the chunk. Incoming staging uses its own bounded source workers so a
blocked Delegate leaves Describe and credit traffic serviceable. A retained
session also keeps its admission slot. The held-Delegate and distinct-location
Pin controls cover these boundaries.
Source staging, batch progress.
Notification routing visits installed channels before the control RPC. TCP and self share the bounded control inbox. An async hook retains a charged set of outstanding destinations. OrbiTE omits RDMA from notification routing because its data grant QPs have no notification receive owner. An RPC-only engine can send and receive even without an installed TCP notification channel. A lost receipt precludes automatic replay. See notifications.
Runtime queue ownership#
Owner |
Responsibility |
TENT counterpart |
|---|---|---|
|
Ordered eligible owners, per-batch dependency chains, dispatch counters |
|
|
Compiled global/tenant/intent contracts and fixed per-contract counters |
|
|
Cumulative grant/consumed accounting per peer binding |
|
|
Fixed per-sender windows, tombstones and single-flight cumulative sync |
No counterpart |
runtime/request_merge.rs maps original task IDs to physical owners after
registration, permission and route resolution. TENT sorts by hint and address.
OrbiTE also fences overlapping destinations and self-copy dependencies to
preserve public byte order. When either endpoint is CUDA, a change in the
selected transport, per-request remote staging or local/remote backing types
starts a new merge partition. This prevents a later direct alias from writing
through a pending stage or consuming its dispatch window. Disjoint host-only
hints can still merge across intervening requests. One source epoch, target and operation apply to the
whole submit. The peer advertises its transfer limit in Describe so a merge
cannot create an oversized grant on a smaller receiver.
Each physical owner holds its QoS permit, queue window and receiver credit. Dispatch takes queue → QoS → credit locks. Credit sync and payload I/O run outside them. EDF ranks independent batch heads. Reserving the only slot for a successor would deadlock its FIFO worker. The worker advances its eligible prefix while independent batches share the window. Files have no TENT memory-buffer key. The direct file path keeps its native batches. When the runtime queue is enabled, each file owner enters separately, as in TENT’s queued dispatch.
The source describes one exact location per registered view. TENT’s
coalesceRegions buckets multiple NUMA probe entries within one buffer.
OrbiTE has no such per-page location list. Adjacent registered views keep
their separate IDs, permissions and lifetimes. The queue reserves capacity
for a real proxy session, whose private CROSS tasks already have a bounded
lifetime inside that owner. This avoids reacquiring the parent’s window from
its worker. See admission.
Retry ownership#
The compiled selection fixes candidate order before merge. The existing batch worker keeps a private, bounded decision slot for a drained completion. Poll, explicit progress, cancel and free arbitrate that slot under the batch lock. The same worker starts the next attempt. Public terminal states have no path back to pending.
Frozen TENT permits duplicate same-content WRITEs after partial enqueue. OrbiTE requires proof that a WRITE posted nothing. It has no general remote revocation receipt for replay. A lost TCP ACK or a failed native WRITE can therefore make the effect unknown. A drained READ can replay after QP retirement or a confirmed CUDA copy and stage release. An unquiesced destination, an overlap with the source, a short file result or an expired deadline prevents that replay. The controls hold a real posting call, fail teardown, drop a WRITE ACK and check the untouched guards. The file owners keep confirmed siblings. A Describe RPC alone retries a connection error once, with the original deadline and epoch. A replaced registration cannot inherit an admitted WRITE.
A disconnect can leave a source receive posted while the next grant has already
passed admission. Retiring that CQ lane returns Unavailable to both earlier
and later constructors, rather than misclassifying normal retirement as an
internal invariant failure that prevents policy fallback. The native QP/CQ
barrier still owns every pending receive, backing and charge. The
receive_debt_retirement_rejects_new_grants_until_the_native_receipt control
holds destruction and probes this interleaving; the real-control-socket
source_receive_retirement_keeps_owners_and_allows_unposted_policy_fallback
control checks merged aliases, one fallback, bytes and guards while the old
source owner remains charged. Any SQ or pending-work debt remains fail-stop even with a posted receive, as
mixed_send_receive_debt_keeps_unposted_constructors_fail_stop verifies.
Internal invariant failures remain non-retryable;
this is OrbiTE’s drain-based admission behavior, not TENT’s ambiguous replay.
Metrics ownership#
The optional per-engine registry preallocates the frozen transport, operation
and reason domains, including the failover pairs. Recording uses indexed
relaxed atomics. The physical admission owner carries timestamps and the first
failure stage across retries. The legacy transfer counters describe physical
owners. Lossless request counters include every accepted public alias. The
metrics_attribute_each_retry_and_one_merged_outcome_to_the_real_transport
control checks two aliases, a failed RDMA attempt and one successful TCP
attempt. Polling again leaves the totals unchanged. Counts have no authority to
release backing or admission. Registration gauges change at publication and final unregistration.
The exporter samples existing queue, QoS, receiver-credit and rail owners only
on a scrape. Benchmark-only QoS reports have no runtime series.
Frozen TENT uses process-wide ylt metrics. OrbiTE engines own separate registries and use a bounded standard-library listener, so no C++ runtime or HTTP dependency is needed. Each worker serves one connection with a 4 KiB request-header limit and an absolute 500 ms I/O deadline. Dropping the engine joins the HTTP workers and the optional periodic logger. A busy port degrades to log-only recording. The loader rejects malformed values and caps the pool at 64 threads. Tests exercise a busy port, incomplete and oversized headers, per-engine isolation and toggle-safe gauges. A rejected CROSS and a skipped suffix close recorded outcomes without releasing transfer leases. The CPU-only TCP requester control checks both explicit peer capabilities and guarded device bytes. A GPU Direct disable switch conflicts with an explicit Direct request, preserving the registration policy. Invalid boolean spellings preserve typed settings, and debug affinity logs omit memory addresses and keys. See metrics and RDMA configuration.
The offline diagnostics module and installed orbite binary share compile-time
build identity with the exporter. The CLI uses the standard argument iterator
and serde_json; it never starts an engine or accesses the network. Only version
text/JSON rendering and the exit-code table remain. Neither the CLI nor Python
uses arbitrary-document rendering or redaction, so those APIs, the C diagnostics
header and the regex-lite dependency are removed. The packaged CLI control
checks its identity and verifies the absence of network/thread syscalls.
RDMA resource owners#
CUDA registration adds a deliberate extension to frozen TENT’s XPU dma-buf
exporter in rdma/context.cpp; TENT’s NVIDIA availability check uses peermem.
OrbiTE’s cuda/rdma.rs resolves the immutable route policy and exports the
allocation while restoring the exact caller context. All shared/private-PD and
GPU reuse paths pass the same sys/memory.rs registration boundary. No export
or registration error can fall through to pointer registration. The
dmabuf_export_or_registration_error_never_calls_the_pointer_registration
negative control includes an error after an fd was returned. Host registration
is unchanged. The route guide
records NVIDIA/NCCL alignment, default mapping and fd lifetime contracts.
Export flags remain zero without NCCL’s C2C/PCI topology evidence;
dmabuf_pcie_flag_and_optional_cuda_symbol_follow_the_driver_contract checks
older and newer drivers. Peermem detects the same three version paths as NCCL;
dmabuf_peermem_version_paths_select_each_module_and_reject_directories_only
drives registration through each path and rejects directory-only fixtures.
dmabuf_unregister_and_close_timeout_keep_fds_until_the_real_invalidation
holds real control traffic at the verbs shim, checks private and reused GPU
parents, and proves timeout/unregister cannot release a live grant.
dmabuf_failed_deregister_or_fd_close_keeps_backing_context_and_charges_without_retry
keeps native cleanup failures charged after engine drop. Retained fd count is
bounded by existing cached-MR and active-grant budgets, not a new pool.
A bounded rail table is shared by requester workers and the existing port
monitor. TENT keeps one table per worker. OrbiTE serializes only pair
accounting so two workers cannot arm concurrent trials. A receipt identifies
the admitted generation and trial. A late completion from before a port
fault cannot close the new circuit. an_open_pair_admits_one_probe_and_only_its_receipt_can_reopen_it
exercises that distinction. Discovery keeps peer epoch and NIC/PD identities,
rather than borrowing remote topology pointers. The grant path advertises the
remote memory owner’s eligible NICs directly. A multi-slice submission uses
healthy pairs, or one trial grant when every permitted pair is paused.
The endpoint store fences admission before context shutdown. A record is reclaimable after both native QP close and terminal grant retirement. An external handle cannot construct into a fenced array. A finished logical grant can still point to a live physical QP shared by other grants. Candidates check the physical fence and claim the exact generation under the lane lock. The bounded store retains a live or uncertain entry under pressure.
Owner |
Resource and lifetime |
Frozen TENT correspondence |
|---|---|---|
|
Context slots, registrations, workers and parent dispatch |
|
|
Stable NIC IDs, actual context/PD, route and persistent key domain |
|
|
Persistent host MRs keyed by actual context, allocation view and capability |
Context-keyed |
|
Peer identity, fixed-width construction slots and persistent QPs across CQ lanes |
Context endpoint store, |
|
One QP shared by independently retained grant records |
Endpoint QPs and affiliated slice ownership |
|
Physical send depth, exact outstanding WR IDs and fenced admission |
Per-QP |
|
Receive buffers/MRs, RQ debt and binding/key dispatch |
QP-owned receive WRs; exact receipts are OrbiTE’s |
|
One exact-range DMA loan, source MW, SEND storage and grant charge |
OrbiTE authority extension over registered regions |
|
Per-grant geometry, WR windows and completion identities |
|
|
SQ/RQ CQ pair, grant records, submission, drain and recovery |
Per-context CQs consumed by workers |
|
Data threads, lane assignment, priority promotion and urgency ordering |
|
|
Optional cross-process priority slots on existing workers |
|
|
Event consumption and periodic context reclaim and recovery |
|
|
Compiled NIC eligibility, grant byte charges, bandwidth-weighted selection and estimates |
TENT’s classes of the same names |
|
Aggregate SQ/RQ wakes and whole-parent grant/receive admission |
Bounded slice admission; receive credits are OrbiTE’s |
PostQueue orders the ready records collected in a worker pass. A registration
attached after its context snapshot enters a later pass; an earlier post cannot
be preempted. The priority fixture pauses before every context snapshot, so all
five requests contend together. Its late-registration negative control pauses
between contexts and proves why a single-lane pause cannot establish that boundary.
Each context slot owns its PD, CQ lanes and peer endpoint registry. Several grant records can share one QP, which keeps its send and receive queues across grants; peer arrays hold weak lane references to avoid ownership cycles. Native cleanup is fallible: a failed QP, MR, MW, CQ or channel teardown keeps the remaining owner, backing and charges, and a timeout never substitutes for physical drain. See native cleanup and the rail accounting gate.
Fixed-width cold construction#
EndpointStore::get_or_insert gives every caller the same RdmaEndpoint, with
min(local.num_lanes, remote.num_lanes) slots keyed by local context, remote
process/PD/NIC, role and capability; staged and device paths use per-grant
connections instead. An uncloneable Construction reserves a slot, attaches the
QP owner before native creation and survives until the pair is published.
Waiters release all locks and use their own deadlines, and a unique token stops
an old guard from releasing a replacement slot. The source publishes its pair
before GRANT, so later handshakes share it; a fresh bootstrap may retire an old
idle pair but never an active sibling. TENT’s
getOrInsert / construct / EP_HANDSHAKING follow the same lock discipline,
with an eager bidirectional QP list instead of OrbiTE’s lazy per-capability
arrays.
QP send ownership#
SendQueue admits at most max_send_wr unresolved WRs across all work on a QP,
also charging the aggregate CQ budget. Payload, MW bind and completion SEND share
it. A failed post returns only a proven unaccepted suffix; an unknown post keeps
the whole possible debt and fences admission. An ordered RC CQE repays its
tail and the consecutive unsignaled WRs in the same posted group. Each grant
has its own ID range and each actual post ends in a signaled WR. A non-flush
error preserves successful earlier WRs. A flush retires the prefix with zero
successful bytes and cancels only its own rail probe. An uncertain WRITE keeps
its original effect. The partial-post and guarded WRITE controls exercise these
boundaries. Dropping a work handle never returns SQ credit. CQ lookup resolves the
exact record before checking QPN, WR identity and operation, and native access
follows grant → QP → context lock order. GPU sources that opt into reuse borrow the published local-only MR on rails that prove exact,
revocable indirect keys. A fresh key is bound to one QP and direction. The
private PD and exact MR remain the path for other providers. Frozen TENT
publishes a registration-lifetime key. OrbiTE’s stale-key, guard-byte and
failed-destruction controls exercise its per-grant authority. See
GPU ownership.
The rdma-adaptive Cargo feature retains the hidden, unstable shared v0.3.14
congestion core outside the stability promise. A device and each route have separate byte windows. A domain-local lock
serializes reset and tick against concurrent feedback. Admission in enforce mode
rolls back the device reservation if the route defers. Only an explicit receipt
returns a permit’s charge. The moved/dropped permit control covers this
difference from C++ move assignment. The TENT worker has no call site for this
core and OrbiTE keeps the same integration boundary.
QP receive ownership#
ReceiveQueue owns registered receive slots and their admission credits. A
successful RQ CQE names the consumed buffer by WR ID, while its 141-byte binding
and invalidated rkey identify the completed grant, which may differ from the
buffer’s donor. Duplicate, stale or mismatched messages cannot complete another
receipt. RQ depth uses max_cqes, so a parent can prepare all receives even when
its grants exceed the SQ width, and receive credits stay charged until the
consumed buffer’s MR is reaped. TENT also owns WRs at the QP and dispatches CQEs
to work identities; the bound source receipt is OrbiTE’s authority extension.
Follow one transfer#
Register. Host Direct prepares local-only MRs and MW-bind parents in every usable context, including paused ones, before the range becomes visible.
Plan. TENT’s slice geometry splits the parent;
DeviceSelectorpicks a rail per slice from the owner’s location, masks, strict NUMA, bandwidth estimates and byte charges.Authorize. Both peers reserve the parent’s grants, and the source reserves all receive credits before exposing the first grant. A fresh one-use MW authorizes only the requested range, and every slice is prepared before payload submission.
Progress. Worker
ipolls laneiacross contexts; new grants pick the least-loaded lane, and reused QPs keep theirs.Complete. Payload completion, the fenced completion SEND, the exact source receive and bound DONE/ACK/READY establish grant drain; the parent succeeds only when every slice has. Ambiguous WRITEs are never replayed.
All contexts share aggregate byte, QP, grant and SQ/RQ limits. Fully drained faulted lanes can be replaced while the PD, MRs and worker pool stay alive. Workers bind to NUMA CPU sets; this does not move application pages.
The frozen RDMA quota has two scopes. The selector charges exact slice bytes
in one process. A configured POSIX shm name shares priority eligibility among
processes on the same host. OrbiTE uses a same-UID header, bounded nonblocking
file locks and an aligned atomic slot. A process killed under the lock cannot
strand its peers. A stale Low slot cannot extend eligibility past its clock
boundary. The Linux controls kill a publisher with a live device lease, check
the surviving NIC charge and reject a foreign header without modifying it.
Pool shutdown joins the publisher and consults the single unmap receipt. An
unconfirmed unmap retains the mapping and fd and returns Unquiesced.
TENT’s receiver ledger names four resources and assumes its caller has fenced the old epoch. OrbiTE checks its physical permits and their completion receipts before retiring even a manual epoch. The lost TCP WRITE acknowledgement control checks that an applied payload with no receipt cannot free credit. Manual grant updates never release automatic Pin leases. QoS uses one resolved budget per contract. The parser controls reject unexecutable actions by name at every policy layer.
Periodic monitor reclaim and port recovery#
The monitor scans contexts once per second, even without traffic, using
nonblocking inspection so a destructor waiting for an event ACK cannot stall it.
Faulted lanes and endpoint slots are reclaimed only after native drain.
Discovery keeps an initially down port as a paused context that accepts MR
registration but rejects routes. RdmaContext::refresh_address queries the
selected port and GID; paused automatic configuration reranks GIDs on the same
port, while pins stay strict. A monotonic route generation stops an old query
from clearing a newer fault, and recovery needs drained lanes, a successful query
and an unchanged generation. This follows TENT’s one-second reclaim/resume pass
and activateContext, while keeping physical destruction on the data workers.
See the recovery contract.
Source reading order#
Start at an operation, then follow the resource it borrows. Child modules extend their resource owner; submission and cleanup stay beside the resources they protect.
Read |
Location |
Responsibility |
|---|---|---|
Public request |
|
Engine, tasks, backing and file segments |
Runtime |
|
Selection, admission, QoS and credits |
Transport entry |
|
Registration, parent dispatch and source serving |
Context orchestration |
|
Initialization, selection, CQ attachment, recovery and shutdown |
Context owners |
|
Stable slots, PD/key history, discovery and NIC admission |
Peer rendezvous |
|
Fixed-width connection claims and bilateral publication |
Grant construction |
|
Admitted buffer, credit and connection ownership across MR waits |
Wire data |
|
Grant receipts and static NIC/PD path metadata |
Persistent owners |
|
MR, peer, QP and grant lifetime |
QP work ownership |
|
Physical SQ/RQ ownership and receipt dispatch |
CQ execution |
|
Lane lifecycle, admission, reuse, retirement, dispatch and polling |
Scheduling and events |
|
Data threads, priority and urgency ordering, events and CQ/channel owners |
Verbs boundary |
|
Fallible function table and native calls by resource |
Staging |
|
Pinned chunks, remote operations, rings and CUDA copy/drain |
Host and GPU |
|
Host discovery, NVIDIA copy, GPU DMA and IPC owners |
Paths after the transport entry are relative to src/transport/rdma/, except
the public, runtime, staging and host/GPU rows, which are relative to src/.
The installed fixture tests/hardware/rdma_host.rs crosses these owners through
the public API; see installed host. Private tests live under
tests/unit/, mirroring src/.
P4 source audit#
The audit compared OrbiTE f2c7e7f (PR #55) with frozen TENT
e2d6e8e0f2df113d84f9e39379f98c4b67b199ab. At that point both RDMA
implementations were about 11k physical lines (OrbiTE 38 Rust files, TENT 28
.cpp/.h files, tests excluded). Per-request MW authority, exact source
receipts, bilateral epochs and conservative retention are deliberate OrbiTE
contracts beyond TENT’s registered-region path. The audit found two gaps
against the context implementation at the audited revision, both since implemented:
Behavior |
Frozen TENT |
OrbiTE at |
Current source |
|---|---|---|---|
Initially down port with valid device/GID |
|
Discovery filtered non-active ports |
Paused context/PD kept; MR publication includes it, route admission excludes it |
Automatic GID reselection on recovery |
|
Refresh queried the existing index |
Paused refresh reranks GIDs on the same port; pins stay strict |
Both pass CPU transitions and the focused native gate, with independent review. Physical and fabric qualification stays in the P4 queue.
Correspondence and remaining differences#
The reference is frozen at e1ea5d31bb5e46ff2df9f90e7c2fbaf52fcdee8c; see
TENT’s transport header, endpoint and workers.
Area |
Current OrbiTE |
Difference from TENT |
|---|---|---|
QP arrays |
Fixed-width, lazily populated, concurrent grants, drain-aware rollback |
TENT builds its QP list eagerly and shares it across directions |
Monitor |
Unified get/ACK, periodic reclaim, paused contexts, GID reselection, rate refresh |
OrbiTE keeps retained-resource quarantine |
Path metadata |
|
TENT |
Grant authority |
Exact-range host MW or private GPU PD/MR, with a bound source receipt |
TENT publishes native MR keys; not wire parity |
CQ layout |
Separate SQ and RQ CQ per lane |
TENT uses one lane CQ for data QPs |
Worker load |
Unfinished grant charge, including retained native work |
Failover and remaining deadline policy are P6 |
CUDA copy stream ownership#
The caller-stream local copy engine is an OrbiTE extension beyond TENT. It
takes the DMA compiler and grid-strided copy from OrbitKV. The caller supplies
layout and allocator leases. OrbiTE resolves physical registration identities,
bounds each flight, and records an event after its final enqueue, including
partial failures. An event receipt retires loans only after exact context
restoration. Separate pinned and device descriptor slots let outstanding kernel
launches retain their own upload. pending_events_own_distinct_kernel_slots_payloads_and_abandoned_work
and query_destroy_record_or_restore_failure_quarantines_once_and_keeps_owners
exercise those boundaries. Caller-stream API
The local-copy policy is compiled at bind from typed settings. Direction,
intent and priority identify a bounded diagnostic class; priority stays
advisory on the caller-owned stream. Background D2H has a separate higher
Auto threshold. An optional total CTA cap restricts the unchanged
grid-strided kernel without changing the event receipt or retaining fewer
loans. auto_selection_matrix_keeps_stream_order_and_reports_each_submission_class
and capped_grids_cover_every_descriptor_vector_and_byte_tail_both_directions
cover this OrbiTE extension, including ordered overlap, missing images and
grids smaller than the descriptor count. Invalid caps fail before descriptor
allocation. Calibration remains an unrun installed gate;
no SM isolation or decode-latency claim follows from a shim.
Owner |
Responsibility |
|---|---|
|
Allocation lock and primary-context reference |
|
Lazy nonblocking stream, reusable completion event and pending-work state |
|
Bounded context-matched streams shared by NVLink groups |
|
Fallibly loaded Driver API functions |
|
Producer event generation, its context and retained producer inputs |
|
Pinned host allocation, context and lazily created readiness event |
|
Shared staging chunks, remote sessions, rings and worker shards |
The Driver API guard saves and verifies the caller’s exact context, including a null binding. Copy and final pinned-buffer cleanup errors reach the caller. An uncertain restore poisons the owner even if the visible binding can be repaired. The same-device custom-context and no-context controls inject failures at save, switch, copy and restore. Host and CUDA allocations dispatch by owner, so a handled CUDA failure cannot fall through to host memcpy. External device-plugin callbacks are out of scope; built-in copies retain these context and completion controls.
Each asynchronous copy holds its staging until the recorded event completes. A failed enqueue or record requires a proven stream drain; otherwise the allocation, staging, stream/event and context are retained. Destroying a stream or event never proves DMA completion. This follows TENT’s CUDAStreamPool handles and NVLink’s event-after-copy completion. The staging owners and their progress states are described in CUDA staging, and the readiness handoff in producer readiness.
Platform pointer and location ownership#
memory/location.rs defines host and CUDA memory types and bounded RangeLocation
metadata supplied by the actual owners; CUDA-pinned host memory stays CPU memory.
cuda/probe.rs validates external CUDA pointers with one batched attribute query
and a mapped-range check, and caches each context’s PCI/NUMA discovery. This
follows TENT’s Platform / MemoryProber responsibilities without a global
raw-pointer registry; see memory location.
cuda/probe/topology.rs supplies TENT’s CudaProbe inventory to RDMA discovery,
platform/pci.rs resolves PCI/NUMA and bridge distance, and
runtime/topology.rs::MemEntry stores host and CUDA locations with three NIC
tiers. runtime/topology/config.rs implements TENT’s loadFromConfig,
parsePriorityMatrix and parseCustomTopology. DeviceSelector compiles all of
this into rank arrays and masks once, so slice allocation does no driver, sysfs
or string work, preserving TENT’s CudaProbe → Topology → DeviceSelector chain.
Owner |
TENT responsibility |
|---|---|
|
Bandwidth policy, selection EWMA and busy-time meter ( |
|
|
|
|
|
Priority tiers, |
|
Mapped current priority slot and joined monotonic publisher; per-NIC leases remain in DeviceSelector |
Physical grant release stays separate from payload sampling, and route identity excludes link speed, so a rate change keeps existing QPs. See bandwidth-aware selection.
CUDA IPC transport ownership#
Owner |
Responsibility |
|---|---|
|
Fabric-domain capability and grant binding, CUDA VMM references, VA reservations, access descriptors and EGM socket placement |
|
Legacy or fabric identity checks, grant and close bindings, mapping quotas, stream pool and completion progress |
|
Allocation-wide source loan; one imported reference with its context and touched local allocations |
|
Mapping charge and explicit native close |
|
Consecutive same-region, same-context copies |
|
Deduplicated owner locking, ordered enqueue and native batch arrays |
|
Inline ready check, then one lazy worker querying pending events |
|
Optional import cache and persistent source export lifetime |
The stream handle is acquired before memory locks and returned before close and
control I/O; loans survive event polls without holding allocation mutexes. TENT
keeps a transport-wide relocation cache and queries the task event in
getTransferStatus; OrbiTE closes imports per subbatch by default and adds
explicit close receipts. See CUDA IPC.
MNNVL shares this completion owner and its bounded import/export cache. The creator drops its physical handle reference after mapping. Local free retains the handle again, unmaps, releases it and frees the VA. An imported mapping keeps its physical reference until the same teardown sequence. Frozen TENT clears its relocation table on uninstall. OrbiTE uses explicit native receipts, bounds whole mapped extents and preserves the actual owners on uncertain cleanup. New device and EGM mappings are zeroed through the rounded extent before their first grant. Handle publication requires an admitted range in the original epoch. See MNNVL and EGM.