CPU TCP foundation

This page covers the CPU memory, registration, TCP and notification contracts (P1.1–P1.4, P2.1–P2.2 in the completion plan). The TCP payload path uses Rust sockets; the crates.io alpha does not contain it.

Architecture correspondence

The reference is Mooncake TENT e1ea5d31bb5e46ff2df9f90e7c2fbaf52fcdee8c.

TENT responsibility	Rust owner	Delivered subset
runtime/transfer_engine_impl	src/engine.rs, src/batch.rs	Immutable config, sync/async batches, ordered execution and requester/source drain
runtime/segment, segment_tracker	src/memory.rs, src/segment.rs	Shared allocation owners, reference-counted ranges, descriptors, permissions and epoch fencing
runtime/control_plane, RPC dispatch	src/runtime/control_plane.rs (ControlService)	P2P descriptor open/refresh and source-owned handlers
transport/tcp	src/transport/tcp.rs (TcpTransport)	Host byte movement, deadlines, complete READ response and final WRITE ACK
Notification queues and submit hooks	src/notification.rs, src/engine.rs	Bounded named messages, atomic polling and batch-completion delivery
Config and resource observations	src/config.rs, TransferEngine::stats	Per-engine budgets and resource counters

ControlService owns the shared socket service, descriptor and notification protocol, source admission and dispatch. TENT keeps its TCP payload handlers in ControlService; OrbiTE moves the byte behavior into TcpTransport. RDMA uses a ControlChannel for setup without going through TcpTransport. Cache keys, replica grants and recovery policy stay with OrbitKV.

Ordinary TCP leases one exclusive stream per endpoint and authoritative source epoch to the executing caller or batch worker. Completed frames reuse it; WRITEs on that stream stay ordered. Concurrent direct-TCP submissions to that peer serialize on this lane; independent peers have separate lanes. The requester pool uses max_peer_segments slots, including borrowed and uncertain entries. At capacity it can evict only an idle, clean, unreferenced entry. A busy or failed entry stays charged. close_segment fences admission and drains the peer before closing its cached stream. No new worker thread or public configuration field is added.

The existing control workers monitor idle streams through mio readiness. Idle streams hold neither source-memory admission nor registration loans. Their shared cache is also bounded by max_peer_segments, in addition to the fixed active handlers and bounded accept queue. Each complete TCP frame gets fresh source admission and the existing handler deadline. Direct TCP waits on the socket with deadline-bounded waits; the staged CUDA pipeline retains its bounded nonblocking progress path.

A timeout, partial frame, invalid reply, lost acknowledgement or abandoned lease retires the stream; it never returns to the cache. A confirmed EOF on an idle, already completed stream can reconnect before the next request posts. There is no transport-local replay. Failed socket retirement returns Unquiesced, retains the descriptor and its slot, and is never attempted again. The existing task owner keeps backing and transfer charges on that outcome. Running TCP cancellation remains unsupported; queued cancellation posts nothing.

The control acceptor uses readiness and a bounded handler queue. A full queue closes the new connection. Peer resets/timeouts leave the listener available; resource exhaustion logs once per burst and retries on a bounded interval. Fatal listener errors are logged. Shutdown wakes and joins the acceptor and handlers. Both ends enable TCP_NODELAY. See the profile and matched measurements for the measured costs and limits.

Rust contracts

• Owned memory. register_memory(Box<[u8]>, Permission) consumes the backing, which handles and admitted handlers keep alive through Arc. The source checks remote permissions regardless of requester metadata.

• External memory. Unsafe HostMemory::from_raw_parts(address, length, owner) adopts a pre-existing host allocation; owner must keep it alive, stable and writable. It does not pin pages or make a device pointer host-accessible.

• Subranges and batches. register_host_memory(memory, offset, length, permission) registers a checked subrange; overlaps must share one HostMemory lock. register_host_memory_batch preflights the whole batch and publishes it atomically, so a failure leaves existing state intact.

• Duplicates. As in TENT’s SegmentTracker, an exact duplicate shares one ID and charge, the first permission wins, and each call adds one reference. Duplicates inside one Rust batch are coalesced (TENT can publish them separately).

• IDs and unregister. Registration IDs are never reused. The final unregister_memory reference removes visibility; running handlers keep their backing and charges until they finish.

• Address lookup. resolve_local_address and RemoteSegment::resolve_address need one range covering the whole request, using TENT’s address-ascending, length-descending order, and never dereference an unregistered address.

• Authority. open_segment(endpoint, expected_epoch, timeout) takes the 128-bit epoch from the authoritative source owner, and every operation rechecks it at the source, so a new process at the same address cannot inherit authority.

• Requests. TransferRequest::new(transport, local_address, remote_address, length) takes absolute addresses that carry no authority. Submission resolves and retains every local registration before I/O, so a concurrent unregister cannot free admitted backing.

• Execution. submit_and_wait validates every request first, runs them in order and stops at the first failure; the report has the completed prefix, exact bytes and failed index. A batch is not a transaction.

• READ and WRITE. READ receives the full payload before touching its local slice. WRITE copies its input, waits for source readiness, sends the payload and waits for an ACK after the source commits. A WRITE failure after sending returns RemoteEffect::Unknown and is never retried.

• Deadlines. One absolute requester deadline covers connects, frames and payloads; a separate source deadline starts at accept. Neither is a hard real-time bound.

• Source close. close_source(timeout) permanently closes source admission and waits for handlers; a timeout keeps admission closed and can be retried.

Async batches use the same validation and executor; see async batches.

Notification contracts

Notification::new(name, message) carries UTF-8 strings (empty strings and repeats are valid). send_local_notification delivers in process; send_notification(peer, notification, timeout) uses epoch-fenced TCP for other peers. A receipt confirms enqueue, not consumption. receive_notifications() atomically drains the queue in enqueue order, so concurrent pollers get disjoint messages. Matching, generations and waits belong to the consumer.

submit_and_notify sends once to the batch peer after every request succeeds; empty or failed batches report NotificationStatus::Skipped. Data completion and report.notification are separate, and a failed notification never authorizes replaying data. The receiver reserves count and bytes before accepting a body, and malformed or partial bodies never enter the inbox. After body submission without a final receipt the effect is Unknown; manual retries can duplicate a message.

The sender tries each installed notification-capable channel in order. Only an unavailable channel with no delivery effect permits fallback. OrbiTE’s installed channel is TCP. The independent control RPC can receive without that channel, and uses the same bounded inbox as self delivery. Set tcp_notifications to disable the TCP channel or notification_rpc_fallback to stop after its first attempt and disable RPC fallback. Both default to true. When the RPC also fails, the sender preserves the last channel error and any uncertain RPC effect. Receive drains all installed queues even if one fails. Successful deliveries take precedence over an empty-channel error. Async completion hooks retain only destinations still eligible for retry.

Budgets and profile

Default	Value	Accounting
Bind	127.0.0.1:0	Loopback with an OS-chosen port; other hosts need a routable bind
Retained registrations	1,024 / 1 GiB	Retained ranges, including unregistered handles; protocol limit 4,096 regions
Batch descriptors	1,024	Rejected before effects if exceeded
Allocated async batches / slots	32 / 32,768	Open, finished-but-unfreed and detached batches
In-flight requester submissions	32	Async groups plus synchronous, open and notification calls
In-flight requester bytes	64 MiB	Request bytes plus notification bytes
One payload	8 MiB	Configurable up to 64 MiB; each source enforces its own
Source workers / queue	4 / 4 sockets	Queue overflow closes the connection
Source timeout	5 s	Accept-to-handler deadline, configurable in (0, 300s]
Pending notifications	1,024 / 1 MiB	Queued messages plus reserved incoming bodies
One notification	64 KiB	Name plus message bytes, enforced at both ends

The protocol is plain TCP for trusted environments: epochs fence restarts but do not authenticate peers. There is one connection per request and sequential batch execution. stats() samples registration, admission and notification counters under separate locks. These budgets cover registered ranges, not an external allocator’s own footprint.

Development wire contract

OrbiTE uses its own pre-release control protocol; see the wire policy. Integers are unsigned big-endian and each connection carries one operation. Both headers begin with the eight-byte ORBITTE\0 magic, a two-byte version, a one-byte operation or status and one zero reserved byte.

Frame	Remaining fields
Request, 52 bytes	Expected epoch (16), registration ID (8), offset (8), length (8)
Reply, 36 bytes	Actual source epoch (16), payload length (8)
Descriptor payload	Region count (4), segment capabilities (44, including the eight-byte maximum transfer length), then records ordered by registration ID. Each contains ID (8), base address (8), length (8), permission bits (1), memory type (1), IPC export consent (1), HP-TCP registration flag (1) and MNNVL export location (4)

The selector guide describes the capability fields; each transfer still needs a grant. The retired provider capability byte is reserved and must be zero; old provider requests are rejected before admission. See the wire policy.

Segment capability bit 7 requires the RDMA capability and identifies a source whose immutable RDMA policy directly registers published GPU ranges. Classic scatter checks it before admitting a remote GPU fragment.

In each region’s flag byte, bit 0 marks HP-TCP registration. Bits 1 and 2 record observed dma-buf and peermem registrations respectively, only for a CUDA region on an RDMA-capable source. Unknown bits are rejected. Bit 7 marks a registration with a scoped transport or known device/NUMA ordinal. All fixed records precede the optional five-byte entries, in flagged record order. Each entry contains a TENT transport ID (one byte) and a big-endian ordinal (four bytes, all ones for unknown). Internal and local-only regions are omitted. A C P2P handle has no supplied epoch. A read-only Describe with zero expected epoch first obtains the source’s identity from its mismatch reply. The handle then issues a normal epoch-checked Describe.

For an engine with SHM enabled, capability bit 5 announces a trailer immediately after the optional registration profiles, before any HP-TCP endpoint trailer. It starts with a 32-bit count, a one-byte backend (0 for POSIX, 1 for a configured directory) and three zero bytes. Each 80-byte record starts with seven ordered 64-bit fields for registration ID, allocation offset, mapped size, grant slot, device, inode and PID namespace inode. A 32-bit creator PID, four zero bytes and a 16-byte name nonce follow. The leaf is orbite-shm-{namespace:016x}-{pid:08x}-{nonce:032x}. POSIX opens it with a leading slash. The directory backend opens it relative to the local mount. Record IDs increase and every range must fit the published object. tests/reference/shm/describe.bin freezes a sample trailer.

Capability bit 6 announces MNNVL. A 24-byte trailer follows the SHM records when present and precedes the HP-TCP endpoint. It contains the domain (16), handle kind (1, fabric 0 or POSIX fd 1), EGM enable (1) and six zero bytes. The per-region final big-endian word is zero for an ordinary registration, one for a GPU export, or the high bit plus a host NUMA ID for EGM. tests/reference/mnnvl/describe.bin contains a trailer and an EGM location word.

The control service sends opaque handles only in an authorized transfer grant. MNNVL reuses the NVLink request, grant and close-receipt layouts with MNFG and MNFC hello markers for fabric or MNPG and MNPC for POSIX. Grants use the corresponding non-cache marker. Identity binds the fabric domain and requester process. The 64-byte share field holds CUDA fabric bytes or a big-endian PID and fd (4 bytes each), PID namespace inode and process start ticks (8 bytes each), followed by zero padding. A POSIX import checks the process identity and duplicates the fd through a pidfd.

Operations: 0 describe, 1 TCP READ, 2 TCP WRITE, 3 notify, 4 RDMA READ, 5 RDMA WRITE, 6 sliced RDMA READ and 7 sliced RDMA WRITE (see RDMA/CUDA). Control RPC notify uses operation 22. Describe requires a zero ID and range. Both notification operations use ID zero, with offset as the name length and length as the message length; the body contains the UTF-8 name, message and a zero commit marker. The receiver sends final success only after its registered callback accepts the complete message. Permissions: 0 local-only (never exported), 1 remote read, 2 remote write, 3 read/write. Statuses: 0 success, 1 epoch mismatch, 2 unknown region, 3 permission denied, 4 invalid range, 5 source closed, 6 exhausted, 7 protocol error, 8 internal error, 9 ready, 10 backend unavailable, 11 remote native backend error and 12 explicit RDMA cache miss. Unknown versions, flags, IDs, enum values, out-of-bounds ranges and oversized payloads are rejected.

READ: request → success header and the requested bytes. WRITE: request → ready → bytes → final success. Notify: request → ready → body and marker → final success. Each side commits only complete bodies, a dropped reply never authorizes replay, and the descriptor count, per-region metadata and provider trailers are bounded before decoding.

Verification

From the checkout, with permission to bind loopback sockets:

cargo fmt --check
cargo clippy --locked --all-targets -- -D warnings
cargo test --locked
cargo run --locked --example tcp_roundtrip
cargo run --locked --example external_host
cargo run --locked --example notifications
cargo run --locked --example async_batch
cargo publish --dry-run --locked

Unit tests cover ranges and permissions, epoch parsing, budgets, admission, close/drain and malformed or fragmented frames. Real TCP controls inject short READs, incomplete WRITEs, lost ACKs and trickling responses. tests/tcp.rs covers prevalidation, prefix completion, concurrency and a restarted source process at the same port (the ignored peer_process_fixture is that gate’s child, not a skipped case). tests/notifications.rs covers queue bounds, concurrent senders and pollers, and data completion despite notification failure. The CPU oracle compares one valid-input trace against native TENT. CI runs all of this on stable and Rust 1.85; physical two-host and hardware gates are separate.