Asynchronous CPU batches#
P1.4 implements a bounded CPU/TCP subset of TENT F08/F10. The completion plan remains the only execution queue. This source profile is pending independent acceptance; RDMA/CUDA and installed consumer qualification are separate.
Ownership and API#
Use Arc<TransferEngine>::allocate_batch(capacity) to reserve descriptor slots
and an engine-scoped monotonically increasing BatchId. Capacity is positive
and at most max_batch_requests. max_allocated_batches and
max_allocated_requests bound all live reservations, including completed batches
that have not been freed and dropped handles whose workers remain active.
Batch::submit(operation, peer, requests, notification, timeout) appends a group
and returns its stable public task range. All addresses, permissions, bytes and
source authority are validated before enqueue. The group owns resolved local
registrations and a requester-admission charge. A rejected group adds no public
tasks; accepted earlier groups are never replayed or rolled back. Groups in one
batch execute FIFO; different batches can progress concurrently. Each batch has
at most one worker, bounded by its allocation reservation. Submission records,
including empty submissions, are bounded by capacity too.
seal() stops append. status(), task_status(), task_statuses() and
submissions() only read worker-published state; they perform no transfer,
retry, notification or progress step. Before seal the overall state is Open,
even if every task submitted so far has completed. After seal, Pending becomes
Completed, Failed or Cancelled only after all accepted groups exit.
Per-task terminal states never change, and the sealed overall terminal is stable.
wait(timeout) requires seal and uses a condition variable. Its own timeout
neither cancels nor releases work. A submission’s independent absolute deadline
begins at submit and includes validation/queue time and attached notification.
free() returns BatchBusy while a group still owns local work, preserving the
handle for retry. Once idle it seals, joins its worker, releases descriptor
capacity and its engine reference, and invalidates further batch operations.
Dropping an active handle instead disconnects the queue without discarding
accepted groups: the detached worker retains the engine, registrations and both
allocation/admission budgets until it exits. This prevents repeated handle drops
from bypassing the thread or retained-owner limits.
submit_async(...) allocates, submits one group and seals it. See the runnable
async_batch example for multiple appends.
Results, cancellation and quiescence#
Tasks expose waiting/running/completed/failed/cancelled/skipped states, exact
bytes of fully completed requests, cancellation requests and a retained structured failure. A failed remote staging
request can have visible completed READ chunks while contributing zero completed
bytes; drain status is not rollback. Failure indices are public batch task indices. request_index identifies the first
failed task; affected_requests gives the consecutive failure span. A shared
NVLink close/ACK failure can cover several already attempted tasks. Every task
in that span shares the failure and drain status; only subsequent tasks are
skipped/cancelled without I/O. A clean copy/cancel failure followed by a source
ACK preserves the copied prefix. Other accepted
groups still run. Submission reports preserve each group’s completed prefix and
independent NotificationStatus; successful data can coexist with notification
failure. Empty or failed groups do not send attached messages.
cancel(task) and cancel_all() request cancellation. Waiting work checks that
request before dispatch. A running CPU operation finishes or reaches its existing
deadline; cancellation does not interrupt an unproven physical operation or
relabel committed bytes as cancelled. A late cancellation can therefore coexist
with a completed task. This profile has no provider abort or cancel receipt.
locally_drained means the requester’s physical access to its registered backing
has ended. Group/batch drain is published after dropping the actual prepared
owners and their admission charge. It does not establish remote source
quiescence. In particular, a lost WRITE acknowledgement retains
RemoteEffect::Unknown, forbids automatic replay and still requires source-side
ownership/drain before reusing remote backing. Source handlers retain their own
owners independently.
close_requester(timeout) permanently rejects new outgoing work, including
open, notification and async submissions. It waits for accepted queued/running
work to release local owners. Timeout leaves admission closed; call again to
wait. close_source separately gates source registration and handlers. Existing
source-only close behavior is preserved. Whole-engine close(timeout) fences
both directions before waiting against one deadline; accepted queued work can
still create backend slices while draining. It then explicitly closes shared
RDMA device resources. A native cleanup error is retained without retry; see
the device owner contract. Closing requester admission does not
invalidate existing status/free operations or claim that a remote ACK was read.
Frozen correspondence and limits#
The frozen TransferEngineImpl
allocates batch capacity, appends public tasks, retains runtime references and
defers freeing batches that remain pending. Rust’s explicit busy-on-free plus
detached-handle ownership is a distinct safe API; matching the native deferred
C free convention remains an adapter requirement. Rust adds seal to make an
overall terminal result stable against further append; native TENT can append
after prior tasks complete.
The shared CPU trace now uses real async Rust APIs and native C allocate/append/status/free with the same 25 byte/notification/cleanup observations. Native queue/merge/progress-worker modes remain disabled for that profile. Rust blocked-socket tests exercise queue deadline, cancel, wait timeout, concurrent unregister, requester close and detached engine/allocator release. They do not qualify native fault behavior or device DMA.
RDMA/CUDA now consume these owners with explicit per-request
transport selection. Their Unquiesced failures retain native owners and
admission/capacity: the logical result is terminal while locally_drained is
false, free remains busy and close cannot report drain. No reclaim proof is
implemented for this quarantine yet.
Request merging/alias mappings, full transport selection policy, explicit progress, priority/EDF, provider abort/reclaim and language/wire compatibility remain open. No hardware backend inherits this CPU profile’s drain proof.