vLLM and SGLang integration

OrbitKV integrates external cache recovery with engine-owned GPU pages and request lifecycles. Start with the single-node quickstart. Source/package pins are official vLLM 0.31.0 and SGLang 0.5.22. The SGLang public load-completion path passes installed Qwen3-8B dense TP1/PP1 DRAM/io_uring byte and model/restart controls. See compatibility.

The broader workloads below retain their recorded vLLM 0.31.0 and SGLang 0.5.21 versions. The new 0.5.22 ordinary-cache controls do not extend their Graph, shared-pressure or peer-cache scope.

The installed-wheel gate rejects editable/source imports and changed installed files. Graph evidence requires actual native runtime observations, not capture logs. Peer bytes land in consumer DRAM before its local GPU copy. These healthy controls do not prove physical fault termination, eventual reclamation or page reuse after a cancelled/partially submitted transfer. Historical 0.29/0.30 fork and topology evidence retains its original version and limits.

Ownership contract

Before queueing, adapters call native prepare_save at the producer handoff. Rust holds and budgets immutable source requests, batches submit_saves, and keeps registration/exporter owners until completion. cancel_save succeeds only before native submission. An unknown handoff keeps engine completion pending; SGLang also preserves its FIFO tree locks.

The retained integration gates exercise native producer readiness, complete restored bytes and sentinels, original IPC-credit repayment and registration retirement on vLLM 0.31.0 and SGLang 0.5.22. The SGLang GPU-pool/linker fixture covers both sole ownership and an independent exporter hold; vLLM includes a no-fence counterfactual. SSD gates evict DRAM and require actual io_uring reads. These normal-path controls do not establish complete model shutdown, cancellation, page reuse/generations or physical-fault reclamation. For Restore, only cancellation of the pre-forward activation future proves non-submission. A CancelledError from submission or completion waiting keeps leases and GPU destinations held and propagates an error to the consumer.

Neither selected release currently supplies the allocator-owned page identity needed to reject a late completion after physical slot reuse. The missing vLLM and SGLang interfaces are tracked in vLLM #62810 and SGLang #45493. Local candidate code is contribution evidence only: OrbitKV does not add a version-fallback facade, patch the selected engine package, or call an upstream proposal released support. Current adapters continue to retain uncertain pages. The reviewed contribution candidates are draft vLLM #62819 and SGLang #45548; neither changes the supported engine pins or closes installed page-reuse fault qualification.

Historical per-cohort details and failures remain in the accepted lifetime record. Use the completion plan for current-release acceptance; earlier 0.5.21 results retain their original scope.

The Publish activity candidate is consumed by vLLM’s transfer-result poll and SGLang’s completed-offload count for bounded diagnostics. It observes native active/retained callers and a local active interval without taking the blocked descriptor/lifecycle locks. Missing observations, an overdue call or a dead Manager cannot produce an offload completion. Existing synchronous Save still holds its source pages; this does not qualify asynchronous handles, page-slot reuse or fault reclamation.

A hit is a recoverable state boundary across every required group, not simply a stored key. The engine supplies destination pages and their readiness; native work retains source and destination ownership until drain. Enqueue acknowledgement, per-layer readiness, terminal completion and source retirement are separate facts. Model/adapter identity, cache salt, state representation and shard layout must match. Sharing a Manager does not make engine layouts interchangeable.

Explicit cache reset

Official vLLM 0.31.0 distinguishes an HBM-only prefix reset from POST /reset_prefix_cache?reset_external=true (development API), or the public engine reset_prefix_cache(reset_connector=True) call. OrbitKV refuses an external reset while it tracks requests, query leases, restores or saves. Once idle, it changes the prefix-key generation used by both lookup and publication. New requests cannot reuse the previous generation; newly saved state can be reused after an HBM-only reset. Each immutable process restart still derives its initial domain from the model artifacts and computation configuration.

This reset affects one engine process, not all replicas or physical Manager storage reclamation. Its keys are not shared with other replicas after reset. It does not prove safe live weight updates that skip reset, active-request reset or cancellation of native DMA. SGLang’s ordinary flush_cache continues to clear HBM and retain same-weight external reuse; released skipped-flush weight updates remain unsupported. The completion plan records qualification separately.

Immutable static LoRA

Static adapter support is independently accepted for a fixed local PEFT adapter set on official vLLM 0.31.0 and SGLang 0.5.21, dense TP=1/PP=1 eager with direct transfers and io_uring SSD. The installed-model evidence is tracked in S5.3. At startup OrbitKV hashes each adapter’s configuration and weights and binds the complete named set and LoRA computation configuration to the external cache namespace. Engine prefix hashes still distinguish the base model and individual adapters. Replacing weights at the same name/path between process restarts changes the namespace. Changing any adapter or adding one invalidates reuse for the whole declared set, including the base model; this is conservative deployment isolation, not independent adapter version migration. ORBITKV_MODEL_FINGERPRINT never substitutes for adapter bytes.

For vLLM, declare the same adapters in the released loader and the connector:

vllm serve /path/to/immutable-model --enable-prefix-caching --enable-lora \
  --lora-modules fixed=/path/to/immutable-adapter \
  --kv-transfer-config '{"kv_connector":"OrbitKVConnector","kv_role":"kv_both","kv_connector_module_path":"orbitkv.vllm","kv_connector_extra_config":{"orbitkv.static_lora_adapters":[{"name":"fixed","path":"/path/to/immutable-adapter"}]}}'

The public on_new_request callback validates name/path and fixed bidirectional integer ID bindings before native HBM lookup. It rejects undeclared selections, load_inplace, tensorizer, 3D adapters and resumable requests. The runtime HTTP update flag must remain disabled. A violated request contract stops EngineCore; it is not a graceful per-request HTTP rejection.

For SGLang, OrbitKV reads normalized startup adapter references and effective LoRA options from the official runtime_context.get_lora() configuration bag. ServerArgs retains raw inputs, so --lora-paths can enable LoRA while its raw enable_lora remains unset. Use the released loader as follows:

ORBITKV_STATIC_LORA=1 ORBITKV_SGLANG_ENDPOINT=unix:///tmp/orbitkv-50055.sock \
  sglang serve --model-path /path/to/immutable-model --page-size 64 \
  --lora-paths fixed=/path/to/immutable-adapter \
  --enable-unified-cache-external-linker --radix-cache-backend orbitkv

The existing wrapper checks declared adapter UIDs before external lookup, including complete HBM matches; the existing pending-admission Hook also checks requests before computation. Caller extra_key is disallowed in this profile: SGLang concatenates it with the adapter UID, so a base-model user key could otherwise alias an adapter. Use cache_salt for request isolation. Radix-cache sessions and streaming sessions are rejected because their retained pages can bypass matching when adapters change. The pre-capture and admission Hooks remain required; this does not close the public-lifecycle migration.

Keep adapter files, symlink targets and loaded bindings unchanged for the entire engine lifetime. Startup loader configuration must match the declaration. Do not call engine SDK/RPC add, remove, reload, in-memory tensor loading or weight updates while this profile runs. Those calls can bypass cache callbacks; OrbitKV does not intercept them or establish safe dynamic invalidation. Stop the engine, change artifacts/configuration and restart to update the deployed adapter set. Remote adapters, dynamic LoRA, live base-weight updates, resumable/session requests and broader model/topology profiles remain unsupported or unqualified.

LMCache and FlexKV reference

Use release source, not an unversioned example, when comparing integrations:

Later SGLang main changes toward LMCacheUnifiedRadixCache are not the 0.5.20 release contract. API similarity and upstream support matrices do not qualify OrbitKV. Do not inherit LMCache’s version compatibility branches or assume its documented P/D prerequisite patches are in the selected engine release.

The pinned vLLM 0.30.0 FlexKV entry loads its separately installed implementation when the connector is constructed. The SGLang 0.5.20 FlexKV integration owns task completion and connection lifetime beside the engine cache. OrbitKV uses these explicit backend boundaries while retaining its existing Rust client, Unified tree and per-layer GPU completion dependencies. The separately inspected FlexKV adapter at 738ddc14 is a pinned main reference, not an OrbitKV-supported release. Like LMCache MP, it separates scheduler callbacks from worker callbacks. OrbitKV follows that ownership split without inheriting another cache’s task engine or compatibility branches.

vLLM callbacks and current internal dependencies

OrbitKVConnector implements KVConnectorBase_V1 and SupportsHMA. SchedulerAdapter in scheduler.py discovers legal external coverage and hands allocated pages to WorkerAdapter in worker.py; the worker registers tensors, restores/saves state and reports completion through the pinned engine contract. layout.py and metadata.py describe actual groups and intents rather than a second cache scheduler.

All worker roles implement the released engine’s get_transfer_results directly. P/D failure and receive completion are drained together into KVConnectorTransferResults; the scheduler consumes native failed_recving, including through MultiConnector. There is no P/D failure-metadata class or duplicate failure queue. Ordinary cache publication declares requires_kv_delivery=False; a reliable P/D producer keeps the engine’s delivery requirement. Divergent hybrid hits remain disabled.

Only the selected role implementation is imported and constructed. Metrics come from that role; native sessions close if role initialization fails. The connector inherits unchanged optional callbacks from the pinned engine base. Its required layer-save callback remains explicit because saves are submitted at step end. Restore failures propagate with their retained GPU ownership; there is no separate availability flag or background health-polling thread. Plugin registration errors remain visible instead of silently selecting a conflicting connector.

The official-release profile requires VLLM_USE_V2_MODEL_RUNNER=0 and one attention cache group. The V1 runner invokes the public preemption callback before updating pages. The runtime.py runner replacement and block-pool native-prefix override are deleted. V2 and multi-group/recurrent serving fail before opening a Manager connection; enabling a capability flag is insufficient to restore those profiles. Framework-neutral recovery contracts remain in Rust.

Registration currently uses the external package plugin. Official upstream registration must replace duplicate registration, not hide name conflicts. Use the S5 sequence for release alignment, lifecycle PRs and removal gates.

SGLang linker and current internal dependencies

Select --radix-cache-backend orbitkv with --enable-unified-cache-external-linker. plugin.py constructs a UnifiedRadixCache; OrbitKVLinker implements lookup, queued load, layer progress, offload, completion, reset and close. SGLang retains tree locks and page allocation. The release linker lookup returns ready boundaries, not a pending ticket; current OrbitKV admission Hooks defer a pending request while allowing other requests to progress. Do not assume HiCache scheduler hooks run when OrbitKV rejects the separate hierarchical-cache mode. The hybrid recovery contract defines legal state boundaries.

Request admission and hybrid materialization use scalar decisions when the released attention/TP groups have one participant. With multiple participants, the existing collective decision still waits for every required rank before admitting or exposing restored state. This removes redundant single-rank tensor allocation without changing cancellation, lease expiry or the internal Hooks. The 0.5.22 interface passes the affected CPU controls and the selected installed GPU byte/model gates. Physical multi-rank agreement remains unqualified.

events.py owns the CUDA events and producer/consumer counters needed before graph capture. linker.py consumes those events and owns cache queries, loads, offloads and registration. Plugin discovery imports neither the native extension nor GPU layout code; graph and admission hooks check the selected backend before loading OrbitKV resources. Native bindings load on first native API access.

RecoveryLinkerWrapper now delegates private destination allocation, native cancellation and tree insertion to SGLang 0.5.22. It only translates absolute recovery boundaries and rank agreement. finish_layer_wise_loading(index) waits for native Restore before the engine inserts or frees destinations; unknown completion raises and retains ownership. The old load-polling queue, custom tree insertion and recurrent component override are removed. Full/SWA are the only released external-linker components, so recurrent/SSM startup is rejected. Aborts use BasePrefixCache.finish(ABORT) and the native linker release path; lookup-only cancellation also retires its unsubmitted cache query. There is no additional Scheduler abort Hook.

The ordinary cache profile registers two internal Hooks: graph initialization and pending-query admission. Enqueue preparation adds one Hook only when ORBITKV_PREPARE_REQUESTS=1 or ORBITKV_QUEUE_WARMUP=1 is set before startup. P/D transport factories and fork observation callbacks are removed. Native P/D uses the official engine lifecycle and is qualified separately. The S5.2 evidence accepts normal prepared-adapter retirement on the selected release. It does not qualify crash reclamation or cancellation during DMA. The earlier model-process shutdown warnings and calibrated 0.5.21 measurements remain in the immutable shutdown record.

These remaining targets are version-coupled dependencies, not stable public APIs. Replace them with consumed factory/lifecycle/component contracts and then delete the duplicate logic. Unknown DSA, draft, auxiliary state and unsupported request rings remain rejected until they have complete recovery contracts.

P/D and cache composition

Manager shared-cache recovery uses local metadata discovery, an authoritative source grant and TENT READ before GPU restoration. Live P/D uses each official engine’s native transport and lifecycle. vLLM uses NIXL plus MultiConnector; SGLang uses native disaggregation with the independent external linker.

The candidate vLLM profile reads/writes cache on P and uses save-only cache on D. SGLang likewise disables external restoration on D. Native P/D is the sole incoming destination writer; cache saves read completed state and retain their own leases until drain. Ordinary cache publication remains best effort.

The P/D guide records released configuration and qualification limits. Removed fork factories, callbacks and tests are preserved as immutable upstream contribution material; their passed results do not qualify the official engines.

Native client

from orbitkv import CacheManagerClient

client = CacheManagerClient("/tmp/orbitkv-50055.sock")
ok, message = client.health()
client.close()

client.peer_alive() observes the authenticated Manager process through its pidfd without an RPC or waiting behind a Publish. A live stalled Manager returns true, process exit returns false, and unavailable identity raises an error. Replacing the socket does not rebind an old client. Liveness does not authorize GPU page reuse or release a source grant; native operation drain remains separate. SGLang failed-session probing is still gated by its adapter qualification.

CacheManagerClient is the Rust owner exposed through PyO3. Construct BlockHashes(page_hashes) once per lookup; slices share its native allocation. query_prefetch owns submission, revision and polling. register_context_batch(..., tensors=...) retains actual tensor/exporter objects alongside the IPC metadata used by Publish and Manager SSD/codec routes. start_restore(..., ready_stream=...) takes the engine’s destination-readiness stream and returns a client-bound handle for poll_restore or wait_restore(timeout=...). Optional layer_events bind retained CUDA events to registered layers. wait_restore_enqueued waits for fresh event records, allowing per-layer consumer dependencies; it does not acknowledge destination page reuse. Unencoded DRAM copies execute in the native engine worker; their final local result follows DMA drain without waiting for Manager source reaping. vLLM uses layer waits in piecewise mode and coarser entry waits for supported full-graph layouts; SGLang installs external events before capture. Publish uses the opaque native GpuRegistration returned by successful registration. vLLM save and boundary tasks, and SGLang offload tasks, retain the original object when they enter the queue. The native client rejects a retired registration or a different client before submission. vLLM refuses a batch spanning registrations; it does not switch queued work to a new registration. The object retains exporters until the last queued/user reference is dropped. These checks bind registration lifetime; same-allocation page-slot reuse and physical failure drain still need their own proofs and hardware gates.

The native Save contract distinguishes SaveNotSubmittedError from unknown handoffs. From start_save or prepare_save, it proves the current invocation launched no native Publish worker; from submit_saves, it proves the entire batch was rolled back without submission. Handle allocation, attachment and wait errors retain the page hold. Native producer methods and their registration/handle arguments are resolved before the typed catch. Python method or argument lookup errors retain pages; the exception type alone cannot establish that native code ran and rolled back. SGLang also preserves its official completion FIFO: once an earlier result is unknown, the worker retains later queued tasks without starting them or reporting their completion against the earlier engine node. This does not reconcile unknown owners or prove physical drain; qualification is tracked in the completion plan.

See layer consumption for recurrent, packed-buffer and multi-part limits. Repeated registration of the same binding is rejected, and unregister or close drains accepted operations. Native calls release the GIL. A timeout does not release destination page assignments while a copy may still be running. See the type reference.

Process channel

The connector derives the Cache Manager’s Unix socket from the configured endpoint. Scheduler Query/Release and worker Publish/Restore use iceoryx2; registration, health, session ownership, and cleanup use the bootstrap UDS. Each inference process requires a Cache Manager on its own host. A missing socket fails at startup. No extra configuration is needed on one node.

The process channel connects to /tmp/orbitkv-<orbitkv.port>.sock, matching the Cache Manager default. Use orbitkv.bootstrap_socket for a custom single-manager path. orbitkv.timeout_ms (default 5000) bounds hot requests and health; registration and unregister allow at least 120 seconds for CUDA setup/draining. orbitkv.spin_iterations defaults to 64. Standalone Cache Managers do not start gRPC. Client and Cache Manager must use matching bootstrap protocol versions (currently bootstrap 7, channel ABI 12, cache schema 10, and lifecycle 9). Bootstrap transfers five metadata/notification FDs; GPU registration attaches the shared payload arena FDs separately, with the resolved Manager-local CUDA ordinal preceding arena metadata. Python APIs use client-local CUDA ordinals; the native registration owner resolves tensor UUIDs once and retains the Manager route for Publish/Restore. NUMA detection uses only Manager-visible CUDA devices and their stable UUIDs, independently of physical index order. The identity repair’s hardware qualification is separate from broader multi-GPU deployment. The local executor partitions large raw plans into at most 1 MiB parts under one whole-operation fence. Operation metadata is capped at 32 MiB and per-session prepared metadata at 64 MiB; see execution scope and qualification gates.

orbitkv.wait_for_full_prefix is supported on the local path: pending queries return QueryLoading, and repeated queries with the same instance/request/group identity retrieve the result. Query arguments must remain unchanged while pending. Superseded queries are explicitly cancelled. Each session permits 128 outstanding queries, with 1024 globally and a 60-second reply lifetime. Cancellation revokes waiting interest while submitted reads drain safely; undelivered results release their leases without another readiness poll. Use the same wheel version and CUDA variant for the Cache Manager and clients. Connector shutdown explicitly closes the UDS session; imported CUDA mappings are released after queued GPU transfers finish.

Connector Modes

OrbitKVConnector defaults to read_write: it queries OrbitKV for reusable KV blocks, loads matched blocks into vLLM, and saves newly computed full blocks back to OrbitKV.

Set orbitkv.mode to save_only when another vLLM connector is responsible for reads and OrbitKV should only persist KV blocks for later reuse. This is intended for MultiConnector decode-side setups where an upstream connector owns the external hit/load path, while OrbitKV records the resulting KV cache. In save_only mode, OrbitKV does not query or load KV blocks.

vllm serve /path/to/immutable-model \
  --kv-transfer-config '{
    "kv_connector": "MultiConnector",
    "kv_role": "kv_both",
    "kv_connector_extra_config": {
      "connectors": [
        {
          "kv_connector": "<external-read-connector>",
          "kv_role": "kv_both"
        },
        {
          "kv_connector": "OrbitKVConnector",
          "kv_role": "kv_both",
          "kv_connector_module_path": "orbitkv.vllm",
          "kv_connector_extra_config": {
            "orbitkv.mode": "save_only"
          }
        }
      ]
    }
  }'

Valid values are read_write and save_only.

TP shards and host boundary

CUDA IPC is host-local. Workers register with their local Manager, while the scheduler connects only to its own Manager over UDS. Official vLLM 0.31.0 worker handshake metadata carries opaque sealed query targets to the scheduler. Rust validates those targets against the declared topology and performs bounded parallel queries. Python does not query foreign Managers or choose the common prefix.

The experimental profile requires --enable-query-control on every Manager. For one Manager, explicitly set orbitkv.query_control=true; multiple orbitkv.tp_shard_endpoints require native query control automatically. Example:

{
  "kv_connector": "OrbitKVConnector",
  "kv_role": "kv_both",
  "kv_connector_module_path": "orbitkv.vllm",
  "kv_connector_extra_config": {
    "orbitkv.query_control": true,
    "orbitkv.tp_shard_endpoints": [
      "http://127.0.0.1:50055",
      "http://127.0.0.1:50056"
    ]
  }
}

As a configuration example for TP8 and two endpoints, global ranks 0–3 register with the first Manager and 4–7 with the second. This physical topology is not qualified. The scheduler requires exactly all declared global ranks, and all workers in one node must export the identical registration. Every worker receives the same ordered endpoint list; custom local UDS paths can use orbitkv.tp_shard_bootstrap_sockets.

The Manager acquires initial source prefixes in parallel, cancels longer results before reacquiring the exact common prefix, and claims every selected source lease. It retains the source interests until all declared global ranks report successful native Restore completion through official worker metadata. Duplicate rank reports do not count twice. Cancel, replacement revision, unselected result, connection loss and shutdown have bounded ownership paths; TTL does not prove physical process-death reclamation.

The profile accepts equal contiguous dense TP-only shards, one attention cache group and V1. PP, DP, EP, DCP, PCP, MLA replication, hybrid/recurrent recovery, speculative decoding, full-prefix waiting and queue preparation are rejected. Speculative V1 collects worker metadata before deferred finalization, so it cannot consume this completion contract safely. The ordinary single-Manager profile remains the default.

This implementation currently accepts only trusted same-host literal loopback endpoints. Cross-host activation needs authenticated encrypted transport and its own physical topology/fault qualification. Two Manager processes on one GPU do not qualify TP8, multi-GPU NCCL, HA or S3 lifetime. The 1ab67a58 installed wheel is independently accepted for two-Manager DRAM/io_uring byte gates and official vLLM TP=1 eager handshake/cache recovery on H20. Physical multi-GPU serving and composition with the separate adapter/Graph profiles remain unqualified. See the scoped acceptance in the completion plan.

Complete cache blocks and live prompt tails

The cache stores complete engine blocks. Native P/D transfers the live prompt including its partial tail, so the removed orbitkv.pd_tail_save and orbitkv.pd_tail_load options have no replacement cache mode. orbitkv.wait_for_full_prefix remains a remote cache query option; it does not coordinate live P/D or watch saves in a shared local Manager.

Package responsibilities and qualification

Tests live in python/tests/; benchmark programs live in benches/; neither is runtime wheel content. Keep source-only unit tests separate from native and engine gates. Qualify disabled-backend startup, native HBM hits, cold/partial/full misses, DRAM/SSD/peer restores, request cancellation, preemption, restart and advertised eager/graph modes. Record actual transfer bytes and resource drain, not output alone. TP/PP/attention-DP/EP, heterogeneous P/D and containers each need explicit cells. Latest releases are the upgrade reference; unrun cells remain open.

Registered TP query-control transport

The experimental --enable-query-control Manager option enables a bounded query-interest service on --addr for trusted processes on the same host. Both the bound and advertised addresses must be loopback with nonzero ports; invalid endpoints are rejected before CUDA initialization. The transport does not provide local multi-user authentication or encrypted port forwarding. Cross-host activation requires an authenticated encrypted transport and separate qualification. It is disabled by default and does not require etcd. CacheManagerClient.export_query_target() exports opaque sealed-registration authority over authenticated local UDS, for engine handshake metadata. Network interest cannot register/unregister GPU mappings. Explicit Claim retains the same result for reply-loss retries; Close/expiry releases unconsumed shares and leaves accepted native work with its completion owner.

Each interest admits at most 128 operation identities over its lifetime. Terminal identities remain retained for replay fencing, and Claim does not retire interest admission. Consumers must open a fresh interest per lookup and close it after native ownership handoff/completion; idle expiry is a bounded failure fallback.

This transport primitive is independently accepted for the trusted same-host loopback authority and lease scope. The subsequent native common-prefix coordinator and official vLLM handshake are independently accepted at 1ab67a58 for the same-host installed-artifact scope above. Cross-host native TP model, HA and S3 fault qualification remain open. See S5.5 in the completion plan for physical topology and lifetime gates.