Framework integration

Two engines.
One cache manager.

vLLM and SGLang keep control of HBM. Their adapters register GPU KV pages with a node-local Cache Manager for reuse in pinned DRAM and optional SSD.

vLLM

Enable OrbitKVConnector to save computed prefixes and restore reusable blocks into vLLM-owned GPU memory. The pinned release is 0.31.0. Use the V1 model runner and one attention cache group; V2, multi-group and recurrent external recovery are rejected pending safe released lifecycle contracts. Physical multi-rank and cross-host TP remain unqualified. Start with one rank and verify a restore after engine restart.

vLLM install and deployment

SGLang

Enable the OrbitKV RadixCache backend and external linker on SGLang 0.5.22 with public load completion. Start with the Qwen3-8B dense DRAM/io_uring profile in the support matrix. Broader 0.5.21 profiles, complete hybrid recovery and the remaining private lifecycle Hooks retain separate gates. SGLang owns GPU pages; OrbitKV prepares external state and returns a validated restore. DRAM and SSD recovery have single-rank GPU gates. Multi-rank serving requires further qualification.

SGLang install and deployment Hybrid recovery rules and validation SSD measurements and qualification Concurrent budgets and output controls

P/D and native engine integration

Official vLLM uses native NIXL and MultiConnector; official SGLang uses native disaggregation. OrbitKV adds independent cache adapters while Manager shared-cache traffic continues through TENT. The composition candidates have separate model and fault qualification gates. Remaining SGLang cache Hooks are documented as version-coupled internal dependencies.

P/D versus cache sharing

Model-aware reuse

Cache identities bind model artifacts, computation settings and storage geometry. Supported model rules compile to the page ranges needed at a legal recovery boundary, including complete windows and exact checkpoints. This uses engine-declared state requirements; arbitrary graph analysis and future-token prediction remain future work.

Checkpoint qualification records the exact engine, precision and recovery path. Larger Qwen, GLM, DeepSeek and Kimi targets have separate gates; sparse indexer state and an engine's ability to load a model do not imply a complete OrbitKV restore.

Pretrained models and qualification limits State identity plan and gates

Tune and deploy

Begin with ordinary demand and size DRAM/SSD for your working set. Optional request preparation has three matched Qwen3-8B controls per engine: vLLM improves modestly, while SGLang gains throughput with worse P95 latency. It stays disabled by default. Scoped two-H20 RDMA dense recovery and controlled restarts have acceptance. Native P/D faults, distributed HA, isolation and broader topology remain separate qualifications.

Read the implementation gates Request preparation measurements