ORBITKVGitHub

Architecture / Ownership before placement

One cache API.
Across tiers.

The engine owns GPU memory and model execution. OrbitKV owns external cache replicas, finds reusable prefixes, and holds pages until transfers finish.

From one host to a shared cache

Engine-owned HBM and raw DRAM Restore; Manager-owned Publish, SSD and codecs. Peer grants control TENT READs; inventory streams feed local indexes and etcd holds membership. Native engine backends own P/D.
Single-node recovery is validated. The multi-node path is experimental; each Manager maintains explicitly covered local inventory views. A shared API does not imply interchangeable KV layouts. Declared page demand is implemented; general lifetime and physical planning remain future work.
01 / Single node

Keep useful prefixes.

Run an independent Manager per node and connect engines to its shared DRAM and optional SSD cache. Engines retain HBM ownership and execute raw DRAM restores through imported payload arenas. Manager workers retain Publish, SSD and encoded restores. Standalone deployment needs neither etcd nor peer gRPC. Configure the SSD path and capacity; automatic selection tries native cuFile and falls back to io_uring without engine-side backend settings. Rust submits cuFile reads and writes through two bounded GPU slots, allowing demand reads to progress alongside writeback. Native GDS performance and multi-engine container qualification remain separate.

02 / Independent replicas

Reuse across workers.

Managers discover compatible replicas in complete local global indexes, synchronized through bounded inventory streams. etcd maintains leased membership; peer gRPC authorizes exact source replicas; TENT READ moves their bytes into the consumer Manager's host memory before GPU restore. Scoped dense RDMA serving passes; HA and broader topology remain separate gates.

03 / Prefill & decode

Compose the handoff.

Official vLLM NIXL/MultiConnector and SGLang native disaggregation own current-request handoff and transport. OrbitKV supplies independent cache adapters. Cache reuse has a separate source grant and completion contract.

How a request reuses KV

  1. Compile

    The adapter declares prefix, window and checkpoint rules. The shared contract turns them into page requirements for known token ranges from a valid HBM origin.

  2. Prepare

    The manager finds DRAM, SSD or remote candidates within query byte budgets. The same compiled requirements validate leased evidence and yield complete recovery boundaries across ranks.

  3. Restore

    Compiled ranges validate the state declared by a supported adapter. External hybrid model profiles need separate lifecycle qualification. Leases hold pages through transfer completion.

  4. Continue

    The engine resumes computation; completed KV can be published for later reuse.

The vLLM adapter defers new admissions until an admitted restore reaches computation, avoiding deferred-queue capacity stalls. SGLang waits for leased readiness before allocating restore destinations. Each retains its own scheduler and HBM allocator.

Shared page requirements remove duplicate adapter arithmetic. They do not predict future tokens, analyze arbitrary model graphs or authorize reclamation. Automatic hybrid warming remains disabled. Synthetic GPU-byte gates validate declared layouts; complete released hybrid model recovery and its latency benefit remain unqualified. Ordinary recovery and request preparation have separate Qwen3-8B performance reports.

Read the architecture and ownership contracts →

Implementation boundaries

orbitkv-state

Versioned model/storage keys and compiled rules for demanded page ranges and leased-evidence validation, consumed by engine adapters within their qualified model profiles.

orbitkv-channel

Rust ownership of query revisions, warming and transfer waits; versioned iceoryx2 requests plus UDS lifecycle for both adapters.

orbitkv-core

KV blocks, leases, tiers, batched GPU codecs with reusable workspace, CPU SIMD, and encoded cuFile storage.

orbitkv-catalog

Scoped local inventory indexes and member admission; the server owns inventory streams and leased etcd membership.

orbitkv-transfer

Pinned Mooncake TENT with registered-region ownership and terminal batch drain for shared-cache transfers between Managers.

Cache Manager

Independent per-node service sharing external DRAM/SSD capacity across attached engines, with automatic SSD selection, health and peer transfer control.

orbitkv.sglang

Official UnifiedRadixCache external linker with validated selected-range reads; public lifecycle and complete hybrid model recovery remain open.

orbitkv.vllm

Independent cache connector; native NIXL and MultiConnector handle candidate P/D composition.

What comes next

Rust records bounded, opt-in route costs. Experimental selectors cover matching peer owners and equal-coverage host routes; a unified optimizer across cache restoration and P/D handoff remains planned. Bounded raw-plan partitioning and per-layer CUDA dependencies pass single-GPU gates, including graph replay. Manager codec/SSD overlap and earlier publication across fragmented parts remain execution work.

Bounded native P/D model controls do not close transfer-fault gates. Cross-host P/D and independent failure domains still need qualification. Single-node preparation and stopping controls are implemented and stay opt-in after the first repeated measurements. Global-index scale, metadata churn and cross-host failure behavior require qualification before production claims. General lifetime analysis, cross-engine byte conversion and KV-aware routing remain future work.

Implementation plan and upstream references →Transfer costs and deployment policies →Distributed index design →Model-aware identity →LMCache and Mooncake deployment paths →Completion and qualification gates →