ORBITKVGitHub

KV cache for vLLM and SGLang

Reuse KV.
Across
cache tiers.

Keep repeated documents, system prompts and conversation prefixes useful beyond GPU memory.
One Cache Manager per host. Your engine keeps control of execution.

Read the docs Get started
Current single-node path
Serving runtimevLLM or SGLang
Engine-owned HBMGPU
PAGESCUDA IPC registration
Cache APIQUERY
& SAVEUDS + iceoryx2 control
CACHE MANAGER.Pinned DRAM · SSD
Remote cache fetch via Mooncake is experimental.

01 / Capabilities

Compute once.
Keep it useful.

Explore the architecture
Reuse

Reuse prefixes.

Keep useful KV beyond one request and restore it when a compatible prefix returns.

Store

Extend capacity.

Offload reusable KV to pinned DRAM and optional SSD. Keep the Manager alive across engine restarts.

Model recipes

Choose your model.

Start with qualified dense Qwen3 profiles. Check engine, precision and state requirements before enabling a hybrid model.

Integrate

Connect both engines.

Enable the vLLM connector or SGLang plugin. Both use direct GPU transfers and the same Rust cache client.

Experimental

Share across replicas.

Local inventory indexes discover compatible replicas. etcd maintains leased membership; Mooncake TENT moves cache bytes.

Native ownership

Own resources in Rust.

Native cache and transfer owners manage budgets, registrations and completion while engines retain scheduling and GPU allocation.

Now Engine-owned HBM · pinned DRAM · optional SSD Experimental Mooncake remote fetch

02 / Deployment

Start on one host.
Build toward a pool.

Single-node caching → independent replicas → prefill/decode reuse

Use official vLLM and SGLang with independent DRAM and SSD caching. Check the selected release and qualified profiles before deployment. Explore shared-cache deployments after validating your single-node workload; multi-node serving remains experimental.

Choose a deployment

03 / Get started

Connect.
Run. Reuse.

Build a matching CUDA wheel, start a manager, then enable your engine's adapter. The first Python release is in preparation.

vLLM quickstart →
SGLang quickstart →
Terminal / Cache Manager
# After installing OrbitKV and your inference engine
orbitkv-cache-manager \
  --addr 127.0.0.1:50055 \
  --pool-size 8gb

04 / Documentation

From first hit
to system design.

Browse all documentation →
Installation and engine setup →Architecture and ownership →Metrics and operation →Benchmarks and measured limits →Distributed cache design →Completion and validation gates →