Buoyant Releases Kinference: An Open Source Inference Control Plane for Kubernetes and vLLM

Buoyant, the company behind Linkerd, has released Kinference, an Apache 2.0 inference control plane (v0.1.0-alpha.7, written in Rust) that sits in front of vLLM workers on Kubernetes to apply routing policies, KV-cache-aware load balancing and per-tenant metrics, configured through five CRDs. In Buoyant's self-reported benchmark using three L40S GPUs per pool, median TTFT on shared-prefix traffic fell from 6.88 s to 0.31 s versus plain Kubernetes Service routing, while prompts without shared prefixes showed no significant gain (6.92 s vs 6.83 s). The project is in alpha, with rate limiting, classifiers and Anthropic API support not yet shipped.