Preface¶

In 2026, large model inference is no longer a simple matter of “spinning up a Deployment and exposing a Service”. The distinct characteristics of the Prefill and Decode stages, dynamic KV Cache occupancy changing with context length, and the coexistence of multiple models and LoRA adapters—these realities have left traditional Round-Robin load balancers trapped in the dilemma of “some GPUs sitting idle while others queue requests” in GPU clusters.

The cloud native community has a clear direction: treat LLM inference as a first-class citizen for Kubernetes scheduling. The Volcano community under CNCF launched Kthena in January 2026; NVIDIA open-sourced Grove in November 2025, which complements KAI Scheduler. Meanwhile, the CNCF Sandbox project KAITO continues to evolve. The three projects each occupy their niche in the layers of “model deployment — traffic routing — GPU gang scheduling”, finally giving platform engineering teams the chance to build a unified K8s AI infrastructure layer using declarative YAML.

This article cross-references official blogs and GitHub repositories to sort out what each part of this ecosystem solves, how they collaborate, and the details worth paying attention to during implementation.

Why is LLM Inference Scheduling on K8s So Difficult¶

Before diving into specific projects, let’s clarify the pain points. According to CNCF’s introduction to Kthena, production-grade LLM Serving faces at least four types of challenges:

1. Low resource utilization: The dynamic memory footprint of KV Cache causes significant fluctuations in GPU/NPU pressure. Traditional load balancers cannot perceive these characteristics, easily leading to a coexistence of resource fragmentation and request queuing.
2. Trade-off between latency and throughput: Inference is divided into two stages: Prefill (compute-intensive) and Decode (memory-bandwidth-limited). Coupled scheduling makes it difficult to optimize both phases separately; the industry generally adopts Prefill-Decode (PD) separation, but routing and scheduling remain complex.
3. Difficult multi-model management: Enterprises often serve multiple model versions and LoRA adapters simultaneously. Fair scheduling, priorities, and dynamic routing are difficult to unify in a single gateway.
4. Disconnected from the K8s ecosystem: Many solutions are independent systems, requiring operations teams to maintain an additional control plane.

These four issues correspond exactly to the core battlefields of Kthena (routing and orchestration), Grove (multi-component lifecycle), and KAI Scheduler (GPU gang and topology-aware scheduling).

Kthena: The “Intelligent Brain” of Inference in the Volcano Ecosystem¶

Kthena is a sub-project of Volcano, positioned as a Kubernetes-native LLM inference routing, orchestration, and scheduling system. It does not replace inference engines like vLLM or SGLang, but instead provides an intelligent orchestration layer on top of these engines, deeply integrated with K8s.

Architecture: Separation of Control Plane and Data Plane¶

Kthena consists of two core components:
- Kthena Router: A high-performance multi-model router that acts as the entry point for all inference requests, distributing traffic to backend ModelServers according to ModelRoute rules.
- Kthena Controller Manager: The control plane that coordinates workload lifecycle via CRDs, responsible for ServingGroup, Prefill/Decode role orchestration, topology-aware affinity, gang scheduling, rolling updates, and failure recovery.

The main CRDs include ModelServing (hierarchical workloads), ModelBooster (out-of-the-box templates), and AutoScalingPolicy (elastic scaling strategies).

Hierarchical Workloads: ModelServing → ServingGroup → Role¶

Kthena introduces a hierarchical workload model. A complex PD-separated deployment can be declared as a single ModelServing resource, which contains multiple ServingGroups internally, and each Group is further subdivided into roles such as Prefill and Decode. Gang scheduling ensures that Pods within the same ServingGroup are scheduled as an atomic unit, avoiding deadlocks caused by “only Prefill Pods are started while Decode Pods remain pending”.

The core logic of PD separation is: compute-intensive Prefill tasks are routed to nodes with stronger computing power, while memory-bandwidth-limited Decode tasks are routed to nodes with more abundant HBM. The two can scale independently, dynamically adjusting the Prefill/Decode ratio.

KV Cache-Aware Routing¶

This is one of Kthena’s most distinctive capabilities. The Router provides pluggable scheduling plugins, including Least Request, Least Latency, KV Cache Awareness, Prefix Cache Awareness, LoRA Affinity, and Fairness Scheduling.

The working principle of the KV Cache Awareness plugin, according to Kthena’s official documentation:
- A Kthena Runtime sidecar is deployed alongside each vLLM Pod, subscribing to vLLM’s ZMQ kv-events stream and writing token block hashes to Redis.
- When a request arrives, the Router queries Redis and prioritizes routing the request to Pods that already have matching KV Cache, reducing repeated Prefill computations.

In a scenario with a 4096-token system prompt, benchmark tests published in the CNCF blog show that “Least Request + KVCacheAware” achieves a ~2.73x throughput increase, a ~73.5% reduction in TTFT (Time to First Token), and a over 60% reduction in end-to-end latency compared to random routing. The gap narrows in short prompt scenarios, but KV Cache awareness still has a decisive advantage in multi-turn conversation and templated Prompt workloads.

Supported Inference Engines and KV Transport¶

Kthena supports engines including vLLM, SGLang, Triton/TGI via a unified API abstraction. In PD separation mode, KV Cache state needs to be transferred between Prefill Pods and Decode Pods. Kthena supports three connector backends:

The application layer does not need to perceive the specific transport implementation, and the Runtime Agent will handle it automatically.

Quick Experience: ModelBooster Templates¶

For common models, Kthena provides ModelBooster templates that build in mainstream deployment modes such as PD separation, automatically generating routing and lifecycle resources. Complex scenarios can then use ModelServing for fine-grained control.

GitHub repository: volcano-sh/kthena
Official documentation: kthena.volcano.sh

Grove: Describe an Entire Inference System with One CR¶

If Kthena focuses on “how requests are routed and how models are orchestrated”, NVIDIA Grove focuses on “how a multi-component inference system can be scaled, scheduled, and recovered as a single unified entity”.

Grove was released as part of NVIDIA Dynamo modularity in November 2025, fully open-sourced, with its repository located at ai-dynamo/grove (there is also a mirror under the NVIDIA official GitHub organization).

From “N Pod Replicas” to “Collaboration of a Set of Components”¶

Modern inference deployments often include multiple roles such as Prefill, Decode, Vision Encoder, and KV Router. Agentic pipelines require collaboration between multiple model instances. Grove’s design motivation is to extract the work of “coordinating a set of components as a logical system” from scripts and custom Controllers, allowing it to be fully declared via a single Custom Resource.

Grove provides three levels of hierarchical CRs:
- PodClique: A Pod group with specific roles (such as prefill-leader, prefill-worker, decode-leader, decode-worker), each with independent configuration and scaling logic.
- PodCliqueScalingGroup: A tightly coupled combination of PodCliques that must scale together, for example, a prefill leader and its workers form a single model instance.
- PodCliqueSet: The top-level workload definition that specifies startup order, scaling policies, and gang scheduling constraints, ensuring all components start together or fail together.

Hierarchical Gang Scheduling and Topology Awareness¶

Traditional gang scheduling is “all or nothing”, but PD separation scenarios require more flexible strategies: at least one Prefill instance and one Decode instance must be ready at the same time, while allowing Prefill and Decode to scale independently at different ratios.

Grove generates PodGang resources (part of the Scheduler API) to translate workload requirements into scheduling constraints. Each PodGang encapsulates minimum replica guarantees, network topology packing preferences, and spread constraints. On architectures such as GB200 NVL72, scheduling relevant Prefill and Decode Pods to the same NVLink domain can significantly reduce KV Cache transfer latency.

Grove also implements multi-level automatic scaling: single component level, component group level, and entire service replica level, with the three scaling tiers aware of each other—for example, scaling Prefill workers may trigger adjustments to Decode capacity.

Collaboration with KAI Scheduler¶

Grove itself does not handle Pod placement; it requires a scheduler in the cluster that understands PodGang resources. KAI Scheduler was built for this exact purpose: it continuously listens for PodGang resources, applies gang scheduling logic, and makes placement decisions under GPU topology awareness and cluster locality constraints. Grove defines “what is needed”, while KAI decides “where to place it”.

KAI Scheduler: The Gang Scheduling Engine for GPU Clusters¶

KAI Scheduler is an open-source Kubernetes-native AI scheduler from NVIDIA (licensed under Apache 2.0), originating from the Run:ai platform, targeting large-scale GPU clusters.

Its core capabilities include:
- Gang Scheduling: Via the PodGroup CRD, ensuring a set of interdependent Pods are either all scheduled successfully or all remain pending, avoiding distributed tasks starting partially and leaving GPUs idle.
- Hierarchical PodGroups: Supporting nested SubGroups, allowing topology constraints and gang thresholds to be specified separately for multi-layer components such as Prefill/Decode, adapting to complex pipelines like Grove/Dynamo.
- Topology-Aware Scheduling (TAS): Optimizing placement based on physical topology such as racks and NVLink domains.
- Fair Queuing and Priority Preemption: Allocating GPUs by quota in multi-tenant scenarios, allowing high-priority inference tasks to preempt low-priority batch jobs.

In November 2025, KAI Scheduler completed integration with Grove; KAI’s PodGrouper plugin can directly consume PodGangs generated by Grove, map them to KAI PodGroups, and set fields such as MinAvailable.

GitHub repository: NVIDIA/KAI-Scheduler

KAITO: Model Deployment Operator in CNCF¶

Beyond “scheduling and routing”, KAITO (Kubernetes AI Toolchain Operator) is responsible for deploying models themselves into the cluster. It entered the CNCF Sandbox in October 2024, with the latest version v0.11.0 released in July 2026.

KAITO simplifies model deployment via the Workspace CRD: specify a HuggingFace model ID, and the Operator will automatically estimate GPU memory requirements, select appropriate node sizes, and integrate with node auto-provisioners such as Karpenter. Currently, vLLM is the primary supported inference engine, with LoRA adapter support and KV Cache offloading enabled by default.

Important updates in the 2026 v0.11.0 release include:
- MultiRoleInference: Native support for Prefill/Decode separated deployment.
- InferenceSet promoted to v1beta1: More mature replica management and auto-scaling, with integration with KEDA.
- ProductionStack optional sub-chart: Based on the llm-d reference stack, integrating Istio gateways, KV-cache-aware routing, and KEDA metric-driven scaling.

KAITO also creates InferencePool and EPP (Endpoint Picker) via the Gateway API Inference Extension, enabling integration with external gateways for KV Cache-aware routing.

GitHub repository: kaito-project/kaito

Ecosystem Layering: Who Manages What¶

When the above projects are organized into a logical diagram, the boundaries of responsibilities become clear:

There is overlap in capabilities between Kthena and Grove in PD separation, gang scheduling, and topology awareness, but their starting points differ: Kthena builds on Volcano’s batch scheduling experience, emphasizing inference routing and a unified ModelServing API; Grove originates from NVIDIA Dynamo’s multi-component inference architecture, emphasizing one-stop declaration via PodCliqueSet and deep integration with KAI Scheduler. Platform teams can choose based on their existing tech stack, or combine layers—for example, KAITO for model deployment, Kthena for traffic governance, and KAI Scheduler for GPU placement.

Hands-On Try: Minimal Kthena PD Separation Example¶

The following YAML snippet demonstrates the basic structure of a Kthena ModelServing declaring a PD-separated deployment (field names refer to official documentation, please cross-check against the current version CRD before actual deployment):

apiVersion: inference.volcano.sh/v1alpha1
kind: ModelServing
metadata:
  name: llama-pd-serving
spec:
  model: meta-llama/Llama-3.1-8B-Instruct
  engine: vllm
  servingGroups:
    - name: pd-group-0
      roles:
        - name: prefill
          replicas: 2
          resources:
            limits:
              nvidia.com/gpu: "1"
          template:
            spec:
              containers:
                - name: vllm
                  image: vllm/vllm-openai:latest
                  args:
                    - --kv-events-config
                    - '{"enable_kv_cache_events":true,"topic":"kv-events"}'
        - name: decode
          replicas: 4
          resources:
            limits:
              nvidia.com/gpu: "1"

Combined with the Kthena Router’s kvcache-aware plugin and Redis coordination layer, it can deliver considerable TTFT improvements in long-context scenarios. Refer to the official repository’s gpu-kvcache-aware.yaml for the complete example.

For Grove, users only need to submit a single PodCliqueSet, and the Operator will automatically generate PodCliques, PodCliqueScalingGroups, Services, HPAs, and PodGangs, with subsequent scheduling handled by KAI Scheduler. See the NVIDIA Grove technical blog for specific Helm installation steps.

Final Thoughts¶

The K8s AI inference ecosystem in 2026 is moving from “teams piecing together YAML manually” to “layered components and declarative collaboration”. Kthena brings KV Cache-aware routing and PD separation scheduling into the Volcano unified framework; Grove and KAI Scheduler provide topology-aware gang scheduling capabilities for multi-node, multi-component inference; KAITO continues to lower the barrier to model deployment.

For platform engineering teams, the key is not “which one to choose”, but “at which layer to introduce which capability”—the routing layer, orchestration layer, and scheduling layer each perform their own duties,才能真正将GPU利用率、TTFT和多模型治理做到生产可用。All the above projects are open source, so it is recommended to start with official documentation and GitHub sample clusters for validation, then gradually replace existing manually orchestrated inference stacks.