Kimi K3: From Day-0 Availability to Optimized Inference

Kimi K3 combines Kimi Delta Attention, which updates a fixed-size recurrent state, with Multi-head Latent Attention, which maintains a cache that grows with the sequence. These two state systems require different allocation and reuse strategies.

For prefix caching, the serving system must align the recurrent-state checkpoint and the attention cache at the same token boundary. The session also explains how routing follow-up requests to a replica that already holds their prefix can avoid repeating prompt processing.

The team profiled the full token-generation path to identify bottlenecks. At low batch sizes, moving expert weights through GPU memory was a major constraint, so kernel selection needed to reflect the model’s actual weights and workload.

The optimizations included selecting kernels for individual operations, fusing communication with residual additions and normalization, and removing unnecessary copies and type conversions. The speakers emphasize that a faster isolated kernel does not necessarily make the complete serving system faster.

Speculative decoding uses a smaller drafter to propose several tokens, then asks the target model to verify them. It improves speed when drafting and verification together cost less than generating the accepted tokens individually.

Kimi K3's recurrent attention state creates an additional challenge when proposed tokens are rejected. The implementation described stores recurrent inputs and replays the accepted portion from the last committed state. The team evaluates drafters by measured throughput, since longer accepted sequences alone do not guarantee faster inference.

Loading model weights does not allocate every buffer needed during inference. Runtime workspaces, attention caches and CUDA graphs compete for GPU memory, and some allocations happen only when requests arrive.

The team describes a configuration with roughly 1.92 GB of free memory per GPU that later needed a workspace of about 2.5 GB. Startup succeeded, but traffic triggered a failure. This is why readiness checks need representative requests, including demanding prompt lengths and concurrency levels.

The team reports single-stream generation improving from 79 to 120 tokens per second, and aggregate throughput above 2,600 tokens per second under a high-concurrency workload. These measure different operating conditions and should not be treated as interchangeable per-user speeds.

Evaluate throughput alongside P95 and P99 latency, queue time and sustainable concurrency. Benchmark prompts should resemble production traffic: context length, conversation structure and content can affect caching, batching and speculative decoding performance.

Test the complete request path, including the public gateway, routing and streaming. A model can pass checks inside the inference engine while integrations alter or lose information that applications need.

The session highlights tool calls, structured output, reasoning controls, conversation history and multimodal inputs as validation targets. It also explains why autoscaling needs capacity planning: starting a new replica includes loading weights and preparing execution, so it cannot instantly absorb every traffic burst.