NVIDIA and Princeton Researchers Optimized LLM Decoding
The new ReplaySSM method accelerates inference by reducing redundant memory writes during token generation.
Updated on Oct. 13, 2026 in Artificial Intelligence
Live Poll
Should developers prioritize new methods that increase the speed and efficiency of AI systems?
NVIDIA and Princeton University have introduced ReplaySSM, a technical method that improves large language model inference by reducing repeated memory writes during token decoding. The researchers released the paper and code for the method on October 12, 2026.
Why it matters
By optimizing how recurrent states are updated and read, ReplaySSM improves throughput in speculative-serving environments, which are essential for scaling modern large language models. The method has already been integrated into production-grade software frameworks like TensorRT-LLM and vLLM.
ReplaySSM achieved a 2.10x increase in speculative-serving throughput at a 512 batch size compared to standard approaches. The method utilizes a 16-input replay window to defer recurrent state checkpoints until the buffer is full.
The players
NVIDIA
A developer of high-performance GPUs and AI-focused software stacks including TensorRT-LLM.
Princeton University
A research university that collaborates on advanced machine learning and hardware optimization projects.
The details
ReplaySSM improves state management by representing a model's recurrent state as a stable checkpoint combined with a temporary recent input buffer. This approach enables the system to separate the precision used for recurrent updates from the precision used for output reads, minimizing redundant memory operations. The method was successfully validated using Nemotron and Qwen models on NVIDIA H100 and Blackwell Ultra B300 hardware architectures.
Timeline
October 9, 2026: The research paper was submitted to arXiv.
October 12, 2026: NVIDIA released the ReplaySSM code and paper.
The Tech Race
This method follows a pattern of kernel-level optimizations designed to extract maximum performance from modern GPU architectures like the Blackwell B300. It competes with other speculative-decoding acceleration techniques by specifically targeting memory write efficiency.
Developers and AI infrastructure engineers can now implement the ReplaySSM method via integration with vLLM, FlashInfer, and SGLang. The technology is already utilized in production serving for the GLM-5.3-Flash and Kimi-K3 models.
The takeaway
ReplaySSM demonstrates that memory write reduction remains a critical frontier for accelerating real-time LLM inference. Watch for future benchmarks in the official GitHub repository or updated performance data in the next release of TensorRT-LLM.
Further reading
For broader context on how researchers are improving model inference speeds, see the Artificial Intelligence section.
Source note: This article includes information reported by TokenPost.
Live Poll
Should developers prioritize new methods that increase the speed and efficiency of AI systems?
