New AI Scheduler Reduces LLM Latency and Recomputes
The research-stage tool uses reinforcement learning to dynamically manage memory footprints and improve inference throughput.
Updated on Oct. 26, 2026 in Artificial Intelligence
Live Poll
Do you believe new AI scheduling technology will make your online services more reliable?
Researchers have developed URSA, an uncertainty-aware reinforcement learning scheduler designed to optimize Large Language Model (LLM) inference serving. The research-stage system uses offline reinforcement learning to predict output lengths and memory requirements, aiming to improve system performance.
Why it matters
Current inference schedulers rely on static heuristics that struggle to manage the unpredictable memory footprint of LLM outputs. URSA addresses this by dynamically reserving memory, which reduces costly recompute and eviction events in production GPU environments.
URSA utilizes a graph neural network to analyze the joint state of request queues, GPU availability, and KV cache occupancy. By training on historical trace data from Azure and Alibaba, the scheduler reserves memory at a high-quantile prediction of the expected footprint.
The players
Azure
The cloud computing subsidiary of Microsoft that provided LLM inference traces for the URSA study.
Alibaba
A multinational technology conglomerate whose GPU-cluster traces were used to evaluate URSA performance.
The details
URSA—Uncertainty-aware Reinforcement learning Scheduler for LLM serving—replaces rigid scheduling logic with a policy learned through distributional offline reinforcement learning (RL). A Graph Neural Network (GNN)—a model architecture designed to process data structured as nodes and edges—evaluates the current cluster state to predict request output lengths. By reserving memory based on a high quantile of this prediction, URSA minimizes the recompute and eviction events that typically plague inference pipelines when static memory limits are breached.
Timeline
October 26, 2026: Article publication.
The Tech Race
URSA marks a shift from static, rule-based scheduling toward adaptive intelligence in AI infrastructure. By utilizing historical traces like those from BurstGPT, the researchers move closer to solving the memory-efficiency bottleneck currently limiting high-throughput LLM inference.
Engineers and infrastructure teams managing LLM inference workloads may see reduced latency and fewer GPU memory errors once this scheduler is integrated into production stacks. As this remains in the research stage, practitioners should monitor future GitHub releases or library implementations for integration.
The takeaway
URSA demonstrates that offline reinforcement learning can successfully manage the variable memory demands of LLM inference serving. Technical teams should watch for the release of open-source implementations to test whether these latency improvements hold across their specific infrastructure setups.
Further reading
For a broader look at the challenges facing model deployment, explore our coverage in Artificial Intelligence.
More information
Read the complete peer-reviewed research article to review the methodology.
Source note: This article includes information reported by Nature.
Live Poll
Do you believe new AI scheduling technology will make your online services more reliable?
