Gemma 4 Drafters Now Boost Local Model Inference Speed

Support for multi-token prediction allows Gemma 4 12B models to generate tokens 137% faster on local hardware.

Updated on Oct. 30, 2026 in Artificial Intelligence

Isometric editorial illustration of sleek, stackable server hardware units with geometric heat sinks, illustrating AI model inference acceleration. — New speculative decoding support for Gemma 4 models on llama.cpp enables faster token generation by using small drafter files to anticipate outputs. AI Illustration. Upload story photo >

Live Poll

Do you plan to optimize your local AI models to increase their generation speed?

In June 2026, llama.cpp added support for multi-token prediction drafter files for Google's Gemma 4 models. This speculative decoding method, originally released by Google in May 2026, accelerates token generation by using a small, 242-megabyte drafter file to anticipate outputs.

Why it matters

Multi-token prediction improves generation efficiency by enabling models to compute multiple tokens per pass, which is particularly effective for accelerating local model inference on consumer hardware. This approach is part of an industry-wide effort to reduce latency for dense LLMs, which derive greater performance benefits from speculative drafting than mixture-of-experts architectures.

Speculative drafting with Gemma 4 12B on an RTX 4070 Ti Super with 16GB VRAM and 32GB system RAM yielded a 137% average speed gain, jumping from 75 to 179 tokens per second. The process utilizes a 242-megabyte drafter file to guess up to 3 tokens ahead, achieving specific speed gains of 167% on JSON tasks and 172% on code tasks.

The players

Google

A global technology company that develops the Gemma family of open-weights language models and related machine learning architectures.

Llama.cpp

An open-source software project that provides high-performance inference libraries for large language models on consumer-grade hardware.

The details

Multi-token prediction works through speculative decoding, where a small, secondary 'drafter' model generates a sequence of potential tokens while the main model simultaneously verifies them. If the main model confirms the drafter's guesses, the system accepts multiple tokens in a single forward pass, significantly reducing the bottleneck of serial token generation. Users enable this by loading the drafter file into llama-server and configuring four specific execution flags to read the main model's cache while checking predictions.

Google released multi-token prediction drafters for Gemma models in May 2026.
Llama.cpp added support for Gemma 4 drafters in June 2026.

The Tech Race

Speculative decoding has emerged as the primary standard for reducing inference latency in local LLM deployments. By implementing this for Gemma 4, developers are moving beyond simple model quantization to structural optimizations that mimic the performance efficiency of proprietary cloud-based APIs.

Users can now implement speculative decoding on consumer hardware to achieve faster response times when running Gemma 4 12B. To benefit, you will need to download the 242-megabyte drafter file from Hugging Face and use the required flags within the latest version of llama-server.

The takeaway

Speculative decoding is rapidly lowering the hardware barrier for running large, dense models at high speeds on home workstations. Monitor the llama.cpp repository for upcoming support for even larger Gemma 4 models to see if these speed gains scale beyond the 12B parameter size.