Google Found a Way to Make Local AI Up to 3x Faster—No New Hardware Required

Google released Multi-Token Prediction (MTP) drafters for its Gemma 4 models, a speculative-decoding serving optimization that can speed up local inference up to 3x without degrading output quality. The approach pairs a tiny, fast “drafter” model with the main Gemma 4 model to predict multiple tokens which the larger model then verifies in one pass. Google’s benchmarks show roughly 2x tokens/second on an Nvidia RTX Pro 6000 (Gemma 4 26B) and ~2.2x on Apple Silicon with batch sizes of 4–8. The drafters are open-sourced under Apache 2.0 and available on Hugging Face, Kaggle and Ollama. Market impact: efficiency gains like this reduce the need for raw hardware, posing competitive pressure on GPU vendors (e.g., Nvidia) while improving the usability of on-device and low-latency AI features — a positive for Google’s AI positioning but potentially negative for hardware-dependent incumbents.