認識 EAGLE 3.1：修復大型語言模型推理中注意力漂移的推測解碼演算法

Speculative decoding is a technique for speeding up large language model inference. A small, fast draft model proposes several tokens. The large target model verifies them in parallel. If accepted, inference is faster. If rejected, the system falls back gracefully. EAGLE Team, vLLM Team, and TorchSpec Team has launched the EAGLE series including EAGLE 1, EAGLE 2, and EAGLE 3 has become one of the most widely adopted and practically deployed families of speculative decoding algorithms across both research and production systems. Today, that family gets a targeted reliability upgrade with introduction of EAGLE 3.1. What was Going Wrong While speculative decoding performs well in controlled settings, performance often degrades under different chat templates, long-context inputs, or out-of-distribution system prompts. The EAGLE team traced this fragility to a phenomenon called attention drift as speculation depth increases, the drafter gradually shifts attention away from sink tokens and toward its own generated tokens. In simpler terms: the drafter is a small model that predicts future tokens. As speculation gets deeper, it starts attending to its own prior outputs instead of the original context. This degrades acceptance length and output stability. Two underlying issues were identified. First, the fused input representation becomes increasingly imbalanced as higher-layer hidden states dominate the drafter input. Second, hidden-state magnitude grows across speculation steps due to the unnormalized residual path. Together, these effects make the drafter progressively less stable at deeper speculation depths. Two Architectural Fixes in EAGLE 3.1 To address attention drift, EAGLE 3.1 comes with two key architectural improvements: FC normalization after each target hidden state and before the FC layer, and feeding post-norm hidden states into the next decoding step. FC normalization stabilizes the hidden states that the drafter receives from the target model. Without it, hidden-state magnitude grows across steps, making the drafter increasingly unreliable. Applying normalization at each step keeps the inputs bounded. The post-norm design makes the method behave more like recursively invoking the drafter across decoding steps, rather than simply appending additional layers to the target model. https://vllm.ai/blog/2026-05-26-eagle-3-1 What These Fixes Deliver Compared with EAGLE 3, EAGLE 3.1 demonstrates: better training-time to inference-time extrapolation, stronger long-context robustness, higher resilience to chat template and system prompt variation, and more stable acceptance length across diverse serving environments. In long-context workloads, EAGLE 3.1 achieves up to 2× longer acceptance length compared with EAGLE 3. Training Infrastructure: TorchSpec TorchSpec now provides efficient training support for EAGLE 3.1 and future speculative decoding algorithms. By lowering training overhead and simplifying experimentation workflows, TorchSpec helps accelerate iteration and exploration for next-generation speculative decoding research and deployment. Based on TorchSpec and vLLM, the research team also trained and open-sourced an EAGLE 3.1 draft model for Kimi K2.6, available on HuggingFace. The model serves as an example of deploying EAGLE 3.1 with TorchSpec training and vLLM serving support on a real-world serving model vLLM Integration: Config-Driven and Backward-Compatible EAGLE 3.1 lands in vLLM as a config-driven extension of the existing EAGLE 3 implementation. The integration includes FC normalization support, post-norm hidden-state feedback, and removal of hardcoded assumptions around target hidden states. Backward compatibility with existing EAGLE 3 checkpoints is fully preserved. EAGLE 3.1 draft models can be plugged directly through the same speculative-decoding code path. Copy CodeCopiedUse a different Browservllm serve nvidia/Kimi-K2.6-NVFP4 \ --trust-remote-code \ --tensor-parallel-size 4 \ --tool-call-parser kimi_k2 \ --enable-auto-tool-choice \ --reasoning-parser kimi_k2 \ --attention-backend tokenspeed_mla \ --speculative-config '{"model":"lightseekorg/kimi-k2.6-eagle3.1-mla","method":"eagle3","num_speculative_tokens":3}' \ --language-model-only Benchmark Results on Kimi K2.6 The research team benchmarked the Kimi K2.6 EAGLE 3.1 draft model on Kimi-K2.6-NVFP4 with vLLM (TP=4, GB200, non-disagg) on the SPEED-Bench coding dataset. EAGLE 3.1 delivers 2.03× higher per-user output throughput at concurrency 1. The speedup stays meaningful as concurrency scales: 1.71× at C=4 and 1.66× at C=16. Marktechpost’s Visual Explainer #mtp-eagle31,#mtp-eagle31 *{box-sizing:border-box!important;margin:0!important;padding:0!important;font-family:'Courier New',Courier,monospace!important} #mtp-eagle31{width:100%!important;max-width:780px!important;margin:0 auto!important;background:#0c0c0c!important;border:1px solid #444!important;position:relative!important;overflow:hidden!important;display:block!important} #mtp-eagle31 .mtp-track{display:flex!important;flex-wrap:nowrap!important;transition:transform .45s cubic-bezier(.77,0,.18,1)!important} #mtp-eagle31 .mtp-slide{width:100%!important;flex-shrink:0!important;flex-grow:0!important;padding:44px 52px 36px!important;background:#0c0c0c!important;position:relative!important;overflow:hidden!important} #mtp-eagle31 .mtp-inner{min-height:280px!important;display:flex!important;flex-direction:column!important;justify-content:center!important} #mtp-eagle31 .mtp-tag{display:inline-block!important;font-size:10px!important;letter-spacing:.18em!important;text-transform:uppercase!important;color:#aaa!important;border:1px solid #444!important;padding:4px 10px!important;margin-bottom:18px!important;border-radius:2px!important;width:auto!important} #mtp-eagle31 .mtp-h1{font-size:28px!important;font-weight:700!important;color:#ffffff!important;line-height:1.25!important;margin-bottom:14px!important;letter-spacing:-.02em!important} #mtp-eagle31 .mtp-h2{font-size:20px!important;font-weight:700!important;color:#ffffff!important;line-height:1.3!important;margin-bottom:18px!important;letter-spacing:-.01em!important} #mtp-eagle31 .mtp-divider{width:28px!important;height:1px!important;background:#444!important;margin-bottom:20px!important;display:block!important} #mtp-eagle31 .mtp-sub{font-size:14px!important;color:#d0d0d0!important;line-height:1.75!important;margin-bottom:10px!important;display:block!important} #mtp-eagle31 .mtp-list{list-style:none!important;margin-top:10px!important;display:block!important;width:100%!important} #mtp-eagle31 .mtp-list li{font-size:14px!important;color:#d0d0d0!important;line-height:1.7!important;padding:9px 0 9px 22px!important;border-bottom:1px solid #222!important;position:relative!important;display:block!important;width:100%!important} #mtp-eagle31 .mtp-list li:last-child{border-bottom:none!important} #mtp-eagle31 .mtp-list li::before{content:'\2192'!important;position:absolute!important;left:0!important;top:9px!important;color:#aaa!important;font-size:13px!important} #mtp-eagle31 .mtp-hi{color:#ffffff!important;font-weight:700!important} #mtp-eagle31 .mtp-metric{display:grid!important;grid-template-columns:repeat(3,1fr)!important;gap:10px!important;margin-top:8px!important;width:100%!important} #mtp-eagle31 .mtp-card{background:#181818!important;border:1px solid #444!important;padding:18px 10px!important;border-radius:4px!important;text-align:center!important;display:block!important} #mtp-eagle31 .mtp-num{font-size:26px!important;font-weight:700!important;color:#ffffff!important;display:block!important;letter-spacing:-.03em!important;margin-bottom:6px!important} #mtp-eagle31 .mtp-label{font-size:11px!important;color:#aaa!important;text-transform:uppercase!important;letter-spacing:.1em!important;display:block!important;line-height:1.4!important} #mtp-eagle31 .mtp-code{background:#141414!important;border:1px solid #444!important;border-radius:4px!important;padding:16px 18px!important;margin-top