Your Experts Have Partners. There's HOPE for Keeping Them Together

Back to Articles Your Experts Have Partners.

There's HOPE for Keeping Them Together Enterprise Article Published November 9, 2026 Upvote - Alex M Tseng alexmtseng Follow amazon Higher-Order Pruning of Experts in Mixture-of-Experts Language Models Introduction Mixture-of-Experts (MoE) models have enabled state-of-the-art LLMs to be scaled to hundreds of billions of parameters, but they come with a fundamental and costly caveat: every expert MLP must occupy memory at inference time, even though the vast majority are idle for any given token.

For example, Qwen3.5-122B has 256 experts in each of its 48 MoE layers, and 95% of its ~250 GB parameter memory footprint is taken up entirely by these expert MLPs alone.However, for any given token, only 8 experts in each layer are actually used.

Expert pruning is one of the most direct fixes to this memory bottleneck.That is, permanently delete the least useful experts.Pruning avoids the latency issues which make dynamic loading/off-loading of experts impractical; it does not require any training; and it composes with quantization.

Given these attractive benefits and the simplicity of the idea, it's easy to see why expert pruning has become a go-to approach for shrinking MoEs.The difficulty lies in deciding which experts are safest to remove.

Existing pruning methods score each expert independently, assigning every expert a scalar importance score (based on a calibration dataset), and cutting out the least important experts (in each layer independently).

These are first-order methods, because they assume each expert's contribution is independent of all others.In reality, experts form cooperative groups.

We observe consistent clusters of co-activated experts in our own experiments, and independent work has shown that expert cooperation is a stable property of MoEs and is important for model performance.Ignoring these interactions leads to suboptimal pruning, especially at aggressive rates.

Our newest paper derives HOPE (Higher-Order Pruning of Experts), a second-order method that accounts for pairwise expert interactions.At 50% pruning, HOPE achieves the best average rank of any method, with gains of up to +6.1% on agentic coding tasks over the next-best pruning method.

In this post, we explain how HOPE works, show where it outperforms existing methods, and walk through how to use it for your own models.For readers unfamiliar with MoE architectures, see Mixture of Experts (MoEs) in Transformers.

Experts don't work alone In a standard MoE, each layer has EEE experts, and each input token xxx is routed independently to KKK such experts ( K=8 K = 8 K=8 is a typical choice).

Routing is performed by a single layer which produces softmax-normalized gate weights gk(x)g{k}(x)gk​(x) for each expert kkk, wherein the KKK experts with the highest gate weights are selected to form the active set of experts, T(x)⊂{1,..,E}T(x) \subset\{1,..,E\}T(x)⊂{1,..,E}.

The output of the layer is then ∑k∈T(x)gk(x)fk(x)\sum\limits{k\in T(x)}g{k}(x)f{k}(x)k∈T(x)∑​gk​(x)fk​(x), where fk(x)f{k}(x)fk​(x) is the output of expert kkk for token xxx.Left: typical MoE layer, where each token is routed to the top-K experts.

Right: first-order methods rank and delete experts based on a scalar importance score, but the best pruning decisions will also consider the strength of interactions between experts.Importantly, experts in the selected set T(x)T(x)T(x) are not independent.

Experts form stable cooperative groups that get consistently co-activated together.This has been observed before in the literature, as well: expert co-selection patterns emerge early in training and stabilize throughout.

This means that when we prune experts from an MoE layer, pruning decisions need to consider the higher-order effects between these experts.

Removing expert A might seem fine on its own, but if expert B—which frequently cooperates with and relies on A—is highly important, then removing A would dismantle a cooperative group just through a single deletion.First-order methods miss this entirely, because they score A and B separately.

This effect compounds at high pruning rates.When we remove up to 50% of experts, we risk breaking many cooperative links.This is where second-order pruning is most advantageous.

How HOPE identifies a prune-set (See Appendix A in our corresponding paper for the full derivation of the HOPE objective) When we delete a set of PPP experts from some MoE layer, the selected set of experts T(x)T(x)T(x) (for any token xxx) can change.

The pruned experts that would have been selected are replaced by the experts with the next-highest gate weights.The resulting change in layer output is the pruning error.

Our goal is to find the prune-set PPP (of some predefined pruning budget ∣P∣\vert P\vert∣P∣) that minimizes the resulting pruning error (over tokens xxx from our calibration set).Unfortunately, minimizing the pruning error over PPP directly is intractable.

The pruning error depends on the identities of the replacement experts, which in turn depend on the prune-set PPP itself.

HOPE sidesteps this by defining a tractable surrogate ZZZ, which strictly upper-bounds the dominant component of the pruning error: Z=[∑j∈P∩T(x)gj(x)∥fj(x)∥2]2≥true errorZ = \bigl[\sum\limits{j \in P\cap T(x)} g{j}(x)\Vert fj(x)\Vert{2}\bigr]^{2} \geq \text{true error}Z=[j∈P∩T(x)∑​gj​(x)∥fj​(x)∥2​]2≥true error Because ZZZ is a squared sum, expanding it naturally produces cross-terms gi(x)gj(x)∥fi(x)∥2∥fj(x)∥2g{i}(x)g{j}(x)\Vert f{i}(x)\Vert{2}\Vert f{j}(x)\Vert{2}gi​(x)gj​(x)∥fi​(x)∥2​∥fj​(x)∥2​ between pairs of experts.

These are the interaction terms that first-order methods discard.To find the optimal prune-set, HOPE minimizes E[Z]\mathbb{E}[Z]E[Z] over the calibration set, which can be reformulated as the following quadratic program: P∗=arg min⁡p p⊤Fps.t.

p∈{0,1}E,∑k=1Epk=∣P∣P^{*} = \argmin{p}\; p^{\top}Fp \quad \text{s.t.} \quad p \in \{0,1\}^{E}, \sum\limits{k=1}^{E}p{k} = \vert P\vertP∗=pargmin​p⊤Fps.t.

p∈{0,1}E,k=1∑E​pk​=∣P∣ where F∈RE×EF\in\mathbb{R}^{E\times E}F∈RE×E is defined as: Fi,j=1∣Xi,j∣∑x∈Xi,jgi(x)gj(x)∥fi(x)∥2∥fj(x)∥2whereXi,j={x:i,j∈T(x)}F{i,j} = \frac{1}{\vert \mathcal{X}{i,j}\vert} \sum\limits{x\in\mathcal{X}{i,j}} g{i}(x)g{j}(x)\Vert f{i}(x)\Vert{2}\Vert f{j}(x)\Vert{2}\quad\text{where}\quad \mathcal{X}{i,j} = \{x: i,j \in T(x)\}Fi,j​=∣Xi,j​∣1​x∈Xi,j​∑​gi​(x)gj​(x)∥fi​(x)∥2​∥fj​(x)∥2​whereXi,j​={x:i,j∈T(x)} This F-matrix is collected on the calibration dataset, and the optimal prune-set is obtained by solving the resulting quadratic program (QP) with a continuous relaxation (then rounding to binary solutions).

Each entry Fi,jF{i,j}Fi,j​ captures the average gate-weighted output product of experts iii and jjj, across tokens where both are co-activated.

Whereas the diagonal Fk,kF{k,k}Fk,k​ measures individual expert importance, the off-diagonal entries ( Fi,jF{i,j}Fi,j​ for i≠ji\neq ji=j) measure pairwise cooperative contribution.What does FFF look like in practice?

It shows the structure of expert cooperation, which depends on layer: Visualization of the interaction matrix F for GLM-4.5-Air (128 experts per layer, 106B parameters total) at three representative layers (early, middle, late).Experts have been clustered to reveal block structure.

Bright blocks along the diagonal indicate groups of experts with strong pairwise interaction scores.In terms of practical cost, HOPE requires the same calibration pass as any other expert-pruning method (one forward pass over calibration data).

The only addition is solving the quadratic program, which is non-convex, but can be solved efficiently via a standard QP solver (1–2 seconds each layer).The total additional overhead is negligible compared to first-order methods, as the time for calibration dominates.

Connection to REAP Notably, HOPE has a strong connection to Router-weighted Expert Activation Pruning (REAP), a state-of-the-art first-order expert-pruning method.

HOPE's F-matrix encodes individual expert importance scores on the diagonal, and this diagonal effectively recovers the scores used by REAP to independently rank experts for pruning (differing only by the variance of expert contributions).

The diagonal of FFF recovers REAP's expert rankings almost perfectly (Spearman ρ=0.988\rho = 0.988ρ=0.988).Thus, any improvement HOPE achieves over REAP is attributable to the second-order interaction terms in the off-diagonal entries of the FFF-matrix.

Results We evaluated HOPE as an expert pruning method using 3 models (Qwen3.5-122B, Qwen3.5-35B, and GLM-4.5-Air), with 2 calibration datasets (Evol-CodeAlpaca and SWE-Bench Verified trajectories).

We used the Tulu3 suite and SWE-Bench Pro (agentic coding) to test the resulting pruned models, comparing HOPE to 4 first-order baselines (REAP, MAN, EAN, and Frequency) across pruning rates spanning from 10% to 50%.

HOPE achieves the best overall rank across conditions and dominates at aggressive pruning rates.Left: average rank (lower = better) across all models, calibration sets, and benchmarks.Right: SWE-Bench Pro performance on Qwen3.5-35B.Across all conditions, HOPE achieved the best average rank (2.

07 out of 5 methods), with the highest rate of top-1 finishes (39% of conditions) and top-2 finishes (67%).In a head-to-head comparison, HOPE beats every other baseline in the majority of conditions.As expected, HOPE's advantage grows at higher pruning rates.

As more experts are removed, more cooperative links are broken, and the off-diagonal terms of FFF become more important.At 50% pruning, HOPE's average rank improves to 1.58 (vs.2.42 for the next-best method, REAP).

This is also the most deployment-relevant setting, where memory savings are large enough to accommodate longer contexts, deeper reasoning, and more tool calls.HOPE's advantage is most pronounced on SWE-Bench Pro, an agentic coding benchmark where models must solve real GitHub issues autonomously.

These problems involve compositionally diverse tasks (e.g.coding, tool calls, reasoning, etc.) over many turns and likely exercise diverse expert combinations—exactly the regime where preserving cooperative structure matters most.On Qwen3.5-35B-A3B at 50% pruning, HOPE achieved 42.

3 on SWE-Bench Pro, matching the unpruned model's 41.8 (while REAP dropped to 36.2, a 13% regression).Notably, pruned models can occasionally match or exceed the unpruned baseline, likely because the removal of low-relevance experts reduces noise on the target tasks.

This effect is not unique to HOPE and has been observed across pruning methods.

Furthermore, our experiments showed that code-focused calibration with any pruning method (including HOPE) did not severely degrade non-coding benchmarks, and HOPE performed well across the Tulu3 suite regardless of calibration set.

HOPE retains more cooperating experts Average strength of cooperation (measured by F-matrix value) with retained experts, split by experts on which HOPE and REAP disagree.HOPE's advantage stems from optimizing over the entire interaction matrix FFF.

To understand how this affects pruning decisions in practice, we examine the experts where HOPE and REAP disagree.For each disputed expert, we compute its cooperative score (i.e.its average F-value with other retained experts).

Experts retained only by HOPE consistently have higher cooperative scores than experts retained only by REAP.In other words, as part of the QP optimization, HOPE sacrifices some individually high-scoring experts in favor of preserving cooperative clusters.

The gains in benchmark performance (shown above) validate that this trade-off is advantageous.How to run HOPE We created a basic package to run HOPE.It has three steps: 1) calibrate; 2) solve; 3) prune.Our package has been tested on Qwen3, Qwen3-Next, Qwen3.5, and GLM-4.5.

Step 1: Calibrate hope calibrate \ --model-path Qwen/Qwen3.5-35B-A3B \ --prompts calibration_prompts.txt \ --out-path observati