By 苏剑林 | May 15, 2026
Anyone who has trained an MoE (Mixture-of-Experts) model knows that if the MLP parts of the entire model are replaced with MoE, the MoE layers closer to the Embedding (the first few layers) often struggle to achieve load balancing. In response to this, DeepSeek V3, as well as our Kimi K2, adopted a strategy called "first_k_dense"—which, as the name implies, means using conventional Dense GLUs instead of MoE for the first $k$ layers. In DeepSeek V4, this strategy was changed to "first_k_hash."
The "hash" here refers to "Hash Routing," proposed in "Hash Layers For Large Sparse Models." It assigns an Expert to each Token through a pre-determined mapping table called tid2eid. This article explores how to generate this tid2eid table.
First, it should be noted that for the initial MoE layers, using Quantile Balancing (as proposed by the author in "MoE Odyssey 6: Optimal Allocation for Balance" and "MoE Odyssey 7: Minimalist Solution for Dynamic Activation") can already significantly alleviate their imbalance issues.
As for the Hash Routing adopted by V4, it can be seen as an exploration of the same problem from a different direction. The idea is that for the first few MoE layers, there is not much contextual information. Therefore, selecting an Expert based on the input vector may not be significantly different from selecting one directly based on the Token Id, which carries no contextual information. Since this is the case, one might as well pre-define the Experts to be activated for each Token in the first few layers based on the Token Id; this is the origin of "tid2eid (Token Id to Expert Id)."
So, how do we generate this tid2eid table? Is complete randomness acceptable? It seems not, because the frequency of each Token is different, and complete randomness would actually lead to imbalance. DeepSeek has not released the generation details of this table, so we can only speculate based on our own reasoning.
Assume there are $m$ different Tokens, and the frequency of the $i$-th Token is $p_i$. Without loss of generality, assume they are sorted in descending order, i.e., $p_i \geq p_{i+1}$. We use $x_{i,j} \in \{0, 1\}$ to indicate whether Token $i$ should activate Expert $j$ ($0$ for no, $1$ for yes). There are $n$ Experts in total, and each Token selects $k$ Experts. We actually need to solve the following system of equations:
\begin{equation}x_{i,j}\in\{0, 1\},\qquad\sum_{j=1}^n x_{i,j} = k,\qquad \sum_{i=1}^m p_i x_{i,j}\approx \frac{k}{n}\end{equation}Note that we used the approximation symbol $\approx$ in the last equation because, strictly speaking, the system might not have a solution if it were an exact equality $=$. Thus, we keep the $\approx$ notation to mean the two sides should be as close as possible. It can also be formulated as an optimization problem:
\begin{equation}\min_{x_{i,j}\in\{0, 1\}} \sum_{j=1}^n \left(\sum_{i=1}^m p_i x_{i,j} - \frac{k}{n}\right)^2 \qquad\text{s.t.}\qquad\sum_{j=1}^n x_{i,j} = k\end{equation}A relatively simple solution to this problem is a greedy algorithm:
Process tokens one by one in order of frequency from high to low. First, count the current allocated load of each Expert, then select the $k$ Experts with the lightest load as the activated Experts for the current Token, and subsequently update the load distribution of the Experts.
Although the greedy algorithm is greedy in principle, it yields a near-optimal solution in most cases. A reference implementation is as follows:
import numpy as np
# Simulate distribution
m, n, k = 80000, 128, 4
p = 1 / (10 + np.arange(m))
p /= p.sum()
# Greedy processing
x, f = np.zeros((m, k), dtype='int32'), np.zeros(n)
for i in range(m):
j = f.argsort()[:k] # Select the k lightest loads
f[j] += p[i] / k # Update load distribution
x[i] = j # Record into tid2eid
# Evaluate balance
max_vio, min_vio = f.max() * n - 1, f.min() * n - 1Note that there is nothing random about this code; in principle, it is a deterministic algorithm. What if we want different tid2eid tables for the first few layers? We can consider adding some randomness, such as changing range(m) to np.random.permutation(m)—that is, not executing in order of frequency. This will still produce a usable solution, although the balance might be slightly worse. We can run it multiple times and choose the solutions with the lowest max_vio.
Now let's consider an extreme scenario: $p_1 \gg k / n$, meaning the frequency of a certain Token already far exceeds the equilibrium level. In this case, no matter how the tid2eid is arranged, it is impossible to achieve load balancing. In other words, making decisions based purely on a single Token Id simply won't balance. How should we handle this?
The answer is simple: switch to a Hash method that relies on more input information. Previously, we made decisions based only on the current Token Id. In reality, every Token exists within a sequence and has context. If the previous approach was "1-gram to expert," we can now consider combining it with the preceding Tokens to form "2-gram to expert," "3-gram to expert," and so on.
Taking a 2-gram $(a, b)$ as an example, a naive hashing method is: pick a prime number $q$ larger than $m$, and calculate $(aq + b) \mod n$ as the activated Expert. Repeat this $k$ times with different prime numbers to get $k$ Experts. By increasing the number of grams, the number of input combinations increases significantly. When these are mapped to a finite $n$, each bucket is highly likely to be "filled," so as long as the Hash function is not particularly terrible, the result will be almost balanced.
Therefore, its advantage is that there is no need to consider frequency, nor is there a need to pre-memorize a tid2eid table. As for the disadvantages, the calculation of the Hash function is slightly more complex, and the same Token might select duplicate Experts, but these are not particularly serious issues.
This article briefly summarized the basic idea of Hash Routing in DeepSeek V4 and focused on exploring the construction principles of its tid2eid mapping table.