By 苏剑林 | May 29, 2026
In last year's technical report "Muon is Scalable for LLM Training", to compare the differences between models trained with Muon and Adam, we introduced the concept of "Singular Value Entropy." We observed that the matrix parameters trained by Muon generally have higher singular value entropy than those trained by Adam, which we used as empirical evidence that "Muon utilizes model parameters more effectively."
However, is singular value entropy really "the higher, the better"? Is there an optimal singular value entropy? Or to ask the reverse: if we had to pre-define the singular value entropy of matrix parameters, what value should we choose?
First, let's review some concepts. We know that the singular values of a matrix are non-negative, so we can perform some kind of normalization on the singular values, such as direct normalization or squared normalization:
\begin{equation}p_i = \frac{\sigma_i}{\sum_{j=1}^n \sigma_j}\qquad\text{or}\qquad p_i = \frac{\sigma_i^2}{\sum_{j=1}^n \sigma_j^2}\end{equation}
This yields an $n$-dimensional probability distribution $\boldsymbol{p} = (p_1,p_2,\cdots,p_n)$. Once we have the probability distribution, we can calculate related variables, such as entropy:
\begin{equation}H(\boldsymbol{p}) = -\sum_{i=1}^n p_i \log p_i\end{equation}
This is the concept of "Singular Value Entropy." We know that maximum entropy corresponds to a uniform distribution, so singular value entropy is also used to measure the uniformity of singular values. However, there is more than one concept of entropy. The logarithmic expectation form above is called "Shannon Entropy"; additionally, there are other types of entropy, such as Rényi Entropy:
\begin{equation}H_q(\boldsymbol{p}) = -\frac{1}{q-1}\log \sum_{i=1}^n p_i^q \end{equation}
In fact, Shannon entropy is a special case of it, corresponding to $q\to 1$. For other values of $q$, it also reaches its maximum value of $\log n$ at the uniform distribution and its minimum value of $0$ at a One-Hot distribution. The singular value entropy of a matrix is essentially equivalent to the Effective Rank of the matrix; another interpretation is the sparsity level of the matrix's singular values (refer to "How to Measure Data Sparsity?").
The question now is how to transform the doubt "is singular value entropy really the higher the better" into a computable proposition to provide a reference answer.
Intuitively, since current models are generally over-parameterized, there are many redundant degrees of freedom. This allows us to impose certain constraints on the model parameters—such as constraining their norm or singular value entropy—without significantly losing performance. After imposing constraints on the parameters, their degrees of freedom are naturally compressed. We can empirically assume: the larger the remaining degrees of freedom, the stronger the model's capability. This is the principle behind our search for the "optimal singular value entropy."
So, given a singular value entropy, how many degrees of freedom remain for the matrix? Let's look at two extreme cases: the first is minimum entropy, where only one singular value is non-zero, making the matrix a Rank-1 matrix with no more than $2n$ degrees of freedom; the second is maximum entropy, where all singular values are non-zero and equal, making the matrix a multiple of an orthogonal matrix with approximately $n^2/2$ degrees of freedom. In this light, maximum entropy is indeed better than minimum entropy, but is there a better intermediate value?
We can also set aside the matrix context and look directly at the "entropy density" of the singular value distribution. The maximum entropy of an $n$-dimensional distribution is $\log n$, which only the uniform distribution satisfies; the minimum entropy is $0$, corresponding to a One-Hot distribution, for which there are $n$ different choices. Clearly, the "density" at these two boundaries seems low—they can only represent a small fraction of distributions. Therefore, we can guess that the optimal value indeed lies somewhere in the middle.
Transforming the problem this way, it becomes: if we uniformly sample $n$-dimensional distributions and then calculate the entropy, what does the distribution of entropy look like? Where is the point of maximum probability density? The point with the highest probability density means that the largest number of distributions exists in that vicinity, so we consider this the entropy value with the maximum expressive power.
Some readers might not understand "uniformly sampling $n$-dimensional distributions"—usually, we are given a probability distribution and then sample a number, but here we want to sample the probability distribution itself. However, as long as we temporarily set aside the probabilistic meaning of $\boldsymbol{p}$, we can gain a very clear geometric understanding.
Specifically, an $n$-dimensional probability distribution $\boldsymbol{p}$ is an $n$-dimensional vector satisfying the following constraints:
\begin{equation}0 \leq p_i \leq 1 \, (\forall i=1,2,\cdots,n)\qquad \sum_{i=1}^n p_i = 1\end{equation}
The first constraint describes a unit hypercube in $n$-dimensional space, and the second constraint describes a hyperplane. Their intersection yields a finite sub-plane, known as the "$(n-1)$-simplex," denoted as $\Delta^{n-1}$. "Uniformly sampling $n$-dimensional distributions" is effectively sampling a point uniformly on this sub-plane, which makes the concept much more intuitive.
I wonder if readers are curious why the triangle symbol $\Delta$ is used to represent the simplex? In fact, when $n=3$, this sub-plane is exactly an equilateral triangle with vertices at $(1,0,0)$, $(0,1,0)$, and $(0,0,1)$; it is not difficult to imagine that when $n=4$, it forms a regular tetrahedron in three-dimensional space. By extension, it can be seen that from a geometric perspective, the simplex is simply the generalization of an equilateral triangle in higher-dimensional space, hence the use of the triangle $\Delta$.
Back to the topic. Let's consider calculating the probability density of entropy, which can be formally written as:
\begin{equation}\rho(h) = \frac{\int_{\boldsymbol{p}\in\Delta^{n-1}} \delta(H(\boldsymbol{p}) - h) d\boldsymbol{p}}{\int_{\boldsymbol{p}\in\Delta^{n-1}} d\boldsymbol{p}}\end{equation}
where $\delta(\cdot)$ is the Dirac function. The physical meaning of this equation is quite intuitive: it "counts" the distributions where $H(\boldsymbol{p})=h$ and سپس divides by the total number. However, this is purely a formal solution; calculating it analytically is nearly impossible.
Considering that in practical scenarios $n$ is usually quite large (hundreds or even thousands), we can assume based on the "Central Limit Theorem" that $\rho(h)$ approximately follows a normal distribution. A normal distribution has only two parameters, mean and variance, so we only need to calculate the mean and variance (second moment) of the entropy, namely $\mathbb{E}_{\boldsymbol{p}\sim\Delta^{n-1}}[H(\boldsymbol{p})]$ and $\mathbb{E}_{\boldsymbol{p}\sim\Delta^{n-1}}[H(\boldsymbol{p})^2]$. If we only want to find the point of maximum probability, we only need to calculate the mean, as the maximum probability point of a normal distribution is its mean.
The difficulty in calculating the mean $\mathbb{E}_{\boldsymbol{p}\sim\Delta^{n-1}}[H(\boldsymbol{p})]$ is how to implement the sampling $\boldsymbol{p}\sim\Delta^{n-1}$. While this sampling is conceptually intuitive, it is not clear how to implement it in calculations. If readers are familiar with Dirichlet distributions, they can use existing conclusions to speed up understanding, but here we intentionally avoid introducing the Dirichlet distribution to lower the hurdle for general readers.
Fortunately, uniform sampling in a simplex can be transformed into independent repeated sampling from an Exponential distribution $\text{Exp}(1)$:
\begin{equation}\boldsymbol{p}\sim\Delta^{n-1}\qquad\Leftrightarrow\qquad p_i = \frac{x_i}{\sum_{j=1}^n x_j},\quad x_1,x_2,\cdots,x_n\sim \text{Exp}(1)\end{equation}
In other words, sampling $n$ numbers independently from $\text{Exp}(1)$ and then normalizing them is equivalent to uniform sampling on the simplex. How to understand this? A rigorous proof is certainly possible, but here we prefer to introduce an analogy from the perspective of isotropy.
Let's first change to another problem: uniformly sampling $n$-dimensional unit vectors. This is easy to achieve: just sample $n$ numbers $(x_1,x_2,\cdots,x_n)=\boldsymbol{x}$ from a standard normal distribution $\mathcal{N}(0,1)$, and then return $\boldsymbol{x}/\Vert\boldsymbol{x}\Vert$. Why does this work? Because the probability density of $\mathcal{N}(0,1)$ is proportional to $e^{-x^2/2}$, and sampling $n$ times yields $e^{-\sum_{i=1}^n x_i^2/2} = e^{-\Vert\boldsymbol{x}\Vert^2/2}$, which depends only on the magnitude of $\boldsymbol{x}$ and not its direction. Thus, sampling from $\mathcal{N}(0,1)$ is uniform with respect to direction.
A unit vector is a vector whose "sum of squares equals 1." The probability distribution we want to sample now is a vector whose "sum equals 1." In this light, the probability distribution is a unit vector based on the "sum" norm. We know that the probability density of the exponential distribution $\text{Exp}(1)$ is $e^{-x}$, and sampling $n$ times yields $e^{-\sum_{i=1}^n x_i}$, which depends only on the "sum" magnitude. Therefore, it is uniform for this new type of "directional vector."
With all that preparation, we can now formally enter the calculation. Under the new parameterization, the entropy $H(\boldsymbol{p})$ can be transformed into:
\begin{equation}H(\boldsymbol{p}) = \log \sum_{i=1}^n x_i - \frac{\sum_{i=1}^n x_i \log x_i}{\sum_{i=1}^n x_i}\end{equation}
Taking the expectation on both sides leads to a rather complex integral problem. Interestingly, although this integral is complex, it can be solved analytically using certain techniques. However, we won't delve into those details here; instead, we'll use a mean-field trick to quickly obtain an approximate solution:
\begin{equation}\begin{aligned}
\mathbb{E}\left[\log \sum_{i=1}^n x_i - \frac{\sum_{i=1}^n x_i \log x_i}{\sum_{i=1}^n x_i}\right] \approx \log \sum_{i=1}^n \mathbb{E}[x_i] - \frac{\sum_{i=1}^n \mathbb{E}[x_i \log x_i]}{\sum_{i=1}^n \mathbb{E}[x_i]} = \log n - (1 - \gamma)
\end{aligned}\end{equation}
where $\gamma$ is the Euler-Mascheroni constant. Here, $\mathbb{E}[x_i]=1$ is obvious; slightly more difficult is $\mathbb{E}[x_i \log x_i]=1-\gamma$, but this result is also classic and related to the derivative of the Gamma function. If you didn't know this result, it could also be calculated directly using Mathematica.
We know that the maximum entropy of an $n$-dimensional distribution is $\log n$. Now an additional term $-(1-\gamma) = - 0.42278\cdots$ has appeared. This is exactly what we wanted: it indicates that the point of maximum entropy density is roughly $0.42$ lower than the maximum entropy. For a matrix, the expressive power is stronger for matrices with singular value entropy near this value, rather than being "the higher the better." If we need to fix the singular value entropy, this value would be a good choice.
Using the same logic, we can calculate the expectation of a general Rényi entropy:
\begin{equation}\begin{aligned}
\mathbb{E}[H_q(\boldsymbol{p})] =&\, \frac{1}{q-1}\mathbb{E}\left[q\log \sum_{i=1}^n x_i -\log \sum_{i=1}^n x_i^q\right] \\
\approx&\, \frac{1}{q-1}\left[q\log \sum_{i=1}^n \mathbb{E}[x_i] - \log \sum_{i=1}^n \mathbb{E}[x_i^q]\right] \\
=&\, \frac{1}{q-1}\left[q \log n - \log n\Gamma(q+1)\right] \\
=&\, \log n - \frac{\log \Gamma(q+1)}{q-1}
\end{aligned}\label{eq:E-Hq}\end{equation}
As $q\to 1$, the corresponding limit is exactly $\log n - (1-\gamma)$, consistent with the previous section's result. If $q$ is an integer greater than 1, we can write directly:
\begin{equation}\mathbb{E}[H_q(\boldsymbol{x})] \approx \log n - \frac{\log q\,!}{q-1}\end{equation}
If we take $q=2$, the result is a very simple $\log (n/2)$. We know that $\log (n/2)$ is the maximum entropy of an $n/2$-dimensional distribution, so the visual interpretation of this result is: singular value entropy only needs to "fill" half of the dimensions; there is no need to fill all dimensions, otherwise diversity might actually be lost.
In the previous two sections, we focused on estimating the mean, which is sufficient for our original goal. However, for the sake of completeness, this section supplements the estimation of variance and discusses why asymptotic normality holds for $\rho(h)$.
First, we perform an identity transformation on $H_q(\boldsymbol{p})$:
\begin{equation}H_q(\boldsymbol{p}) = \log n + \frac{1}{q-1}\Bigg[q\log \underbrace{\frac{1}{n}\sum_{i=1}^n x_i}_u -\log \underbrace{\frac{1}{n}\sum_{i=1}^n x_i^q}_v\Bigg]\end{equation}
When $q\neq 1$, $u$ and $v$ are two different statistics. The central limit theorem tells us that as $n\to\infty$, $(u,v)$ asymptotically follows a bivariate normal distribution. Given $x_i\sim \text{Exp}(1)$, we can find that the mean and variance of $u$ are $1$ and $1/n$, while the mean and variance of $v$ are $\Gamma(q+1)$ and $[\Gamma(2q+1) - \Gamma(q+1)^2]/n$.
From this, it can be seen that when $q$ is small and $n$ is large, they are concentrated near their respective means. This allows us to perform a Taylor expansion around the mean, such as a first-order approximation:
\begin{equation}H_q(\boldsymbol{p}) \approx \log n + \frac{1}{q-1}\Bigg[q(u-1) - \left(\log \Gamma(q+1) + \frac{v - \Gamma(q+1)}{\Gamma(q+1)}\right)\Bigg]\end{equation}
Taking the expectation of both sides yields equation $\eqref{eq:E-Hq}$. If necessary, we could expand to higher orders to find remainders. Since $H_q(\boldsymbol{p})$ is now linear regarding $u$ and $v$, we can also calculate the variance using the following formula:
\begin{equation}\mathbb{V}\mathrm{ar}[H_q(\boldsymbol{p})] \approx \mathbb{V}\mathrm{ar}\left[\frac{q u}{q-1}\right] + \mathbb{V}\mathrm{ar}\left[\frac{v}{(q-1) \Gamma(q+1)}\right] - 2\mathbb{C}\mathrm{ov}\left[\frac{q u}{q-1},\frac{v}{(q-1) \Gamma(q+1)}\right]\end{equation}
Note that $u$ and $v$ are generally not independent, so the covariance term cannot be ignored:
\begin{equation}\mathbb{C}\mathrm{ov}[u, v] = \frac{1}{n^2}\sum_{i=1}^n \sum_{j=1}^n \mathbb{C}\mathrm{ov}[x_i, x_j^q] = \frac{1}{n^2}\sum_{i=1}^n \mathbb{C}\mathrm{ov}[x_i, x_i^q]\end{equation}
The first equality follows from the definitions of $u$ and $v$ and the bilinearity of $\mathbb{C}\mathrm{ov}$. The second equality is because $x_i$ and $x_j$ are independent when $i\neq j$, so their covariance is 0. Calculating further, we get $\mathbb{C}\mathrm{ov}[x_i, x_i^q] = \mathbb{E}[x_i^{q+1}] - \mathbb{E}[x_i] \mathbb{E}[x_i^q]=\Gamma(q+2) - \Gamma(q+1) = q \Gamma(q+1)$. Substituting these results, we get:
\begin{equation}\mathbb{V}\mathrm{ar}[H_q(\boldsymbol{p})] \approx \frac{1}{n}\left[\frac{\Gamma(2q+1)}{(q-1)^2 \Gamma(q+1)^2} - \frac{q^2+1}{(q-1)^2}\right]\end{equation}
When $q=2$, the result is $1/n$; when $q\to 1$, the result is $(\pi^2/3-3)/n$.
In summary, the central limit theorem does not directly provide the asymptotic normality of the entropy $h$. We only obtain the asymptotic normality of the intermediate variables $u,v$ and use them to estimate the mean and variance of $h$. The results show the variance of $h$ is also $\mathcal{O}(1/n)$, indicating that the variance decreases as $n$ increases. Therefore, finding the point of maximum probability density via the mean is scientifically sound.
This article transforms the question "Is higher singular value entropy always better?" into a quantifiable mathematical proposition. The core hypothesis is: after fixing the singular value entropy, the remaining degrees of freedom of the matrix parameters represent its expressive capability. Subsequently, we interpreted the position with the maximum degrees of freedom as the point of maximum entropy density. Through a series of transformations and approximations, we obtained an optimal entropy value.
Of course, the result itself is not particularly important; the significance of this article lies more in providing a reference framework for the analysis of related problems.