By 苏剑林 | October 15, 2026
In the previous article "Slow is Fast: 1. Is the End of Shampoo Muon?", we started from the average convergence conclusion of convex optimization and established the "Orthogonal Slowest Descent" framework to improve any preconditioner optimizer in Kronecker product form. The results showed that under the optimization of this framework, these optimizers tend to converge towards Muon, leading the author to issue the somewhat pessimistic sentiment that "the end of Shampoo is Muon."
In addition to the Kronecker product, another simpler structural approximation for preconditioner matrices is the diagonal approximation, with common examples being RMSProp, Adam, etc. Note that while diagonal approximation is computationally simpler than the Kronecker product, there is no inclusion relationship between them. Therefore, a new question arises: will applying Orthogonal Slowest Descent to this class of optimizers yield new results?
Target Analysis
Following the notation of the previous article, let the gradient be $\boldsymbol{G}\in\mathbb{R}^{n\times m}, \boldsymbol{g}=\mathop{\text{vec}}(\boldsymbol{G})\in\mathbb{R}^{nm}$, and introduce the matrix $\boldsymbol{N}\in\mathbb{R}_+^{n\times m}$. The so-called diagonal approximation refers to a preconditioner matrix that is diagonal in its vectorized representation, i.e., \(\newcommand{diag}{\mathop{\text{diag}}}\boldsymbol{H}^{-1}=\diag(\mathop{\text{vec}}(\boldsymbol{N}))\), so that
\begin{equation}\boldsymbol{H}^{-1} \boldsymbol{g} = \mathop{\text{vec}}(\boldsymbol{N}\odot\boldsymbol{G})\end{equation}
Or rather, the update rule is $\boldsymbol{W} - \eta \boldsymbol{N}\odot\boldsymbol{G}$, where $\odot$ is the Hadamard product. What we want to do is introduce an orthogonal transformation in Kronecker product form to $\boldsymbol{N}$, so that under the principle of Orthogonal Slowest Descent, it becomes a better optimizer.
Specifically, we introduce orthogonal matrices $\boldsymbol{X}\in\mathbb{R}^{n\times n}, \boldsymbol{Y}\in\mathbb{R}^{m\times m}$, and let $\boldsymbol{O} = \boldsymbol{Y}\otimes\boldsymbol{X}$. We replace the diagonal preconditioner matrix with $\boldsymbol{O}^{\top}\diag(\mathop{\text{vec}}(\boldsymbol{N}))\boldsymbol{O}$. That is, the update amount changes from $\boldsymbol{N}\odot\boldsymbol{G}$ to $\boldsymbol{X}^{\top}(\boldsymbol{N}\odot\boldsymbol{X}\boldsymbol{G}\boldsymbol{Y}^{\top})\boldsymbol{Y}$. This transformation requires some foundation in Kronecker products, which we won't expand upon here.
Intuitively, it involves finding a new set of orthogonal bases for $\boldsymbol{G}$, performing the diagonal preconditioning on them, and then transforming back. According to the Orthogonal Slowest Descent principle, we want to minimize its inner product with $\boldsymbol{G}$, namely
\begin{equation}\newcommand{tr}{\mathop{\text{tr}}}\min_{\boldsymbol{X},\boldsymbol{Y}} \tr(\boldsymbol{G}^{\top}\boldsymbol{X}^{\top}(\boldsymbol{N}\odot\boldsymbol{X}\boldsymbol{G}\boldsymbol{Y}^{\top})\boldsymbol{Y})\qquad\text{s.t.}\qquad \boldsymbol{X}^{\top}\boldsymbol{X} = \boldsymbol{I}_n,\quad \boldsymbol{Y}^{\top}\boldsymbol{Y} = \boldsymbol{I}_m\end{equation}
Unified Form
Notice that
\begin{equation}\tr(\boldsymbol{G}^{\top}\boldsymbol{X}^{\top}(\boldsymbol{N}\odot\boldsymbol{X}\boldsymbol{G}\boldsymbol{Y}^{\top})\boldsymbol{Y}) = \tr(\boldsymbol{Y}\boldsymbol{G}^{\top}\boldsymbol{X}^{\top}(\boldsymbol{N}\odot\boldsymbol{X}\boldsymbol{G}\boldsymbol{Y}^{\top})) = \Vert\boldsymbol{N}^{\odot/2}\odot\boldsymbol{X}\boldsymbol{G}\boldsymbol{Y}^{\top}\Vert_F^2\end{equation}
Here $\boldsymbol{N}^{\odot/2}$ refers to the square root in the sense of the Hadamard product, i.e., taking the square root element-wise. Furthermore, in the previous article, we solved
\begin{equation}\min_{\boldsymbol{X},\boldsymbol{Y}} \tr(\boldsymbol{G}^{\top}\boldsymbol{X}^{\top}\boldsymbol{P}\boldsymbol{X}\boldsymbol{G}\boldsymbol{Y}^{\top}\boldsymbol{Q}\boldsymbol{Y})\qquad\text{s.t.}\qquad \boldsymbol{X}^{\top}\boldsymbol{X} = \boldsymbol{I}_n,\quad \boldsymbol{Y}^{\top}\boldsymbol{Y} = \boldsymbol{I}_m\end{equation}
The objective function can be rewritten as
\begin{equation}\tr(\boldsymbol{G}^{\top}\boldsymbol{X}^{\top}\boldsymbol{P}\boldsymbol{X}\boldsymbol{G}\boldsymbol{Y}^{\top}\boldsymbol{Q}\boldsymbol{Y}) = \tr(\boldsymbol{Y}\boldsymbol{G}^{\top}\boldsymbol{X}^{\top}\boldsymbol{P}\boldsymbol{X}\boldsymbol{G}\boldsymbol{Y}^{\top}\boldsymbol{Q}) = \Vert\boldsymbol{P}^{1/2}\boldsymbol{X}\boldsymbol{G}\boldsymbol{Y}^{\top}\boldsymbol{Q}^{1/2}\Vert_F^2\end{equation}
It can be found that the objective functions of both problems have the form $f(\boldsymbol{X} \boldsymbol{G} \boldsymbol{Y}^{\top})$, so we generally consider
\begin{equation}\min_{\boldsymbol{X}, \boldsymbol{Y}} f(\boldsymbol{X} \boldsymbol{G} \boldsymbol{Y}^{\top}) \qquad\text{s.t.}\qquad \boldsymbol{X}^{\top}\boldsymbol{X} = \boldsymbol{I}_n,\quad \boldsymbol{Y}^{\top}\boldsymbol{Y} = \boldsymbol{I}_m\end{equation}
Stationary Point Conditions
Let's first find the stationary point conditions for the above problem. To do this, we introduce Lagrange multipliers $\boldsymbol{A}\in\mathbb{R}^{n\times n}, \boldsymbol{B}\in\mathbb{R}^{m\times m}$ and define the auxiliary function
\begin{equation}F = f(\boldsymbol{X} \boldsymbol{G} \boldsymbol{Y}^{\top}) - \tr(\boldsymbol{A}^{\top}(\boldsymbol{X}^{\top}\boldsymbol{X} - \boldsymbol{I}_n)) - \tr(\boldsymbol{B}^{\top}(\boldsymbol{Y}^{\top}\boldsymbol{Y} - \boldsymbol{I}_m))\end{equation}
Let $\boldsymbol{Z} = \boldsymbol{X} \boldsymbol{G} \boldsymbol{Y}^{\top}, \boldsymbol{K} = \nabla_{\boldsymbol{Z}} f(\boldsymbol{Z})$. We calculate the gradients of $F$ with respect to $\boldsymbol{X}$ and $\boldsymbol{Y}$ and set them to zero:
\begin{gather}\nabla_{\boldsymbol{X}} F = \boldsymbol{K} \boldsymbol{Y} \boldsymbol{G}^{\top} - \boldsymbol{X}(\boldsymbol{A} + \boldsymbol{A}^{\top}) = \boldsymbol{0} \\
\nabla_{\boldsymbol{Y}} F = \boldsymbol{K}^{\top} \boldsymbol{X} \boldsymbol{G} - \boldsymbol{Y}(\boldsymbol{B} + \boldsymbol{B}^{\top}) = \boldsymbol{0}\end{gather}
Multiplying the first equation by $\boldsymbol{X}^{\top}$ and the second by $\boldsymbol{Y}^{\top}$ on the left, and combining this with the orthogonality of $\boldsymbol{X}$ and $\boldsymbol{Y}$, we get the stationary point conditions:
\begin{equation}\boldsymbol{X}^{\top}\boldsymbol{K}\boldsymbol{Y}\boldsymbol{G}^{\top}\,\,\text{is symmetric},\qquad\boldsymbol{G}^{\top}\boldsymbol{X}^{\top}\boldsymbol{K}\boldsymbol{Y}\,\,\text{is symmetric}\end{equation}
As we proved in the previous article, this implies that $\boldsymbol{X}^{\top}\boldsymbol{K}\boldsymbol{Y}$ has the form $\boldsymbol{U}\boldsymbol{D}\boldsymbol{V}^{\top}$, where $\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^{\top}$ is the SVD of $\boldsymbol{G}$, and $\boldsymbol{D}$ is a diagonal matrix. This is a common feature of the optimal solution to the above optimization problem, independent of the specific choice of $f$, which is a very strong conclusion.
Washing Away the Basis
Let's use this conclusion to review the target $\Vert\boldsymbol{P}^{1/2}\boldsymbol{X}\boldsymbol{G}\boldsymbol{Y}^{\top}\boldsymbol{Q}^{1/2}\Vert_F^2$ from the previous article. In this case, $\boldsymbol{K} = 2\boldsymbol{P}\boldsymbol{X}\boldsymbol{G}\boldsymbol{Y}^{\top}\boldsymbol{Q}$. The update amount we are concerned with is $\boldsymbol{\Phi} = \boldsymbol{X}^{\top}\boldsymbol{P}\boldsymbol{X}\boldsymbol{G}\boldsymbol{Y}^{\top}\boldsymbol{Q}\boldsymbol{Y} = \boldsymbol{X}^{\top}\boldsymbol{K}\boldsymbol{Y}/2$. According to the results of the previous section, $\boldsymbol{\Phi}$ has the form $\boldsymbol{U}\boldsymbol{D}\boldsymbol{V}^{\top}$, meaning the characteristic matrices of $\boldsymbol{P}$ and $\boldsymbol{Q}$ are completely eliminated.
To the author's surprise, the same conclusion can be extended to the protagonist of this article—the target $\Vert\boldsymbol{N}^{\odot/2}\odot\boldsymbol{X}\boldsymbol{G}\boldsymbol{Y}^{\top}\Vert_F^2$. In this case, $\boldsymbol{K} = 2\boldsymbol{N}\odot\boldsymbol{X}\boldsymbol{G}\boldsymbol{Y}^{\top}$. The update amount we care about is $\boldsymbol{\Phi} = \boldsymbol{X}^{\top}(\boldsymbol{N}\odot\boldsymbol{X}\boldsymbol{G}\boldsymbol{Y}^{\top})\boldsymbol{Y} = \boldsymbol{X}^{\top}\boldsymbol{K}\boldsymbol{Y}/2$. Thus, according to the stationary point condition, $\boldsymbol{\Phi}$ still takes the form $\boldsymbol{U}\boldsymbol{D}\boldsymbol{V}^{\top}$.
In other words, whether it is a Kronecker product or a diagonal approximation, the "basis" of the preconditioner matrix will be washed away by the Orthogonal Slowest Descent: the left and right singular matrices of the update amount consistently degenerate into the left and right singular matrices of the gradient (or momentum). What preconditioning can change, at most, is only the singular values—which is identical to the construction of Muon. This is the first layer of meaning for the title "The End is Muon."
Can it be Uniform?
If we can prove that the diagonal elements of $\boldsymbol{D}$ tend to be balanced, then it becomes even more convergent with Muon. This is the second layer of meaning for "The End is Muon."
In the previous article, we conducted a detailed argument for the Kronecker product case, but that process is not easily generalized. Therefore, we use a heuristic method here to sense the characteristics of $\boldsymbol{D}$. Without loss of generality, assume $n\geq m$. We have
\begin{equation}\tr(\boldsymbol{G}^{\top}\boldsymbol{\Phi}) = \tr(\boldsymbol{V}\boldsymbol{\Sigma}^{\top} \boldsymbol{D} \boldsymbol{V}^{\top}) = \tr(\boldsymbol{\Sigma}^{\top} \boldsymbol{D}) = \sum_{i=1}^m \sigma_i d_i \geq m\left(\prod_{i=1}^m \sigma_i d_i\right)^{1/m} = m\left(\prod_{i=1}^m \sigma_i \prod_{i=1}^m d_i\right)^{1/m}\end{equation}
Where $\boldsymbol{\Sigma}=\diag(\sigma_1,\sigma_2,\cdots)$ and $\boldsymbol{D}=\diag(d_1,d_2,\cdots)$. The condition for equality is $d_i\propto \sigma_i^{-1}$. This result indicates that as long as the product of the singular values of $\boldsymbol{\Phi}$ is constant, the ideal minimum of $\tr(\boldsymbol{G}^{\top}\boldsymbol{\Phi})$ is reached at $\boldsymbol{D}\propto \boldsymbol{\Sigma}^{-1}$ (taking the inverse of the diagonal elements, while keeping the shape unchanged).
Of course, since we can only change the orthogonal matrices in $\boldsymbol{\Phi}$, this ideal minimum is difficult to achieve. We can only guess that it is an intermediate result between the most ideal ($\boldsymbol{D}\propto \boldsymbol{\Sigma}^{-1}$) and the worst ($\boldsymbol{D}\propto \boldsymbol{\Sigma}$) scenarios, and their geometric mean is exactly $\boldsymbol{D}\propto \boldsymbol{I}$, which is the equalization of singular values.
Product of Spectra
Is the product of the singular values of $\boldsymbol{\Phi}$ a constant? We will discuss this in the order of Kronecker product first and then diagonal approximation. First, for the "Kronecker product + square matrix" case, it is indeed a constant:
\begin{equation}\prod_{i=1}^m d_i = |\det(\boldsymbol{\Phi})| = |\det(\boldsymbol{X}^{\top}\boldsymbol{P}\boldsymbol{X}\boldsymbol{G}\boldsymbol{Y}^{\top}\boldsymbol{Q}\boldsymbol{Y})| = |\det(\boldsymbol{P})\det(\boldsymbol{G})\det(\boldsymbol{Q})|\end{equation}
That is, under the transformation of orthogonal matrices $\boldsymbol{X}$ and $\boldsymbol{Y}$, the absolute value of the determinant of $\boldsymbol{\Phi}$ remains unchanged, thus the product of singular values remains unchanged, and the conclusion from the previous section applies. However, for non-square matrices, the above identity no longer holds. For this, we have two options: one is to discuss it carefully according to the method in the previous article, and the other is to assume without proof that the above equation approximately holds for non-square matrices.
More problematic is the diagonal approximation case: although the Hadamard product is simple, it is not an essential matrix operation. Its introduction makes various analyses of matrices extremely difficult. Therefore, regardless of whether it is square or not, we cannot effectively estimate the singular values/determinant of $\boldsymbol{\Phi} = \boldsymbol{X}^{\top}(\boldsymbol{N}\odot\boldsymbol{X}\boldsymbol{G}\boldsymbol{Y}^{\top})\boldsymbol{Y}$.
However, since only the orthogonal matrices in $\boldsymbol{\Phi}$ can change, and orthogonal transformations often do not change the determinant, we guess that "the determinant of $\boldsymbol{\Phi}$ will not change significantly either," thus obtaining a trend for $\boldsymbol{D}$ toward uniformity. Note that this guess is very "weak": it certainly does not hold as an identity, and as for the degree of approximation, it cannot be determined. It is more of a "belief" regarding the determinant and orthogonal matrices.
Article Summary
This article extends "Orthogonal Slowest Descent" to diagonal preconditioners (the RMSProp and Adam families). By extracting an abstract form common to two targets, we unified the conclusion that "whether it is a Kronecker product or a diagonal approximation, the original basis will be eliminated by Orthogonal Slowest Descent, at most changing the singular value spectrum of the gradient or momentum." Furthermore, we briefly discussed whether the singular values tend toward equilibrium.