By 苏剑林 | October 09, 2026
As we know, the core of DeltaNet is an update matrix in the form of $\boldsymbol{I}-\eta\boldsymbol{k}\boldsymbol{k}^{\top}$ (refer to "A Brief History of Linear Attention: From Imitation, Innovation to Feedback"). When $\eta \leq 2$ and L2 Normalization is applied to $\boldsymbol{k}$, its continuous product possesses good numerical stability (refer to "Why does DeltaNet need L2 Normalize?" and "Elements of DeltaNet's Core Inverse Matrix are Always within [-1, 1]"). This is key to its stable training, and subsequent works like GDN and KDA have also adopted this setting.
Later, GDN2 attempted to generalize the update matrix to $\boldsymbol{I}-\eta\boldsymbol{j}\boldsymbol{k}^{\top}$, a similar choice also appearing in RWKV7. However, this generalization may introduce new instability issues. This article provides a brief analysis of this problem.
Problem Description
The general form of GDN2 is
\begin{equation}\boldsymbol{S}_t = \boldsymbol{S}_{t-1}\boldsymbol{D}_t(\boldsymbol{I} - \eta_t \boldsymbol{j}_t\boldsymbol{k}_t^{\top}) + \eta_t \boldsymbol{v}_t \boldsymbol{k}_t^{\top}\end{equation}
where $\boldsymbol{j}_t, \boldsymbol{k}_t \in \mathbb{R}^d$, $\eta_t \geq 0$, and $\boldsymbol{D}_t$ is a diagonal matrix whose elements are typically restricted to $[0, 1]$, representing a forget matrix. KDA corresponds to the special case where $\boldsymbol{k}_t = \boldsymbol{j}_t$, while GDN further corresponds to the special case where $\boldsymbol{D}_t = \gamma_t \boldsymbol{I}$.
When we expand the recursion, the product of $\boldsymbol{D}_t(\boldsymbol{I} - \eta_t \boldsymbol{j}_t\boldsymbol{k}_t^{\top})$ appears in $\boldsymbol{S}_t$. To ensure numerical stability, we desire that
\begin{equation}\Vert\boldsymbol{D}_t(\boldsymbol{I} - \eta_t \boldsymbol{j}_t\boldsymbol{k}_t^{\top})\Vert_2 \leq 1\end{equation}
where $\Vert\cdot\Vert_2$ is the spectral norm. Since $\boldsymbol{D}_t$ is a diagonal matrix with elements in $[0, 1]$, it satisfies $\Vert\boldsymbol{D}_t\Vert_2 \leq 1$. Therefore, a sufficient condition for the above is
\begin{equation}\Vert\boldsymbol{I} - \eta_t \boldsymbol{j}_t\boldsymbol{k}_t^{\top}\Vert_2 \leq 1\end{equation}
Unfortunately, as we will see, when $\boldsymbol{j}_t$ and $\boldsymbol{k}_t$ are not collinear (which is the range GDN2 aims to expand into), the above inequality can never hold.
The Spectral Norm
For simplicity, the following discussion omits the subscript $t$ and assumes that $\boldsymbol{j}$ and $\boldsymbol{k}$ are already normalized (otherwise $\Vert\boldsymbol{j}\Vert$ and $\Vert\boldsymbol{k}\Vert$ can be absorbed into $\eta$). Let $\boldsymbol{A} \triangleq \boldsymbol{I} - \eta \boldsymbol{j}\boldsymbol{k}^{\top}$. We have
\begin{equation}\Vert\boldsymbol{A}\Vert_2 = \Vert\boldsymbol{I} - \eta \boldsymbol{j}\boldsymbol{k}^{\top}\Vert_2 = \frac{\eta + \sqrt{\eta^2 - 4\eta\,\boldsymbol{j}^{\top}\boldsymbol{k} + 4}}{2}\label{eq:spec}\end{equation}
This conclusion is the core of this article; the proof is provided at the end. Since $\boldsymbol{j}$ and $\boldsymbol{k}$ are normalized, $\boldsymbol{j}^{\top}\boldsymbol{k} \in [-1, 1]$. When $\boldsymbol{j}^{\top}\boldsymbol{k} < 1$ and $\eta > 0$, we have
\begin{equation}\frac{\eta + \sqrt{\eta^2 - 4\eta\,\boldsymbol{j}^{\top}\boldsymbol{k} + 4}}{2} > \frac{\eta + \sqrt{\eta^2 - 4\eta + 4}}{2} = \frac{\eta + |\eta - 2|}{2} = \max(\eta, 2) - 1 \geq 1\end{equation}
In other words, as long as $\boldsymbol{j} \neq \boldsymbol{k}$ and $\eta > 0$, no matter how you adjust them, the spectral norm of $\boldsymbol{A}$ is strictly greater than 1! More accurately:
\begin{equation}\Vert\boldsymbol{I} - \eta \boldsymbol{j}\boldsymbol{k}^{\top}\Vert_2 \leq 1 \quad\Longleftrightarrow\quad \boldsymbol{j}=\boldsymbol{k}\ \text{and}\ \eta\leq 2\quad(\text{or}\ \eta=0)\end{equation}
But if $\boldsymbol{j}=\boldsymbol{k}$, it degrades to KDA, losing the generalization significance of GDN2. When $\boldsymbol{j} \neq \boldsymbol{k}$, $\Vert\boldsymbol{A}\Vert_2$ is always greater than 1, meaning the stability of GDN2 is not guaranteed. How can we break through this?
Simple Strategy
Fortunately, we still have the forget gate matrix $\boldsymbol{D}$. Since its elements are restricted to $[0, 1]$, the spectral norm of the product $\boldsymbol{D}\boldsymbol{A}$ will likely be less than 1 in practice. Thus, actual training may not necessarily fail. However, this is only a probabilistic outcome, not a theoretical guarantee.
The most ideal method would be to directly calculate the spectral norm of $\boldsymbol{D}\boldsymbol{A}$ and then perform operations like spectral clipping. However, for a general $\boldsymbol{D}$, there is no universal closed-form solution for the spectral norm of this matrix.
Since the spectral norm of $\boldsymbol{A}$ has an analytical solution, we can settle for the next best thing: using its spectral norm to perform a clipping operation on $\boldsymbol{D}$:
\begin{equation}\boldsymbol{D}\boldsymbol{A}\quad\to\quad \min(\boldsymbol{D},1/\rho)\boldsymbol{A}\end{equation}
where $\rho = \Vert\boldsymbol{A}\Vert_2$. In this way, we only modify the diagonal elements of $\boldsymbol{D}$ that are greater than $1/\rho$. Since $\eta$ is usually not large in practice, $1/\rho$ will not be significantly smaller than 1 on average, meaning the extent of modification won't be broad. This achieves the goal of "guaranteeing GDN2 stability with minimal and concise modifications."
It is worth noting that $\rho$ and $\min(\boldsymbol{D}, 1/\rho)$ are both token-wise operations that can exist independently outside the RNN operator. Since $\rho$ has a closed-form solution, once fused, it barely affects efficiency. Thus, it is a modification with very low cognitive load.
Modifying Itself
Besides modifying $\boldsymbol{D}$, one could also modify $\boldsymbol{A}$: a sufficient condition for $\Vert\boldsymbol{D}\boldsymbol{A}\Vert_2 \leq 1$ is $\Vert\boldsymbol{A}\Vert_2 \leq 1/\Vert\boldsymbol{D}\Vert_2$. Since $\boldsymbol{D}$ is a diagonal matrix with elements in $[0, 1]$, its spectral norm is its maximum diagonal element, denoted as $\gamma_{\max}$.
Next, we absorb $\eta$ into $\boldsymbol{j}$. Thus $\eta$ disappears, but $\boldsymbol{j}$ is no longer normalized. This parametrization is more convenient for this section. In this case:
\begin{equation}\Vert\boldsymbol{A}\Vert_2 = \Vert\boldsymbol{I}-\boldsymbol{j}\boldsymbol{k}^{\top}\Vert_2 = \frac{\Vert\boldsymbol{j}\Vert + \sqrt{\Vert\boldsymbol{j}\Vert^2 - 4\boldsymbol{j}^{\top}\boldsymbol{k} + 4}}{2}\end{equation}
When $\Vert\boldsymbol{A}\Vert_2 > \gamma_{\max}^{-1}$, we hope to replace $\boldsymbol{j}$ with $\boldsymbol{j}_*$ such that the corresponding $\Vert\boldsymbol{A}_*\Vert_2 \leq \gamma_{\max}^{-1}$. During the transformation, we want to maintain $\boldsymbol{j}_*^{\top}\boldsymbol{k} = \boldsymbol{j}^{\top}\boldsymbol{k} = c$, as it corresponds to the erase rate of the Delta Rule. Solving for this yields:
\begin{equation}\Vert\boldsymbol{j}_*\Vert \leq (c-1)\gamma_{\max} + \gamma_{\max}^{-1} \triangleq \tau \end{equation}
The cluster of vectors satisfying $\boldsymbol{j}_*^{\top}\boldsymbol{k}=c$ is $\boldsymbol{j}_* = c\boldsymbol{k} + \lambda(\boldsymbol{j} - c\boldsymbol{k})$. Its magnitude is $\sqrt{c^2 + \lambda^2(\Vert\boldsymbol{j}\Vert^2 - c^2)}$. Truncating it to $\tau$ gives:
\begin{equation}\lambda_* = \min\left(1, \sqrt{\frac{\tau^2 - c^2}{\Vert\boldsymbol{j}\Vert^2 - c^2}}\right)\end{equation}
This solution is particularly clever: when $\gamma_{\max} = 1$, then $\tau = c$, and thus $\boldsymbol{j}_*= c \boldsymbol{k}$, which degrades to KDA—and KDA guarantees stability! In other words, it satisfies the condition by moving closer to KDA, making it at least as stable as KDA. This is exactly the new modification introduced in GDN2.
Note: A limitation of this scheme is $\tau \geq c$. A sufficient condition for this is $c \geq 0$, otherwise there might be no solution. GDN2 ensures this by parameterizing $\boldsymbol{j}$ as $\boldsymbol{k} \odot \boldsymbol{b}, \boldsymbol{b} \in [0, 1]^d$, so it is self-consistent.
Proof Process
Finally, let's complete the proof of equation $\eqref{eq:spec}$. Without loss of generality, assume $\boldsymbol{j}$ and $\boldsymbol{k}$ are not collinear. Orthogonalize them into $\boldsymbol{u}_1 = \boldsymbol{k}$ and $\boldsymbol{u}_2 = \frac{\boldsymbol{j} - c\boldsymbol{k}}{\sqrt{1-c^2}}$, where $c = \boldsymbol{j}^{\top}\boldsymbol{k}$. Let $\boldsymbol{U}_1 = [\boldsymbol{u}_1, \boldsymbol{u}_2]$, and expand it into a complete orthonormal basis $\boldsymbol{U} = [\boldsymbol{U}_1, \boldsymbol{U}_2]$.
For any vector $\boldsymbol{x}$ orthogonal to both $\boldsymbol{j}$ and $\boldsymbol{k}$, we have $\boldsymbol{A}\boldsymbol{x} = \boldsymbol{x}$ and $\boldsymbol{x}^{\top}\boldsymbol{A} = \boldsymbol{x}^{\top}$. Thus $\boldsymbol{A}\boldsymbol{U}_2 = \boldsymbol{U}_2$ and $\boldsymbol{U}_2^{\top}\boldsymbol{A} = \boldsymbol{U}_2^{\top}$, so
\begin{equation}\boldsymbol{U}^{\top}\boldsymbol{A}\boldsymbol{U} = \begin{pmatrix}\boldsymbol{U}_1^{\top} \\ \boldsymbol{U}_2^{\top}\end{pmatrix}\boldsymbol{A}\begin{pmatrix}\boldsymbol{U}_1 & \boldsymbol{U}_2\end{pmatrix} = \begin{pmatrix}\boldsymbol{U}_1^{\top}\boldsymbol{A}\boldsymbol{U}_1 & \boldsymbol{U}_1^{\top}\boldsymbol{A}\boldsymbol{U}_2 \\ \boldsymbol{U}_2^{\top}\boldsymbol{A}\boldsymbol{U}_1 & \boldsymbol{U}_2^{\top}\boldsymbol{A}\boldsymbol{U}_2\end{pmatrix} = \begin{pmatrix}\boldsymbol{U}_1^{\top}\boldsymbol{A}\boldsymbol{U}_1 & \boldsymbol{0} \\ \boldsymbol{0} & \boldsymbol{I}_{d-2}\end{pmatrix}\end{equation}
Since orthogonal transformations do not change singular values, the non-trivial singular values of $\boldsymbol{A}$ must come from the matrix $\boldsymbol{U}_1^{\top}\boldsymbol{A}\boldsymbol{U}_1$. Calculating it directly yields:
\begin{equation}\boldsymbol{B}\triangleq\boldsymbol{U}_1^{\top}\boldsymbol{A}\boldsymbol{U}_1 = \begin{pmatrix} 1-\eta c & 0 \\ -\eta \sqrt{1-c^2} & 1 \end{pmatrix}\end{equation}
There is a clever direct calculation method for the singular values of a $2\times 2$ matrix. Let the two singular values be $\sigma_1 \geq \sigma_2$. Using the relationship between singular values, the Frobenius norm, and the determinant, we get:
\begin{gather}\underbrace{\sigma_1^2 + \sigma_2^2 = \Vert\boldsymbol{B}\Vert_F^2 = \eta^2 - 2\eta c + 2,\qquad \sigma_1 \sigma_2 = |\det(\boldsymbol{B})| = |1 - \eta c|} \\
\Downarrow \notag \\
\sigma_1 \pm \sigma_2 = \sqrt{\sigma_1^2 + \sigma_2^2 \pm 2\sigma_1\sigma_2} = \sqrt{\eta^2 - 2\eta c + 2 \pm 2|1 - \eta c|}\end{gather}
We are primarily concerned with $\sigma_1$, which equals the average of the $\pm$ terms. We can remove the absolute value signs (as both positive and negative cases are accounted for), and finally simplifying gives:
\begin{equation}\sigma_1 = \frac{\eta + \sqrt{\eta^2 - 4\eta c + 4}}{2}\end{equation}
We have already shown that this is not less than 1, so $\sigma_1$ is the spectral norm. Q.E.D.
Summary
This article analyzed the stability issues that may arise when GDN2 generalizes the update matrix from $\boldsymbol{I}-\eta\boldsymbol{k}\boldsymbol{k}^{\top}$ to $\boldsymbol{I}-\eta\boldsymbol{j}\boldsymbol{k}^{\top}$, and discussed corresponding strategies.