By 苏剑林 | November 1, 2026
In the article "Linearizing Softmax Attention into Gated DeltaNet", we provided a reference process for linearizing Softmax Attention into Gated DeltaNet. It demonstrated how we can start from Softmax Attention, gradually introduce approximations, and finally arrive at a linear model. While this result might not necessarily be highly practical, it helps us better understand the approximations and costs behind Linear Attention.
This article provides a new linearization route. It starts from distributional generalization and introduces a three-step averaging operation to establish an approximate connection between Attention and TTT. Under this broad framework, DeltaNet and MesaNet naturally emerge as special cases.
Generalization of Definition
Given vectors $\boldsymbol{q},\boldsymbol{k}_1,\cdots,\boldsymbol{k}_n,\boldsymbol{v}_1,\cdots,\boldsymbol{v}_n$, Softmax Attention is defined as
\begin{equation}\boldsymbol{o} = \sum_{i=1}^n a_i \boldsymbol{v}_i, \qquad a_i = \frac{e^{\boldsymbol{q}\cdot \boldsymbol{k}_i}}{Z},\qquad Z = \sum_{i=1}^n e^{\boldsymbol{q}\cdot \boldsymbol{k}_i}\end{equation}
Each $\boldsymbol{q}$ must be calculated and summed with all preceding $(\boldsymbol{k}_i, \boldsymbol{v}_i)$ pairs, which is the root cause of its quadratic complexity and the main bottleneck in the decoding stage (KV Cache). Most linearized Attentions can essentially be boiled down to different ways of compressing the KV Cache.
The starting point of this article is also KV Cache compression, but at the beginning, we first generalize Softmax Attention:
\begin{equation}\boldsymbol{o} = \sum_{i=1}^n a_i\int p(\boldsymbol{v}|\boldsymbol{k}_i) \boldsymbol{v} d\boldsymbol{v} = \sum_{i=1}^n a_i \mathbb{E}_{p(\boldsymbol{v}|\boldsymbol{k}_i)}[\boldsymbol{v}]\end{equation}
In other words, instead of providing $\boldsymbol{v}_i$ directly, we rewrite it as the expectation of the conditional distribution $p(\boldsymbol{v}|\boldsymbol{k}_i)$. This is the first meaning of the "path of averaging." Why is this a generalization? Because by taking the Dirac delta distribution $p(\boldsymbol{v}|\boldsymbol{k}_i)=\delta(\boldsymbol{v} - \boldsymbol{v}_i)$, the above formula recovers the original Softmax Attention. Therefore, the above formula theoretically encompasses a broader class of functions.
Distribution Estimation
Taking $p(\boldsymbol{v}|\boldsymbol{k}_i)=\delta(\boldsymbol{v} - \boldsymbol{v}_i)$ is essentially memorizing all $(\boldsymbol{k}_i, \boldsymbol{v}_i)$ pairs by heart. Since the theme of this article is compression, we cannot retain this "hard-memorization" approach and instead switch to some computable distribution. For simplicity, we choose a conditional normal distribution:
\begin{equation}p(\boldsymbol{v}|\boldsymbol{k}_i) = \mathcal{N}(\boldsymbol{v};\boldsymbol{\mu}_{\boldsymbol{W}}(\boldsymbol{k}_i), \sigma^2\boldsymbol{I})\end{equation}
Where $\boldsymbol{\mu}_{\boldsymbol{W}}(\boldsymbol{k}_i)$ is a mean prediction model with $\boldsymbol{k}_i$ as input and $\boldsymbol{W}$ as parameters, and $\sigma$ is a preset hyperparameter. In theory, we could also replace the covariance matrix $\sigma^2\boldsymbol{I}$ with a general prediction model $\boldsymbol{\Sigma}_{\boldsymbol{W}}(\boldsymbol{k}_i)$, but as seen from $\mathbb{E}_{p(\boldsymbol{v}|\boldsymbol{k}_i)}[\boldsymbol{v}]$, we ultimately only care about the mean vector, so generalizing the covariance does not bring additional benefits.
Under the current choice, we have
\begin{equation}\boldsymbol{o} = \sum_{i=1}^n a_i \boldsymbol{\mu}_{\boldsymbol{W}}(\boldsymbol{k}_i)\label{eq:o-mu-W}\end{equation}
Assuming the calculation of $\boldsymbol{\mu}_{\boldsymbol{W}}(\boldsymbol{k}_i)$ is efficient, we do not need to cache it; instead, we only need to cache K, and $\boldsymbol{\mu}_{\boldsymbol{W}}(\boldsymbol{k}_i)$ is calculated on the fly. This successfully eliminates the V Cache, halving the KV Cache.
Streaming Update
To estimate $\boldsymbol{W}$, we treat $(\boldsymbol{k}_1,\boldsymbol{v}_1),\cdots,(\boldsymbol{k}_n,\boldsymbol{v}_n)$ as training data arriving sequentially and use an SGD optimizer for streaming updates. For a probabilistic model, the most natural training objective is Maximum Likelihood, which means minimizing the negative log-likelihood:
\begin{equation}-\log p(\boldsymbol{v}_i|\boldsymbol{k}_i) \propto \frac{1}{2}\Vert\boldsymbol{v}_i - \boldsymbol{\mu}_{\boldsymbol{W}}(\boldsymbol{k}_i)\Vert^2 \triangleq \ell(\boldsymbol{W}_{t-1};\boldsymbol{k}_t,\boldsymbol{v}_t)\end{equation}
Performing one step of gradient descent:
\begin{equation}\boldsymbol{W}_t = \boldsymbol{W}_{t-1} - \eta_t \nabla_{\boldsymbol{W}_{t-1}}\ell(\boldsymbol{W}_{t-1};\boldsymbol{k}_t,\boldsymbol{v}_t)\label{eq:ttt-sgd}\end{equation}
This essentially forms an RNN with $\boldsymbol{W}$ as the state variable. It continuously refines $\boldsymbol{W}$ based on current inputs, with the goal of maximizing K's ability to predict V. Thus, it can be said that it compresses the incremental information of V relative to K into the finite-dimensional parameter variable $\boldsymbol{W}$.
This recursive form is actually the same as TTT (Test-Time Training), except our current starting point is compressing the information of V given K to "lighten the load" of the KV Cache, without directly defining full writing and reading processes like TTT.
Two Averaging Steps
Through the above transformation, we have temporarily eliminated V, but K remains, and equation \(\eqref{eq:o-mu-W}\) still actually possesses the form of Softmax Attention, so the complexity has not decreased. To address this, we first use a mean-field approximation:
\begin{equation}\boldsymbol{o} = \sum_{i=1}^n a_i \boldsymbol{\mu}_{\boldsymbol{W}}(\boldsymbol{k}_i) \approx \boldsymbol{\mu}_{\boldsymbol{W}}\Big(\sum_{i=1}^n a_i\boldsymbol{k}_i\Big)\end{equation}
In this way, we transform the core calculation of the remaining part into $\sum_{i=1}^n a_i\boldsymbol{k}_i$, which is a Softmax Attention where K=V. Furthermore, since $a_i\propto e^{\boldsymbol{q}\cdot\boldsymbol{k}_i}$, if we assume that $\boldsymbol{k}_1,\cdots,\boldsymbol{k}_n$ are relatively isotropic, or more accurately, assume that the weighted average of their components perpendicular to $\boldsymbol{q}$ is approximately zero, then we have
\begin{equation}\sum_{i=1}^n a_i\boldsymbol{k}_i \approx \lambda \boldsymbol{q}\end{equation}
Thus, the input to $\boldsymbol{\mu}_{\boldsymbol{W}}(\cdot)$ can be replaced with $\lambda \boldsymbol{q}$, or $\lambda$ can be absorbed directly into $\boldsymbol{q}$, yielding the clean result:
\begin{equation}\boldsymbol{o} = \boldsymbol{\mu}_{\boldsymbol{W}}(\boldsymbol{q})\end{equation}
This produces a readout result completely consistent with TTT. Together with equation \(\eqref{eq:ttt-sgd}\), it forms an RNN model with $\boldsymbol{q},(\boldsymbol{k}_1,\boldsymbol{v}_1)\cdots,(\boldsymbol{k}_n,\boldsymbol{v}_n)$ as inputs and $\boldsymbol{W}$ as the state variable. Full K and V no longer appear; they are gradually compressed into the parameter $\boldsymbol{W}$.
Reflections on Results
Now let's review the entire derivation process and see what exactly was sacrificed and what was gained in the transition from Attention to TTT.
In the first step, we used the average vector model $\boldsymbol{\mu}_{\boldsymbol{W}}(\boldsymbol{k}_i)$ to predict and replace $\boldsymbol{v}_i$. From the perspective of Attention, this step saves the V Cache. In terms of fitting capability, the theoretical capacity of $(\boldsymbol{k}_i, \boldsymbol{\mu}_{\boldsymbol{W}}(\boldsymbol{k}_i))$ is certainly not as good as the hard pairing of $(\boldsymbol{k}_i,\boldsymbol{v}_i)$, but it also enhances the interaction between tokens via streaming updates. Therefore, the author believes it's a toss-up; one isn't necessarily better than the other.
The second step is the mean-field approximation $\sum_{i=1}^n a_i \boldsymbol{\mu}_{\boldsymbol{W}}(\boldsymbol{k}_i) \approx \boldsymbol{\mu}_{\boldsymbol{W}}(\sum_{i=1}^n a_i\boldsymbol{k}_i)$. Mathematically, one might worry about the degree of approximation, but from a whole model architecture perspective, it merely moves the Attention position from the output to the input. Within multi-layer combinations, this shouldn't be a substantial flaw. Thus, the author still considers this a balanced change.
The third step is the weighted average approximation $\sum_{i=1}^n a_i\boldsymbol{k}_i \approx \lambda \boldsymbol{q}$, thereby changing the input of $\boldsymbol{\mu}_{\boldsymbol{W}}(\cdot)$ to $\boldsymbol{q}$. This step completely transforms Attention into a linear complexity RNN. Does K=V Attention always approximate $\lambda\boldsymbol{q}$? Of course not. The most convincing examples are MLA and DSV4, which are essentially MQA with K=V. They clearly cannot be equivalent only to $\boldsymbol{q}$; otherwise, they couldn't model the interaction between tokens.
In summary, the biggest issue with this "path of averaging" might lie in the step of replacing the input with $\boldsymbol{q}$. It is an overly ideal simplification that is difficult to fully satisfy in practice. Therefore, it is necessary to periodically supplement it with Softmax Attention to bridge this gap, which corresponds exactly to the current mainstream Hybrid approach.
Linear Version
In practice, we usually simplify the $\boldsymbol{\mu}_{\boldsymbol{W}}(\boldsymbol{k})$ of TTT to a linear model $\boldsymbol{W}\boldsymbol{k}$, in which case
\begin{equation}\ell(\boldsymbol{W}_{t-1};\boldsymbol{k}_t,\boldsymbol{v}_t) = \frac{1}{2}\Vert\boldsymbol{v}_i - \boldsymbol{W}\boldsymbol{k}\Vert^2\end{equation}
The corresponding streaming update rule is
\begin{equation}\boldsymbol{W}_t = \boldsymbol{W}_{t-1} - \eta_t (\boldsymbol{W}_{t-1}\boldsymbol{k}_t - \boldsymbol{v}_t)\boldsymbol{k}_t^{\top}\end{equation}
After adding $\boldsymbol{o}_t = \boldsymbol{W}_t \boldsymbol{q}_t$, we obtain DeltaNet. Simplifying $\boldsymbol{\mu}_{\boldsymbol{W}}(\boldsymbol{k}_i)$ to a linear function primarily makes the model parallelizable and lets it purely take on the role of Token-Mixer, leaving non-linear parts to MLP layers and layer stacking. Furthermore, for a linear model, the mean-field $\sum_{i=1}^n a_i \boldsymbol{\mu}_{\boldsymbol{W}}(\boldsymbol{k}_i) = \boldsymbol{\mu}_{\boldsymbol{W}}(\sum_{i=1}^n a_i\boldsymbol{k}_i)$ is an exact identity, no longer just an "approximation."
Specifically, linear regression can actually be solved directly for an analytical solution without resorting to SGD:
\begin{equation}\boldsymbol{G}_t = \boldsymbol{G}_{t-1} + \boldsymbol{v}_t\boldsymbol{k}_t^{\top},\quad\boldsymbol{H}_t = \boldsymbol{H}_{t-1} + \boldsymbol{k}_t\boldsymbol{k}_t^{\top},\quad \boldsymbol{o}_t = \boldsymbol{G}_t\boldsymbol{H}_t^{-1}\boldsymbol{q}_t\end{equation}
If we add Decay to $\boldsymbol{G}$ and $\boldsymbol{H}$, we obtain MesaNet. Assuming MesaNet and DeltaNet belong to models with the same upper bound on capability, MesaNet's iteration of $\boldsymbol{G}$ and $\boldsymbol{H}$ is just traditional Vanilla Linear Attention (VaLA), which is simpler than the Delta Rule. The cost is a 50% increase in state variables, which suggests that VaLA only has a chance to catch up with Delta-series models by increasing the state variables.
Article Summary
This article uses various averaging approximations as a thread to connect models like Attention, TTT, DeltaNet, and MesaNet. Along this derivation chain, we can clearly see the costs paid for gains at each step in the transformation from Attention to TTT and various linear models, thereby forming a relatively natural and more universal understanding of the relationships between these models.