Continuous Softmax Attention

By 苏剑林 | October 19, 2026

Softmax Attention is surely no stranger to everyone; it is a weighted mixture operator defined over a finite set of vectors. Recently, the paper 《Continuum Attention》 proposed a very interesting idea—continuizing attention. This reminds me of EFLA, which I introduced earlier in "Why does DeltaNet need L2 Normalization?", which also continuizes a discrete RNN and then re-discretizes it to obtain interesting results, similar to the earlier SSM series of work.

It seems that "continuization-discretization" is a tried-and-true path for exploration. So, what interesting results can we get from continuizing Softmax Attention? Let's appreciate it together.

Problem Description

Standard Softmax Attention is defined as
$$\begin{equation}\boldsymbol{o}_t = \frac{\sum_{i=1}^t e^{\boldsymbol{q}_t\cdot\boldsymbol{k}_i} \boldsymbol{v}_i}{\sum_{i=1}^t e^{\boldsymbol{q}_t\cdot\boldsymbol{k}_i}}\end{equation}$$
The so-called continuization intends to change it to
$$\begin{equation}\boldsymbol{o}_t = \frac{\int_0^t e^{\boldsymbol{q}_t\cdot\boldsymbol{k}_s} \boldsymbol{v}_s ds}{\int_0^t e^{\boldsymbol{q}_t\cdot\boldsymbol{k}_s} ds}\label{eq:csa}\end{equation}$$
where $\boldsymbol{k},\boldsymbol{v}$ at non-integer points is defined by linear interpolation, i.e., for any $s \in [i-1, i], i\in\mathbb{N}_+$, we define
$$\begin{equation}\boldsymbol{k}_s = (i - s)\boldsymbol{k}_{i-1} + (s - i + 1) \boldsymbol{k}_i,\qquad \boldsymbol{v}_s = (i - s)\boldsymbol{v}_{i-1} + (s - i + 1) \boldsymbol{v}_i \end{equation}$$
This definition requires the additional introduction of two vectors $\boldsymbol{k}_0,\boldsymbol{v}_0$ (used for $s\in[0,1]$), the values of which we will discuss at the end.

Note that the definition in Eq. $\eqref{eq:csa}$ is slightly different from the original paper. The original paper, out of symmetry considerations, changed the integration interval to $[1/2, t + 1/2]$, but I feel that while this adjustment is more symmetric, it is less natural for causal scenarios, so this article uses the integration interval $[0,t]$. Of course, the subsequent derivation actually has little to do with the choice of interval; the technique is the same.

Preparatory Work

Eq. $\eqref{eq:csa}$ is just a formal definition; to actually use it, the integral must be explicitly calculated. According to the additivity of integrals, we only need to calculate each segment $\int_i^{i+1}$ separately. Letting $a_{t,i} = \boldsymbol{q}_t\cdot\boldsymbol{k}_i, \delta_{t,i} = a_{t,i} - a_{t,i-1}$, it is not difficult to obtain
$$\begin{align}&\,\begin{aligned} \int_{i-1}^i e^{\boldsymbol{q}_t\cdot\boldsymbol{k}_s} \boldsymbol{v}_s ds =&\, \int_0^1 e^{(1-s) a_{t,i-1} + s a_{t,i}} ((1-s)\boldsymbol{v}_{i-1} + s\boldsymbol{v}_i) ds \\ =&\, \boldsymbol{v}_i\int_0^1 e^{a_{t,i-1} + s \delta_{t,i}} ds + (\boldsymbol{v}_{i-1} - \boldsymbol{v}_i)\int_0^1 e^{a_{t,i-1} + s \delta_{t,i}} (1-s) ds \\[4pt] =&\, e^{a_{t,i-1}}\varphi_1(\delta_{t,i}) \boldsymbol{v}_i + e^{a_{t,i-1}}\varphi_2(\delta_{t,i}) (\boldsymbol{v}_{i-1} - \boldsymbol{v}_i) \end{aligned} \\[10pt] &\,\int_{i-1}^i e^{\boldsymbol{q}_t\cdot\boldsymbol{k}_s} ds = \int_0^1 e^{(1-s) a_{t,i-1} + s a_{t,i}} ds = \int_0^1 e^{a_{t,i-1} + s \delta_{t,i}} ds = e^{a_{t,i-1}}\varphi_1(\delta_{t,i}) \end{align}$$
Here
$$\begin{equation}\varphi_1(x) \triangleq \int_0^1 e^{s x} ds = \frac{e^x - 1}{x},\qquad \varphi_2(x) \triangleq \int_0^1 e^{s x} (1-s) ds = \frac{e^x - 1 - x}{x^2}\end{equation}$$
where the cases for $x=0$ are handled by limits, resulting in $\varphi_1(0)=1,\varphi_2(0)=1/2$ respectively. They also satisfy the following identity relationship
$$\begin{equation}\varphi_1(x) - \varphi_2(x) = \frac{x e^x - e^x + 1}{x^2} = e^x \cdot \frac{e^{-x} - 1 - (-x)}{(-x)^2} = e^x \varphi_2(-x)\end{equation}$$
This identity will be used in the simplification in the next section. Additionally, as seen from the integral definition, $\varphi_1(x)$ and $\varphi_2(x)$ are always positive, so the subsequent logarithm operations will not have undefined issues.

Assembly and Simplification

Substituting the above integrals into Eq. $\eqref{eq:csa}$, we get
$$\begin{equation}\boldsymbol{o}_t = \frac{\sum_{i=1}^t e^{a_{t,i-1}}\varphi_1(\delta_{t,i}) \boldsymbol{v}_i + e^{a_{t,i-1}}\varphi_2(\delta_{t,i}) (\boldsymbol{v}_{i-1} - \boldsymbol{v}_i)}{\sum_{i=1}^t e^{a_{t,i-1}}\varphi_1(\delta_{t,i})}\end{equation}$$
Simplifying the numerator first:
$$\begin{equation}\begin{aligned} &\, \sum_{i=1}^t e^{a_{t,i-1}}\varphi_1(\delta_{t,i}) \boldsymbol{v}_i + e^{a_{t,i-1}}\varphi_2(\delta_{t,i}) (\boldsymbol{v}_{i-1} - \boldsymbol{v}_i) \\ =&\, \sum_{i=1}^t e^{a_{t,i-1}}(\varphi_1(\delta_{t,i})-\varphi_2(\delta_{t,i}))\boldsymbol{v}_i + e^{a_{t,i-1}}\varphi_2(\delta_{t,i}) \boldsymbol{v}_{i-1} \\ =&\, \sum_{i=1}^t e^{a_{t,i-1} + \delta_{t,i}}\varphi_2(-\delta_{t,i})\boldsymbol{v}_i + e^{a_{t,i-1}}\varphi_2(\delta_{t,i}) \boldsymbol{v}_{i-1} \\ =&\, \sum_{i=1}^t e^{a_{t,i}}\varphi_2(-\delta_{t,i})\boldsymbol{v}_i + \sum_{i=0}^{t-1} e^{a_{t,i}}\varphi_2(\delta_{t,i+1}) \boldsymbol{v}_i \\ =&\, e^{a_{t,0}}\varphi_2(\delta_{t,1}) \boldsymbol{v}_0 + e^{a_{t,t}}\varphi_2(-\delta_{t,t})\boldsymbol{v}_t + \sum_{i=1}^{t-1} e^{a_{t,i}} (\varphi_2(-\delta_{t,i}) + \varphi_2(\delta_{t,i+1})) \boldsymbol{v}_i \end{aligned}\end{equation}$$
From this, it can be seen that the numerator has the form $\sum_{i=0}^t e^{a_{t,i} + b_{t,i}} \boldsymbol{v}_i$, where
$$\begin{equation}a_{t,i} = \boldsymbol{q}_t\cdot\boldsymbol{k}_i,\qquad e^{b_{t,i}} = \left\{\begin{aligned} &\,\varphi_2(a_{t,1} - a_{t,0})&\, i = 0 \\ &\,\varphi_2(a_{t,i-1} - a_{t,i}) + \varphi_2(a_{t,i+1} - a_{t,i})&\, 1\leq i\leq t-1 \\ &\,\varphi_2(a_{t,t-1} - a_{t,t})&\, i = t \\ \end{aligned}\right.\end{equation}$$
Note that the denominator is exactly the result of replacing all $\boldsymbol{v}_i$ with $1$, so the denominator is directly $\sum_{i=0}^t e^{a_{t,i} + b_{t,i}}$, thus
$$\begin{equation}\boldsymbol{o}_t = \frac{\sum_{i=0}^t e^{a_{t,i} + b_{t,i}} \boldsymbol{v}_i}{\sum_{i=0}^t e^{a_{t,i} + b_{t,i}}}\end{equation}$$

Analysis of Results

Compared to standard Softmax Attention
$$\begin{equation}\boldsymbol{o}_t = \frac{\sum_{i=1}^t e^{a_{t,i}} \boldsymbol{v}_i}{\sum_{i=1}^t e^{a_{t,i}}}\end{equation}$$
The change brought by continuization is the introduction of additional $\boldsymbol{k}_0,\boldsymbol{v}_0$ and, for each $a_{t,i}$, the addition of a correction term $b_{t,i}$. The extra $\boldsymbol{k}_0,\boldsymbol{v}_0$ is not a very fundamental change; we will discuss it briefly later. Now let's mainly look at $b_{t,i}$.

For a general $1\leq i\leq t-1$, $b_{t,i}$ depends simultaneously on $a_{t,i}$ and its neighboring $a_{t,i-1},a_{t,i+1}$. Formally, this is very similar to adding a Short Conv to the Attention matrix, which becomes even more apparent after performing a Taylor expansion on $b_{t,i}$:
$$\begin{equation}b_{t,i} = \log (\varphi_2(a_{t,i-1} - a_{t,i}) + \varphi_2(a_{t,i+1} - a_{t,i})) = \frac{a_{t,i-1} + a_{t,i+1} - 2a_{t,i}}{6} + \cdots\end{equation}$$
where $a_{t,i-1} + a_{t,i+1} - 2a_{t,i}$ is precisely the second-order difference of $a_{t,i}$, which is a simple Short Conv operation. Therefore, continuous Softmax Attention indeed shares many similarities with "adding Short Conv to the Attention matrix."

As for $\boldsymbol{k}_0,\boldsymbol{v}_0$, it is essentially equivalent to the Sink token we introduce to solve the Attention Sink problem. For continuization, it is mainly a mathematical trick to complete the $t$ intervals. We generally have three choices: all zeros, $\boldsymbol{k}_0=\boldsymbol{k}_1,\boldsymbol{v}_0=\boldsymbol{v}_1$, or trainable. I chose all zeros in the experiments below.

Simple Experiment

$b_{t,i}$ depends simultaneously on $a_{t,i}$ and its adjacent values, which also means it cannot reuse existing Flash Attention. However, the implementation principle is universal, and the theoretical FLOPs are basically the same. I had K3 write a simple Kernel, and the speed was only about 1/4 of Softmax Attention, mainly because there are too many fragmented operations inside, making it difficult to accelerate, but we can still run it first.

The first experimental result is that although continuization introduces a local prior, it cannot completely replace positional encoding. In a full Full Attention architecture, continuous Softmax Attention without RoPE is significantly worse than standard Softmax Attention with RoPE; conversely, adding RoPE can bridge this gap.

Regarding training dynamics, when everyone uses RoPE, continuous Softmax Attention is significantly better than standard Softmax Attention in the early stages, but the gap gradually narrows, and by the end of training, the two basically overlap and the advantage disappears. It is speculated that the core reason is that natural language is not a true time series, so continuization does not bring substantial benefits. Perhaps for real time series, there would be a chance to measure an advantage.

I also conducted experiments switching to the continuous version midway through standard Softmax Attention training and found that the jump in Loss was not high, and it eventually converged to basically the same effect as the standard version.

Summary

This article briefly derives the continuous Softmax Attention, showing that its result is similar to adding a Short Conv operation to the Attention matrix. Although no advantage was found for this continuization in natural language tasks, it remains an interesting result to appreciate.