Elements of the Core Inverse Matrix of DeltaNet are Always within [-1, 1]

By 苏剑林 | January 26, 2026

From "A Brief History of Linear Attention: From Imitation, Innovation to Feedback", we can see that the parallel form of DeltaNet involves an inverse matrix of the form $(I + KK^\top \odot M^-)^{-1}$. Recently, reader @Arch123 pointed out that experiments show the elements of this inverse matrix always lie within $[-1, 1]$, and asked if this could be mathematically proven or disproven.

In this article, we will prove that this conclusion holds strictly through two different methods.

Problem Description

First, let's restate the problem accurately. Let the matrix be $K=[k_1, k_2, \dots, k_n]^\top \in \mathbb{R}^{n \times d}$, where each $k_i \in \mathbb{R}^{d \times 1}$ is a column vector with a norm not exceeding 1. $M \in \mathbb{R}^{n \times n}$ is a lower triangular mask matrix, defined as:

$I$ is the identity matrix, and $M^- = M - I$. We want to prove that:

Let $X = I + KK^\top \odot M^-$, and denote its inverse matrix as $Y$. Writing $X$ explicitly gives:

Since the norm of each $k_i$ does not exceed 1, it is obvious that every element of $X$ lies in $[-1, 1]$. We need to prove that every element of $Y = X^{-1}$ also lies in $[-1, 1]$.

Mathematical Induction

Since this inverse matrix appears in DeltaNet, it must have a clear model background. Using this background might help us prove it faster, but we'll save that for the next section. In this section, we treat it purely as a mathematical problem.

Diagonal Properties

We are dealing with a lower triangular matrix. As discussed in "Efficient Inversion Methods for 'Diagonal + Low-Rank' Triangular Matrices", this class of matrices has excellent properties. First, it is closed under matrix multiplication. Second, it satisfies the property "the diagonal of the inverse is the inverse of the diagonal." This can be generalized to block lower triangular matrices: "the diagonal blocks of the inverse are the inverse of the diagonal blocks." From this, we can conclude that $Y[:n-1, :n-1] = (X[:n-1, :n-1])^{-1}$ and $Y[1:, 1:] = (X[1:, 1:])^{-1}$, where slicing is understood as in Numpy.

This property suggests using mathematical induction on $n$. Assuming the conclusion holds for any $(n-1) \times d$ matrix $K$, for matrix $X$, we only need to consider the following block partition:

Using $Y[:n-1, :n-1] = (X[:n-1, :n-1])^{-1}$ and $Y[1:, 1:] = (X[1:, 1:])^{-1}$, we can prove that all elements of $Y$ except for the bottom-left $Y_{n,1}$ lie in $[-1, 1]$. Thus, we only need to prove $Y_{n,1} \in [-1, 1]$.

Adjugate Matrix

To prove $Y_{n,1} \in [-1, 1]$, we consider the explicit representation of $X^{-1}$ using the adjugate matrix. Since $X$ is lower triangular with 1s on the diagonal, its determinant is 1. According to the formula for the inverse via the adjugate matrix, we have:

This matrix is derived from a lower triangular matrix where the super-diagonal elements have become 1. Calculating this determinant is more complex, but not overly so, as the last column has only two non-zero elements. Expanding the determinant along the last column reduces it to the subtraction of two determinants of order $n-1$, which still points toward induction.

Continued Product Form

We can try calculating the first few results manually:

From this, we can conjecture:

I invite readers to prove this conjecture using mathematical induction. Assuming the proof is complete, considering the condition $\|k_i\| \le 1$, we have $\|(I - k_i k_i^\top)x\|^2 = \|x\|^2 + (k_i^\top x)^2 (\|k_i\|^2 - 2) \le \|x\|^2$. Therefore:

This completes the proof that $Y_{n,1} \in [-1, 1]$, and by induction, $Y \in [-1, 1]^{n \times n}$ holds for all $n$.

Dual Computation

The previous direct proof was somewhat "brute-force." Later, members of the FLA group reminded me that we can return to the original context of the inverse matrix—DeltaNet—and perform calculations in two different ways. By comparing the results, we can derive the explicit expression of the inverse matrix, leading to a more concise proof.

Inverse Matrix Form

We know the recursive form of DeltaNet is:

where $S_0 = 0$. Looking at the second equality, let $u_t = v_t - S_{t-1} k_t$. Then:

So,

Defining $U = [u_1, u_2, \dots, u_n]^\top$ and $V = [v_1, v_2, \dots, v_n]^\top$, the above equation is equivalent to $U = V - (KK^\top \odot M^-) U$, or $U = (I + KK^\top \odot M^-)^{-1} V$. This is the origin of the inverse matrix $(I + KK^\top \odot M^-)^{-1}$, which is key to parallel computing in DeltaNet and subsequent works like GDN and KDA.

Recursive Expansion

Now looking at the first equality, expanding it recursively gives:

Here $H_{r \to t} \triangleq (I - k_r k_r^\top) (I - k_{r+1} k_{r+1}^\top) \dots (I - k_t k_t^\top)$, with the convention $H_{t+1 \to t} = I$. Substituting this into the definition of $u_t$:

Comparison of Results

Again letting $Y = (I + KK^\top \odot M^-)^{-1}$, then $U = YV$ implies $u_t = \sum_{i=1}^n Y_{t,i} v_i$. By comparing with the equation above, we can identify:

Therefore, we only need to prove $|k_i^\top H_{i+1 \to t-1} k_t| \le 1$. Given $\|k_i\| \le 1$ and the property $\|(I - k_j k_j^\top)x\| \le \|x\|$ established in the first proof, it follows immediately that:

Thus, the theorem is proven.

Summary

This article provides two proofs for the boundedness of the core inverse matrix elements in DeltaNet.