By 苏剑林 | June 19, 2026
In this article, we introduce a mathematical operation called "Margin-Enforcing Projection (MEP)". It divides a vector into two parts in a specified way and then requires the margin between these two parts to be at least $m$.
We know that classification tasks aim to select the correct category, so the training objective is usually just "positive class score > negative class score". However, in some scenarios, we not only want the positive class score to exceed the negative class score but also hope it exceeds it by at least a specified margin $m > 0$.
These scenarios mainly fall into two categories. The first is for making classification results more robust, less susceptible to random noise, especially in low-precision inference scenarios; hence, we hope the margin of prediction results is more significant. The second is for cases where we don't actually use a classification model, but rather aim to learn features through classification for final use in retrieval; in this case, if no margin is set, retrieval results near the boundaries are prone to errors.
In fact, these two scenarios matured years ago, especially the second one, with representative examples being the training of feature models for face recognition. Thus, this article is somewhat of an "exhumation". The standard approach to this problem is to design various Margin Losses, such as Hinge Loss, Margin Softmax, AM-Softmax, etc. We have previously introduced these in "Sentence Similarity Model Based on GRU and AM-Softmax" and "From Triangle Inequality to Margin Softmax".
The premise of this article is: if we have an operation that can project prediction scores $\boldsymbol{x}$ into scores $\boldsymbol{y}$ that satisfy the margin requirements, then we can directly use it as a target for the model to learn, such as minimizing $\Vert\boldsymbol{y} - \boldsymbol{x}\Vert_2^2$. This projection operation is the problem we will investigate next.
Let $\boldsymbol{x}=(x_1,x_2,\cdots,x_n), \boldsymbol{z}=(z_1,z_2,\cdots,z_n)$, where $\boldsymbol{x}$ represents the model's predicted scores. For simplicity, we assume the first $k$ scores represent positive scores and the remaining $n-k$ represent negative scores. We define
\begin{equation}\newcommand{argmin}{\mathop{\text{argmin}}}\mathcal{P}_m(\boldsymbol{x})\triangleq \argmin_{\boldsymbol{z}\in\mathbb{R}^n} d(\boldsymbol{z}, \boldsymbol{x})\quad\text{s.t.}\quad \min(\boldsymbol{z}_{\leq k}) - \max(\boldsymbol{z}_{> k}) \geq m\end{equation}
where $\boldsymbol{z}_{\leq k} = (z_1,\cdots,z_k), \boldsymbol{z}_{> k} = (z_{k+1},\cdots,z_n)$, $k$ is an integer such that $0 < k < n$, $m > 0$ is the given margin, and $d(\boldsymbol{z}, \boldsymbol{x})$ is the distance function to be minimized. We have two choices: L1 distance and L2 distance, which will be discussed separately.
This definition is intuitive: under the premise of satisfying the margin requirement, find the vector closest to the predicted scores. This aligns with the general idea of projection operations, so we call it "Margin-Enforcing Projection (MEP)". Using the result of such a projection as a learning target theoretically allows the model to take the easiest and fastest path toward the goal, while also allowing it to "brake" in time once the target is reached, avoiding overtraining.
If $\boldsymbol{x}$ already satisfies $\min(\boldsymbol{x}_{\leq k}) - \max(\boldsymbol{x}_{> k}) \geq m$, then obviously $\boldsymbol{z}^* = \boldsymbol{x}$, which is trivial. Without loss of generality, we assume below that $\min(\boldsymbol{x}_{\leq k}) - \max(\boldsymbol{x}_{> k}) < m$. It is not difficult to see that regardless of whether $d$ is L1 or L2 distance, the optimal solution $\boldsymbol{z}^*$ must be reached at
\begin{equation}\min(\boldsymbol{z}^*_{\leq k}) - \max(\boldsymbol{z}^*_{> k}) = m\end{equation}
Otherwise, we could always make some $z_i$ closer to $x_i$ to reduce the objective value. Based on this observation, let $\min(\boldsymbol{z}^*_{\leq k})=\ell$, then $\max(\boldsymbol{z}^*_{> k}) = \ell - m$. We then have
\begin{equation}\boldsymbol{z}^*_{\leq k} = \max(\boldsymbol{x}_{\leq k}, \ell),\qquad \boldsymbol{z}^*_{> k} = \min(\boldsymbol{x}_{> k}, \ell - m)\end{equation}
At this point
\begin{equation}\begin{aligned}
\boldsymbol{z}^* - \boldsymbol{x} =&\, [\max(\boldsymbol{x}_{\leq k}, \ell) - \boldsymbol{x}_{\leq k}, \min(\boldsymbol{x}_{> k}, \ell - m) - \boldsymbol{x}_{> k}] \\[4pt]
=&\, [\max(\ell - \boldsymbol{x}_{\leq k}, 0), \min(\ell - m - \boldsymbol{x}_{> k}, 0)] \\[4pt]
=&\, [\max(\ell - \boldsymbol{x}_{\leq k}, 0), -\max(\boldsymbol{x}_{> k} + m - \ell, 0)]
\end{aligned}\end{equation}
The next step is to solve for $\ell$ based on the different distances $d$.
First consider the L2 distance, where we have
\begin{equation}\Vert\boldsymbol{z}^* - \boldsymbol{x}\Vert_2^2 = \sum_{i=1}^k\max(\ell - x_i, 0)^2 + \sum_{j=k+1}^n \max(x_j + m - \ell, 0)^2 \triangleq f(\ell)\end{equation}
Taking the derivative gives
\begin{gather}f'(\ell) = 2\sum_{i=1}^k\max(\ell - x_i, 0) - 2\sum_{j=k+1}^n \max(x_j + m - \ell, 0) \\[5pt]
f''(\ell) = 2\#\{\ell > x_i\} + 2\#\{\ell < x_j + m\} \end{gather}
where $\#$ is the count function, with $1\leq i\leq k < j\leq n$. Clearly $f''(\ell) \geq 0$, but this can be strengthened to $f''(\ell) > 0$.
This is because $f''(\ell) = 0$ would mean $\#\{\ell > x_i\}=0$ and $\#\{\ell < x_j + m\}=0$, which implies both $\min(\boldsymbol{x}_{\leq k}) \geq \ell$ and $\max(\boldsymbol{x}_{> k})\leq \ell - m$. This contradicts the assumption $\min(\boldsymbol{x}_{\leq k}) - \max(\boldsymbol{x}_{> k}) < m$. Therefore $f''(\ell) > 0$, meaning $f(\ell)$ is strictly convex. Combined with the continuity of $f(\ell)$ and $f'(\ell)$, and the fact that $f'(-\infty)=-\infty$ and $f'(\infty)=\infty$, it follows that $f(\ell)$ has exactly one minimum point, which must occur at $f'(\ell)=0$.
Since $f'(\ell)$ is a composition of the piecewise linear function $\max(x, 0)$, $f'(\ell)$ is also a piecewise linear function of $\ell$, with boundary points being all $x_i$ and $x_j + m$. To solve $f'(\ell)=0$, we first sort the boundary points $\{x_1,\cdots,x_k,x_{k+1}+m,\cdots,x_n+m\}$ in ascending order to obtain $n-1$ intervals. Within a single interval $[a, b]$, $f'(\ell)$ is a straight line. By traversing all intervals to find the one where $f'(a) \leq 0$ and $f'(b) \geq 0$, we can solve for the zero of the line in that interval.
Next consider the L1 distance, where we have
\begin{equation}\Vert\boldsymbol{z}^* - \boldsymbol{x}\Vert_1 = \sum_{i=1}^k\max(\ell - x_i, 0) + \sum_{j=k+1}^n \max(x_j + m - \ell, 0) \triangleq g(\ell)\end{equation}
Clearly, $g(\ell)$ itself is a piecewise linear function. The minimum of such a function can only be reached at boundary points. Thus, the most naive solution is to iterate through all boundary points $\{x_1,\cdots,x_k,x_{k+1}+m,\cdots,x_n+m\}$ and pick the one that minimizes $g(\ell)$, with a complexity of $\mathcal{O}(n^2)$. However, we can simplify this further. First, taking the derivative gives
\begin{equation}\begin{aligned}
g'(\ell) =&\, \#\{\ell > x_i\} - \#\{\ell < x_j + m\} \\[4pt]
=&\, \#\{\ell > x_i\} + \#\{\ell \geq x_j + m\} - (n - k)
\end{aligned}\end{equation}
The second equality uses the identity $\#\{\ell < x_j + m\} + \#\{\ell \geq x_j + m\} = n - k$. It is now easy to see that $g'(\ell)$ increases monotonically from negative to positive. Of course, $g'(\ell)$ is not continuous, so we cannot guarantee finding a point where $g'(\ell)=0$.
However, if we change $\#\{\ell > x_i\}$ to $\#\{\ell \geq x_i\}$ (the derivative at discontinuous boundary points can be viewed as arbitrary, so this adjustment is permitted), then the condition $g'(\ell)=0$ exactly means "there are exactly $n-k$ boundary points less than or equal to $\ell$". If we further assume all boundary points are distinct, then $\ell$ is exactly the $(n-k)$-th value of all boundary points sorted in ascending order! Therefore, in the L1 scenario, $\ell$ can be obtained with a single sort, which is quite elegant.
Reference implementations for the two versions of MEP are as follows:
import jax
import jax.numpy as jnp
@jax.jit
def l2mep(inputs, mask, margin):
x, m = inputs, margin
u = jnp.where(mask, x, x + m).sort()[:, None]
v = jnp.where(mask, jnp.fmax(u - x, 0), -jnp.fmax(x + m - u, 0)).sum(axis=1)
i = ((v[:-1] < 0) & (v[1:] >= 0)).argmax()
l = (u[i + 1] * v[i] - u[i] * v[i + 1]) / (v[i] - v[i + 1])
return jnp.where(mask, jnp.fmax(x, l), jnp.fmin(x, l - m))
@jax.jit
def l1mep(inputs, mask, margin):
x, m = inputs, margin
u = jnp.where(mask, x, x + m).sort(axis=-1)
l = jnp.take_along_axis(u, (~mask).sum(axis=-1, keepdims=True) - 1)
return jnp.where(mask, jnp.fmax(x, l), jnp.fmin(x, l - m))A brief explanation: In practical scenarios, positive classes are not necessarily arranged in the first $k$ positions, and their number and positions may vary. Therefore, in the implementations above, we use a mask vector to mark positive and negative classes.
The current implementation of l2mep only supports 1D input; if batch dimensions are needed, wrap it with jax.vmap. This algorithm finds the zero-point interval by traversing all intervals, with a step complexity of $\mathcal{O}(n)$ and a total complexity of $\mathcal{O}(n^2)$. To improve efficiency, binary search can be used to find the zero-point interval, reducing it to $\mathcal{O}(n\log n)$.
Since the algorithm for l1mep is simple, the current version already supports arbitrary batch dimensions. Its core operation is sorting, with a complexity of $\mathcal{O}(n \log n)$. Both in terms of simplicity and speed, it is quite ideal. Unless there are other considerations, the L1 version is recommended for practice.
This article introduced the "Margin-Enforcing Projection (MEP)" operation: given a score vector, project it onto the nearest vector that satisfies the condition "the minimum of the positive class is at least $m$ larger than the maximum of the negative class". This provides some new ideas for traditional Margin Learning.