By 苏剑林 | October 27, 2026
Recently, an increasing number of papers and blogs have begun to analyze and guide model optimization based on parameter symmetry, such as Hyperball, HyperTransfer, and ELR. These works generally revolve around the exploration of the symmetry hypothesis . If we design optimizers to "conform" to the corresponding symmetries, it often makes the training process more robust and reduces the mental burden of hyperparameter tuning.
However, the symmetry hypothesis itself has some controversial aspects. How can we find a more reliable foundation for this path to allow it to proceed more reasonably and further? This article shares the author's reflections on this topic.
In this section, we first organize the logical threads of the three articles mentioned at the beginning.
First, they all share a core assumption: the model satisfies , where is a certain parameter of the model, is any positive number, and is the loss function. Obviously, this assumption implies that the model's performance is independent of the norm of the parameter . Therefore, Hyperball advocates re-normalizing the norm of to a specific value after each update step:
The other two papers propose that we do not need to change the update rules, but since the model performance is independent of the norm, the rate of change of the directional vector of is more fundamental during the training process. Thus, they advocate observing and aligning the AUS (Angular Update Size):
Furthermore, since holds, taking the derivative with respect to on both sides and setting yields , meaning the gradient of is always perpendicular to . Since the update amount is constructed based on the gradient, it is simply assumed that this perpendicularity extends to the update amount, hence , leading to
Compared to , the above formula is additionally divided by . The meaning is the relative update size, which we believe is the more fundamental learning rate, called the "Effective Learning Rate (ELR)". By aligning it, different optimizers can achieve similar performance; even learning rate-related Scaling Laws should be based on it.
However, the most fundamental problem with this entire set of logic is: under current mainstream architectures, " holds" is an assumption that almost completely fails to hold!
Note that our current mainstream model architectures are in the form of In-Norm. Whether paired with regular Residual, MHC, or AttnRes, every sub-layer is , where
can only eliminate the influence of the input scale, but it cannot eliminate the influence of the parameter scale of that specific layer, so is out of the question. In fact, for the mainstream Transformer architecture, almost no parameter can satisfy this scale invariance.
The only exceptions are models with QK-Norm, whose possesses this characteristic. Additionally, since the input to the LM Head also carries RMS Norm, the output remains unchanged if the Embedding layer and the output matrices of all layers are multiplied by a number simultaneously. But these are, after all, isolated and conditional invariances that do not hold universally. Therefore, we say that is an assumption that is difficult to establish, not even approximately.
On the other hand, the conclusions obtained by Hyperball, ELR, and HyperTransfer show a certain degree of effectiveness in practice. This is equivalent to saying "the result is correct, but the process has issues." How can we correct this demonstration process to obtain similar conclusions?
Fortunately, although symmetry like cannot be established, existing model architectures universally satisfy a joint symmetry: for certain parameter pairs , it holds that .
These parameter pairs include:
1. The and of the Attention layer, and the of the GLU layer;
2. If the gating of the GLU layer uses ReLU activation, then and also form such a parameter pair (if using Swish, it can be regarded as approximately holding);
3. Treating as a single large parameter, it forms a parameter pair with the of the current layer's RMS Norm;
4. Similarly, treating as a single large parameter, it forms a parameter pair with the of the current layer's RMS Norm;
5. The parameter of the LM Head's RMS Norm forms such a parameter pair with the parameter matrix of the LM Head.
Points 3 and 4 are particularly noteworthy: here, the entire set of input matrices must be scaled together. Scaling only one of the matrices, while canceling out the scale change of that pair, would cause other parameters in the group to be scaled by an additional factor, and the forward result would usually still change. Furthermore, some joint symmetries can be refined to the vector or even channel level, indicating that scale redundancy is not uncommon in modern models; it just usually depends on the coordination between parameters.
Can the universally existing symmetries introduced in the previous section replace the difficult-to-establish and re-support the conclusions of Hyperball, ELR, etc.?
First, means we can fix the norm of one of the parameters without losing the model's expressive power. Noting that most matrix parameters in the Attention and GLU layers have this joint symmetry with their corresponding parameters, we can consider fixing the norm of the matrix parameters while retaining the trainability of the norm—which is exactly what Hyperball advocates.
However, a detail should be noted here: taking the Attention layer as an example, the joint symmetry of requires treating as a whole and does not involve . But Hyperball's practice is to fix the norm of each parameter separately, so theoretically it still loses some expressive power. Its effectiveness cannot be fully explained by joint symmetry; it suggests more that the model's tolerance is much stronger than we imagine.
Next, by differentiating both sides of with respect to and substituting , we get
This shows that treating as a vector, it remains perpendicular to the gradient . When there are enough joint symmetries (for example, is jointly symmetric with both and ), we assume that this perpendicularity can be decomposed into each vector, i.e., we assume
Thus, the previous analysis regarding AUS and ELR can be replicated. Of course, this involves a long chain of assumptions—from global perpendicularity to local perpendicularity, and from gradient perpendicularity to update perpendicularity—which is not guaranteed to be accurate. Therefore, "Hyperball May Not Be a Free Lunch" explicitly retains the inner product term of the parameter and update amount in the definition of the effective learning rate, which may be a more reasonable choice.
Reviewing the entire derivation chain, we were mainly trying to argue the following points: the norm is not important, AUS is more fundamental, and the equivalence between AUS and ELR. But if we question further, we find that the choice of observation objects—whether norm or AUS—is quite subjective, without explaining what the more fundamental expectation behind it is.
We know that every symmetry represents an invariant property of the model; that is, for forward calculation, the model before and after the transformation represents two mathematically identical functions. For training, we expect two different but forward-equivalent parameterizations to obtain exactly the same training trajectory. This would enhance the transferability of various hyperparameters and reduce the mental burden during training.
For example, suppose there is a model , and then we introduce another model , where . Obviously, these two models are completely equivalent. However, under the same optimizer and hyperparameters, their training trajectories are often not equivalent. For instance, under SGD, we have respectively:
From this, it can be seen that changing from to results in an equivalent forward function, but the latter under SGD is equivalent to scaling the learning rate by times, thus the optimization trajectories are not equivalent. This characteristic is unfriendly to optimization because it requires us not only to ensure the functional equivalence of the model but also to precisely align the parameterization methods to reproduce the model's training process and results.
What kind of update amount is an ideal update amount? Let the general form of the update amount be , where . As seen from the previous section, if the parameter is multiplied by , the gradient will be divided by . At this time, we expect the update amount provided by the optimizer to also be multiplied by , so that the update trajectory remains theoretically unchanged, i.e., satisfying the following covariance:
But this is not enough. We have another basic expectation for the optimizer: if the loss function is multiplied by any positive constant, the optimization trajectory should not change, because the constant theoretically carries no useful information. In this way, in the above formula should be replaceable by any constant. Thus, the complete expected covariance is: for any , it holds that
Dividing both sides by and substituting , we get
Note that when already satisfies , the above formula holds automatically. In other words, when , can be any binary function , so its general solution is
Equation gives the general structure of the solution satisfying the covariance : normalize and respectively, allow them to undergo any operation, and then multiply the result by . From this form, it is easy to see that the simplest way to make SGD satisfy covariance is
For a general optimizer, one can consider replacing with , giving
Since we have explicitly separated and , the here is the effective learning rate mentioned earlier. To this point, we have derived the concept of effective learning rate in a completely different way: it is the simplest way to rewrite an optimizer to satisfy covariance. Or, rather, aligning the effective learning rate to achieve transferability of training trajectories is essentially achieving the covariance of the optimizer.
Note a detail: we wrote the norm as rather than . This means it is not naturally tied to the Euclidean norm or RMS. In fact, substituting any kind of norm can satisfy the expected covariance; even non-norms will work, as long as it is a positive homogeneous mapping from a vector to a scalar. Choosing the Euclidean norm or RMS is more because it is simple to calculate and has intuitive geometric meaning.
Finally, the arbitrariness of in Equation also reminds us: an optimizer that satisfies covariance is not unique; Equation is just one of the simplest-looking ones.
If we revisit the "Making Alchemy More Scientific" series, we find that conclusions similar to Equation appeared long ago. For example, in "Making Alchemy More Scientific (V): Fine-Tuning the Learning Rate Based on Gradients", we derived a classic form of the optimal learning rate when minimizing a convergence upper bound:
Here is the gradient, and is an upper bound on the distance between the parameters and the optimum. Thus, even classical convex-optimization results point toward an optimal learning rate of the form
The proportionality coefficient corresponds to the effective learning rate! The concept of effective learning rate therefore already had precursors. Previously, however, we treated the parameter-norm component as a hyperparameter to search over, without separating it out, and retained only the adaptive learning rate component based on second moments.
It should be noted that, when deriving the relevant convergence results, we usually take to be a constant for ease of analysis: a common upper bound or an initial value, depending on the analytical approach. This is not entirely consistent with the dynamic definition of effective learning rate. In principle, however, more refined assumptions could yield dynamic conclusions more consistent with the current effective learning rate. We will look for an opportunity to discuss these later.
Several different perspectives point toward the effective learning rate, which makes us believe that aligning the effective learning rate might indeed be more fundamental in some sense. Then a natural question arises: how should the scheduling form of the effective learning rate (ELR Schedule) be adjusted?
In articles such as "Making Alchemy More Scientific (VI): Exquisite Top-Down Construction" and "Making Alchemy More Scientific (X): Decomposition and Supplement of Monotonicity Hypothesis", we derived based on certain assumptions that the optimal LR Schedule for optimizers like SGD or adaptive learning rate optimizers like AdaGrad is Linear Decay. We take this as a starting point. Of course, Cosine Decay is also very common in practice, but in the author's experiments, the two performed similarly, so we still take Linear as an example.
For simplicity, we assume the endpoint is 0. Next, we mainly derive for Adam. In "Why is Adam's Update RMS 0.2?" and "Asymptotic Estimation of AdamW's Weight RMS (Part 2)", we derived respectively:
where is the Weight Decay. Thus, a reference ELR Schedule form is
Since we generally take a smaller initialization, i.e., (corresponding to ), the norm tends to increase during training. At this time, the above ELR curve is similar in shape to the reference ELR Schedules given in "From LR to ELR" and "Training nGPT", which are in the form of "higher starting point, faster decline in the early stage, and near-linear in the later stage."
As for optimizers like Muon, although the results are not identical to Adam, we generally believe they have similar asymptotic laws, so the above results can generally be extended to Muon as well.
This article discusses parameter symmetry and effective learning rate, re-examines common logical chains for effective learning rate, introduces more rigorous joint symmetry to correct original derivations, and finally adds a new perspective for understanding based on covariance.