Which Unsupervised Semantic Similarity Method is the Strongest? We Conducted a Comprehensive Evaluation

By 苏剑林 | April 11, 2021

In January, I wrote "You Might Not Need BERT-flow: A Simple Linear Transformation Comparable to BERT-flow", pointing out that the state-of-the-art (SOTA) unsupervised semantic similarity model, BERT-flow, can actually be matched by a simple linear transformation (whitening operation, or BERT-whitening). Subsequently, we further refined our experimental results and wrote a paper titled "Whitening Sentence Representations for Better Semantics and Faster Retrieval". This blog post will summarize the content of that paper and provide supplementary evaluations on five Chinese semantic similarity tasks, involving over 600 experimental results.

Github Link: https://github.com/bojone/BERT-whitening

Method Overview

The logic behind BERT-whitening is simple: after obtaining the sentence vectors $\{x_i\}_{i=1}^N$ for each sentence, a whitening operation (essentially PCA) is applied to these vectors so that the mean of each dimension is 0 and the covariance matrix is the identity matrix. Finally, the top $k$ principal components are retained. The process is illustrated below:

Theoretically, BERT-whitening can be viewed as the simplest implementation of BERT-flow. As noted in previous blogs, this simple implementation is sufficient to match standard BERT-flow models and sometimes even outperform them. Additionally, while transforming, BERT-whitening ranks features by importance, allowing us to reduce the dimensionality of the sentence vectors to improve retrieval speed. Experimental results show that in most tasks, dimensionality reduction not only avoids performance degradation but often leads to performance improvements.

English Tasks

First, we present the testing results of BERT-whitening on English tasks across three main charts, providing a strict comparison with BERT-flow.

Pure Unsupervised

The first table shows the results using sentence vectors extracted directly from pre-trained BERT in a completely unsupervised manner. The BERT-flow paper confirmed that without any post-processing, the best pooling method for BERT-based sentence vectors is the average of all tokens in the first and last layers, denoted as first-last-avg (the BERT-flow paper mistakenly identified this as the average of the last two layers and labeled it last2avg, but it was actually the first and last layers). Therefore, all subsequent results use first-last-avg as the baseline for adding either flow or whitening.

NLI Supervised

The second table shows results for sentence vectors extracted from the Sentence-BERT model (SBERT), which was fine-tuned on NLI datasets. In this case, first-last-avg remains the best base, so it is used as the starting point for flow and whitening. The NLI dataset is a natural language inference dataset, which is similar but not identical to semantic similarity. It serves as supervised pre-training for semantic similarity tasks, but since semantic similarity data is not used directly, it is still considered unsupervised relative to the target task.

Dimension-Effect

In these two tables, bold values indicate the best results. A green arrow $\color{green}{\uparrow}$ indicates that BERT-whitening outperformed BERT-flow under the same conditions, while a red arrow $\color{red}{\downarrow}$ indicates the opposite. More green arrows mean BERT-whitening is generally more effective. The numbers 256 and 384 following "whitening" refer to the retained dimensions after reduction. From these tables, it is clear that BERT-whitening generally outperforms BERT-flow, achieving SOTA on most tasks. In many cases, dimensionality reduction further improves performance.

To further examine the impact of dimensionality reduction, we plotted the relationship between retained dimensions and performance:

The indices above show the performance curves of different models across various tasks after whitening as the number of retained dimensions changes. Markers indicate the dimensions where peak performance was reached. It's visible that for every task, the optimal performance is not reached at the full dimension, implying that dimensionality reduction can bring performance gains. Notably, many tasks maintain or even increase performance when reduced to 1/8 of the original size, demonstrating BERT-whitening's engineering value as lower dimensions significantly accelerate retrieval speed.

Chinese Tasks

Adhering to the philosophy that "a model not tested on Chinese has no soul," I organized several Chinese semantic similarity datasets to evaluate various Chinese pre-trained models, pooling methods, and whitening configurations. The results are summarized here for comparison.

Evaluation Conditions

This evaluation involves 11 models, 5 datasets, and 4 pooling methods. For each combination, we test three post-processing methods: "no whitening," "whitening," and "whitening with dimensionality reduction." This results in approximately $$11 \times 5 \times 4 \times 3 = 660$$ experimental results. (The actual number is slightly lower as some pooling methods aren't applicable to certain models). Due to the significantly higher computational cost of BERT-flow compared to BERT-whitening, we did not reproduce BERT-flow results for Chinese, but based on the English results, the two are usually close, with BERT-whitening often having the edge.

The evaluation metric used is the Spearman correlation coefficient, a ranking metric similar to AUC that relies only on the order of predicted scores and is threshold-independent. It is particularly suitable for STS-B, where labels are continuous values (1–5) rather than 0/1. For those accustomed to accuracy, a rough heuristic is "accuracy ≈ 0.5 + spearman / 2".

The 11 models are as follows:

BERT: Google's official Chinese BERT base. Link;

RoBERTa: HFL's roberta_wwm_ext base. Link;

NEZHA: Huawei's BERT base with relative position encoding (wwm). Link;

WoBERT: Word-based BERT (Plus version used here). Link;

RoFormer: BERT with Rotary Position Embedding (RoPE). Link;

BERTlarge: Tencent UER's BERT large. Link;

RoBERTalarge: HFL's roberta_wwm_ext large. Link;

NEZHA-large: Huawei's NEZHA large (wwm). Link;

SimBERT: BERT base trained on similar sentence pairs. Link;

SimBERTsmall: Smaller version of SimBERT. Link;

SimBERTtiny: Tiny version of SimBERT. Link.

The 5 tasks are:

ATEC: ATEC Semantic Similarity Competition dataset (Financial/Customer Service). Link;

BQ: HIT BQ Corpus (Banking/Finance questions). Link;

LCQMC: HIT LCQMC (General domain question matching). Link;

PAWSX: Google's PAWS-X dataset (Paraphrase identification with high lexical overlap, very difficult for unsupervised methods). Link;

STS-B: Similarity calculation between two sentences (translated from English with manual correction). Link.

The 4 pooling methods are:

P1: Use the [CLS] vector from the encoder's last layer;

P2: Use the vector corresponding to the Pooler (used for the NSP task), which includes a linear transformation after [CLS];

P3: Average all vectors from the encoder's last layer;

P4: Average all vectors from both the first and last layers of the encoder.

Summary of Results

All results are summarized in the following three tables. Each element is in the form $a / b / c$, representing "no whitening" score $a$, "whitening" score $b$, and "whitening with dimensionality reduction" score $c$. If $b \geq a$, $b$ is green; otherwise, it is red. If $c \geq a$, $c$ is green; otherwise, it is red. "Appropriate reduction" means reducing to 256 dimensions for base models, 384 for large models, and 128 for tiny/small models.

The first table compares 6 base models (WoBERT and RoFormer lack NSP weights, so P2 is omitted):

The second table compares 3 large models:

The third table compares different sizes of SimBERT:

Experimental Conclusions

Similar to the English tables, green indicates that the whitening operation improved sentence vector quality, while red indicates a decrease. More green suggests the method is generally effective. From these charts, we can draw several conclusions:

1. Results on Chinese tasks are much more complex and irregular than English. For instance, in English, P4 is almost always better than other pooling methods, and large models are generally better than base models, but these patterns are not obvious in Chinese tasks.

2. Except for SimBERT, there is generally more green than red, indicating that whitening usually has a positive effect on sentence vector quality. Specifically, the third value $c$ (with reduction) is green significantly more often than the second value $b$, showing that dimensionality reduction further improves performance. Whitening is thus a truly "faster and better" algorithm.

3. In the BQ task, whitening almost always led to a decrease. This is similar to the SICK-R task in English, suggesting there is "no free lunch": on some tasks, the "isotropy" assumption may fail, and neither BERT-whitening nor BERT-flow will help.

4. SimBERT is the SOTA for all tasks except PAWS-X. Of course, SimBERT was trained using supervised semantic similarity data (though ideally, the training data shouldn't overlap with evaluation tasks), so it's not a perfectly fair comparison. However, since SimBERT is open-source, it serves as a strong baseline.

5. Applying whitening to SimBERT either degrades performance or provides negligible improvement. This suggests that for sentence vectors trained with supervised methods, further whitening is unnecessary and unlikely to yield gains.

6. PAWS-X is indeed very difficult, and unsupervised semantic similarity still has a long way to go...

Article Summary

This post presented a comprehensive evaluation of unsupervised semantic similarity methods on both English and Chinese tasks. For English, we reiterated results from our BERT-whitening paper, including a direct comparison with BERT-flow. For Chinese, we collected 5 tasks and 11 pre-trained models, testing over 600 combinations of pooling and post-processing methods to provide a clear reference for comparison.

In short: BERT-whitening matches the current unsupervised SOTA, while SimBERT stands as a high-performance open-source baseline for Chinese semantic similarity.