Is it Still SOTA for Chinese Tasks? We Supplemented SimCSE with Some Experiments

By 苏剑林 | April 26, 2021

At the beginning of this year, inspired by BERT-flow, I conceived the "BERT-whitening" method, which temporarily became the new SOTA for semantic similarity (see "You Might Not Need BERT-flow: A Simple Linear Transformation Competes with BERT-flow"; the paper is "Whitening Sentence Representations for Better Semantics and Faster Retrieval"). However, the "good times did not last long." Shortly after BERT-whitening was submitted to Arxiv, at least two new papers appeared on Arxiv with results significantly superior to BERT-whitening.

The first is "Generating Datasets with Pretrained Language Models", which uses templates to unsupervisedly construct data pairs from GPT2_XL to train similarity models. While I find it clarifying and effective, the reproduction cost and variance are somewhat large. The other is the protagonist of this article, "SimCSE: Simple Contrastive Learning of Sentence Embeddings". The proposed SimCSE significantly outperforms BERT-flow and BERT-whitening on English data, and the method is remarkably simple.

So, is SimCSE equally effective in Chinese? Can it substantially improve Chinese semantic similarity results? This article aims to provide some supplementary experiments.

Open Source Address: https://github.com/bojone/SimCSE

SimCSE

First, let's briefly introduce SimCSE. In fact, SimCSE can be seen as a simplified version of SimBERT (for SimBERT, please read "Having Your Cake and Eating It Too: SimBERT Model Integrating Retrieval and Generation"). Its simplifications are as follows:

1. SimCSE removes the generative part of SimBERT, retaining only the retrieval model;

2. Since SimCSE lacks labeled data, it treats each sentence itself as a similar sentence for input.

In short, it essentially uses (self, self) as a positive example and (self, others) as negative examples to train a contrastive learning model. Of course, it is not quite that simple. If exactly identical samples were used as positive pairs, the generalization ability would be greatly compromised. Generally, we use data augmentation techniques to introduce differences between the two positive samples. However, data augmentation in NLP is a difficult problem in itself. SimCSE proposes an extremely simple solution: use Dropout directly as data augmentation!

Specifically, $N$ sentences pass through an Encoder with Dropout to obtain vectors $\boldsymbol{h}^{(0)}_1, \boldsymbol{h}^{(0)}_2, \dots, \boldsymbol{h}^{(0)}_N$. Then, this batch of sentences passes through the Encoder again (this time with a different random Dropout) to obtain vectors $\boldsymbol{h}^{(1)}_1, \boldsymbol{h}^{(1)}_2, \dots, \boldsymbol{h}^{(1)}_N$. We can treat $(\boldsymbol{h}^{(0)}_i, \boldsymbol{h}^{(1)}_i)$ as a pair of (slightly different) positive examples. The training objective is then:

English Effect

The original paper contains quite extensive (English) experiments; readers can read the original text carefully. However, it should be noted that the evaluation metrics in the main text tables of the original paper are inconsistent with those of BERT-flow and BERT-whitening. The tables with consistent metrics are in the appendix:

SimCSE compared with BERT-flow and BERT-whitening

Regardless of the comparison, SimCSE is clearly superior to BERT-flow and BERT-whitening. So, is this advantage of SimCSE universal? Does it have the same advantage in Chinese? Let's proceed with our experiments.

Experimental Configuration

Our Chinese experiments are basically aligned with "Which Unsupervised Semantic Similarity is Strongest? We Conducted a Comprehensive Evaluation", including the five tasks previously tested, four types of Pooling, and all base, small, and tiny versions of the models. The large version was not run because it triggered OOM errors under the same configuration.

Open Source Address: https://github.com/bojone/SimCSE

After parameter tuning, I found that the optimal parameters for Chinese tasks are not entirely consistent with those in the original paper. The specific differences are as follows:

1. The original paper used batch_size=512; here, it is batch_size=64 (I really can't afford such an expensive batch_size);

2. The learning rate in the original paper was 5e-5; here, it is 1e-5;

3. The optimal dropout ratio in the original paper was 0.1; here, it is 0.3;

4. The unsupervised SimCSE in the original paper was trained on additional data; here, 10,000 task data items were randomly selected for training;

5. The original unsupervised training included an MLM task; here, only SimCSE training is performed.

Clarifying the last point further: the unsupervised SimCSE in the original paper was trained on 1 million sentences selected from Wikipedia. For the Chinese experiments, for convenience and fairness in comparison, I directly used task data for training (only using the sentences, no labels, still unsupervised). However, except for PAWSX, for the other four tasks, it is not necessary to use all the data for training. Testing revealed that randomly selecting 10,000 training samples and training for one epoch is sufficient to achieve optimal results (performance degrades with both more or fewer samples).

Chinese Effect

All experimental results for SimCSE in Chinese are as follows:

Each unit of data is in the form of "a/b/c", where 'a' is the original result without any processing, 'b' is the result of BERT-whitening (without dimensionality reduction), and 'c' is the result of SimCSE. If c > b, then c is shown in green, otherwise in red. In other words, more green indicates that SimCSE is better than BERT-whitening. For other experimental details, please refer to the original code and "Which Unsupervised Semantic Similarity is Strongest? We Conducted a Comprehensive Evaluation".

Note that due to Dropout and the random sampling of 10,000 samples during training, the results have a certain degree of randomness. Rerunning the code will definitely result in fluctuations in the indicators; please be aware of this.

Some Conclusions

From the experimental results, it can be seen that except for the "outlier" PAWSX, SimCSE has an overwhelming advantage compared to BERT-whitening, sometimes performing more than 10 percentage points better. On BQ, SimCSE even outperforms the supervised-trained SimBERT. Furthermore, models already supervised-trained like SimBERT can gain further improvements, all of which demonstrate its power. (As for why PAWSX is "odd," a simple analysis was already provided in the article "Which Unsupervised Semantic Similarity is Strongest? We Conducted a Comprehensive Evaluation".)

At the same time, we can see that under SimCSE, the first-last-avg Pooling method, which performed better in BERT-flow and BERT-whitening, no longer has any advantage. Instead, taking the [CLS] vector directly is better. Surprisingly, however, the performance of the Pooler (taking [CLS] and adding a Dense layer) is relatively poor, which is truly confusing.

Since BERT-whitening is only a linear transformation, I also experimented with whether SimCSE alone could replicate the effect of this linear transformation. Specifically, I fixed the weight of the Encoder, then attached a Dense layer without activation functions, and then trained only the final Dense layer with SimCSE as the target. The results showed that SimCSE in this case was not as good as BERT-whitening. This implies that SimCSE must fine-tune the Encoder to be effective. It also suggests that BERT-whitening may contain elements that SimCSE lacks; perhaps combining the two in some way would yield better results (under formulation...).

Related Work

A simple survey revealed that several papers using the "self versus self as positive samples" idea have appeared recently. Besides SimCSE, there are "Augmented SBERT: Data Augmentation Method for Improving Bi-Encoders for Pairwise Sentence Scoring Tasks" and "Semantic Re-tuning with Contrastive Tension", which are extremely similar. In fact, I had similar ideas myself but didn't think they would actually work (so I didn't conduct experiments), and I didn't realize the key point was Dropout. It seems one must experiment more!

Summary

This article shares my Chinese experiments on SimCSE. The results show that on many tasks, SimCSE is indeed quite excellent and significantly outperforms BERT-whitening.