SimBERTv2 Is Here! The RoFormer-Sim Model Integrating Retrieval and Generation

By 苏剑林 | June 11, 2021

Last year, we released the SimBERT model, which has been one of our most successful open-source models, earning recognition from many readers. Simply put, SimBERT is a model that integrates generation and retrieval. it can be used as a relatively high baseline for sentence vectors and can also be used to automatically generate similar questions, serving as an auxiliary tool for data augmentation—a feature that was quite pioneering.

Recently, using RoFormer as the foundation, we further integrated and optimized SimBERT-related technologies and finally released the upgraded RoFormer-Sim model.

Introduction

RoFormer-Sim is the upgraded version of SimBERT; we can also colloquially refer to it as "SimBERTv2," while "SimBERT" by default refers to the old version. Externally, aside from the base architecture switching to RoFormer, there is no obvious difference between RoFormer-Sim and SimBERT. In fact, their primary differences lie in the training details, which can be compared using two formulas:

$$ \text{SimBERT} = \text{BERT} + \text{UniLM} + \text{Contrastive Learning} $$ $$ \text{RoFormer-Sim} = \text{RoFormer} + \text{UniLM} + \text{Contrastive Learning} + \text{BART} + \text{Distillation} $$

In addition, RoFormer-Sim utilizes more training data and has been extended to general sentence patterns. This means that unlike SimBERT, which was limited to interrogative sentences (questions), RoFormer-Sim can be used for similar sentence generation for general sentences, broadening its application scenarios. Other training details include the use of larger batch sizes and max lengths in RoFormer-Sim, which we will introduce further below.

Open Source Address: https://github.com/ZhuiyiTechnology/roformer-sim

Corpus

The key for both SimBERT and RoFormer-Sim lies in the construction of the training corpus. The training corpus for RoFormer-Sim consists of two parts: 1. Interrogative-type similar sentences; 2. General-type similar sentences. For interrogative similar sentences, we followed the SimBERT approach, collecting similar questions from Baidu Zhidao and further cleaning them via rules; this process is very mature for us. For general similar sentences, we did not have a ready-made source, so we proposed two schemes to construct (pseudo) similar sentence pairs in an unsupervised manner to some extent.

The first scheme is based on the idea that "answers to the same question are similar." If we have existing QA corpora where one question has multiple answers, we can segment each answer into sentences. Then, using an existing similarity function to compare the similarity between answers, we select sentence pairs with similarity exceeding a certain threshold as similar sentence pairs.

The second scheme is based on the idea that "sentences within the same passage are similar." This is even more simple and direct: we segment each passage into sentences and then calculate the similarity between all pairs using an existing similarity function, picking pairs that exceed a threshold. Obviously, the rationale for this scheme is weaker, so the threshold used is higher.

The "existing similarity function" mentioned here is a variant of the Jaccard similarity we used directly. In other words, we only needed a rule-based, character-level similarity. Semantic relevance is obtained through the internal associations within passages and the generalization capability of the pre-trained model itself. Through the first scheme, we constructed approximately 4.5 million (pseudo) similar sentence pairs from several reading comprehension datasets. Through the second scheme, we constructed approximately 4.7 million (pseudo) similar sentence pairs from over 30GB of parallel corpora. The crawled questions reached approximately 30 million similar question groups (one group can form multiple pairs). From this perspective, the number of questions far exceeds general sentences, so we sampled them at a 1:1 ratio to ensure a balanced sample for each sentence type.

Generation

The training method for RoFormer-Sim is basically the same as SimBERT, as shown in the figure below. A slight difference is that to enhance the model's generation capability, during the construction of training data, we also randomly replaced some tokens of the input sentence with [MASK]. This pre-training method was first proposed by BART. Our difference from BART is: BART "inputs a noisy sentence and outputs the original sentence," whereas we "input a noisy sentence and output a similar sentence to the original one." Theoretically, our task is even more difficult.

Schematic diagram of SimBERT training method

There are no particularly good evaluation metrics for generation effects; we can just visually inspect some examples:

Overall, similar augmentation for arbitrary sentence structures has been preliminarily achieved, but the augmentation effect for questions is significantly better than for general sentence types. This is because the quality of the questions in the training corpus is significantly higher than that of general sentence types. Due to the BART-like training, in addition to direct similar sentence generation, we can also manually mask certain parts to let the model diverge and expand on its own, for example:

Please explore more ways to play with this on your own.

Retrieval

Adding general sentence corpora and introducing BART-like training are changes that relatively improved the performance of the generative model. However, we unexpectedly discovered that the performance of the retrieval model (i.e., the sentence encoding model) decreased. A likely reason is that while more corpora and greater noise increased the difficulty for the generative model, regarding contrastive learning, these diverse sentence structures or noisy samples acted as negative samples that were actually easier to distinguish. For example, if a batch contains both interrogative and declarative sentences, the model can easily identify many negative samples through sentence structure (rather than semantics), thereby reducing its ability to understand semantics.

Of course, the core positioning of SimBERT and RoFormer-Sim is as similar sentence augmentation models; the retrieval model is just a "by-product." However, we still hope this "by-product" can be as good as possible. To this end, after RoFormer-Sim training was completed, we further transferred the retrieval performance of SimBERT to RoFormer-Sim through distillation, making the retrieval performance of RoFormer-Sim basically equal to or even better than SimBERT. The distillation method is very simple: assuming for the same batch of sentences, the sentence vectors from SimBERT are $u_1, u_2, \cdots, u_n$, and the sentence vectors from RoFormer-Sim are $v_1, v_2, \cdots, v_n$, we then learn using the loss:

\begin{equation} L_{sim} = \frac{\lambda}{n^2} \sum_{i=1}^n \sum_{j=1}^n (\cos(u_i, u_j) - \cos(v_i, v_j))^2 \end{equation}

Here $\lambda=100$. Of course, to prevent the model from "forgetting" the generative model, the generation loss must be added during distillation, i.e., $L = L_{sim} + L_{gen}$. Distillation for the base version does not require many steps; it can be completed in roughly 5,000 steps.

Similar to "Which Unsupervised Semantic Similarity is Strongest? We Did a Comprehensive Evaluation," we used the same tasks to compare the retrieval effects of SimBERT and RoFormer (where the three data points in each cell represent "no whitening," "with whitening," and "with whitening-256" effects, same as the previous evaluation):

As can be seen from the table, regardless of whether whitening is applied, RoFormer-Sim outperforms SimBERT on most tasks. This shows that RoFormer-Sim's retrieval performance can indeed be improved after distillation, making this "by-product" quite decent.

Using the same method, we also created a "small" version of RoFormer-Sim. In this case, distillation used the base version of RoFormer-Sim as the teacher model, but the number of distillation steps needed was higher (around 500,000). The final results are as follows:

Summary

This article introduced and released our upgraded version of SimBERT—RoFormer-Sim (SimBERTv2). It can be used for both augmenting similar sentences and as a strong baseline for semantic similarity problems. Compared to SimBERT, its biggest feature is extending sentence patterns to general types, no longer limited to similar questions. We welcome readers to explore and share more ways to use it~