Jigsaw Toxic Comment Kaggle

The multilingual model XLM-Roberta-large (XLM-R) is trained on Train, which has only en comments. Because of the very large number of pretrained parameters present in the model, and because binary classification is a relatively simple task, not much hyperparameter tuning is required. The training is carried out at a constant, small learning rate for a few epochs.

A subset of 159,473 comments from Train are first selected (though it's not clear how they are selected), with each of these translated into tr, pt, ru, fr, it, and es, using the Google Translate API. Then, the XLM-R just trained above is used to predict on each comment (in en), and the prediction is used as pseudo-labels for the comment's 6 translations. The translations of all the comments and their pseudo-labels make up what's referred to as the Combined Distilled dataset.

Essentially, translations in 6 languages have replaced the en comments, and the hard labels (0 and 1) have been replaced with pseudo-labels, model predictions on the en comments.

The same XLM-R is also used to predict on Test, with the output predictions used as pseudo-labels on Test itself. This dataset will be referred to as Pseudo-labelled Test.

Initially, generating pseudo-labels for all comments in Test in one go is only possible because XLM-R is a multilingual model that works for all those languages in Test. And even though here the XLM-R has made predictions for comments in tr, pt, ru, fr, it, and es, it hasn't yet been trained on any examples in those languages, only the English comments in Train.

Unlike in Combined Distilled, there are no translations in Pseudo-labelled Test, so each comment's pseudo-label is unique, a number beteen 0 and 1.

Pseudo-labelled Test is updated through further training, but initially, this is how it's formed.

Having obtained Combined Distilled and Pseudo-labelled Test, they are concatenated into a larger dataset on which multiple rounds of training will be carried out.

In each round of training, either a monolingual model or a multilingual model can be used. When a monolingual model is used, everything is restricted to that model's particular language. For example, if the model used is a fr language model, the first thing to do will be to select only those examples where the comment is in fr and then only use those examples during training. At the end of every epoch, the model's prediction on Test is saved. After having gone through all epochs of training, these saved predictions are combined in a simple mean.

This mean is used to update the pseudo-labels of the current Pseudo-labelled Test. The update is carrid out through an exponentially weighted average: $$ \hat{y} = \alpha \hat{y} + (1 - \alpha) \hat{y}_{run} \quad . $$ \(\hat{y}\) is the current pseudo-label of Pseudo-labelled Test. \(\hat{y}_{run}\) denotes the mean prediction of the current round of training mentioned just above. \(\alpha = 0.5\) is used in this solution. Note again that if the current training round uses a fr model, then only the pseudo-labels of the fr examples are updated, with the rest of the examples staying the same.

After this update, Pseudo-labelled Test can be submitted for evaluation by Kaggle.

Everything described so far in this section constitutes one training round. Successive rounds with different language models are carried out. For a complete list of monolingual transformers used in this solution, see this table. XLM-R is the only multilingual model used.

Note that while Combined Distilled is the same in all rounds, Pseudo-labelled Test changes. As it improves, the next training round will have better data to train on, and the submission will have a higher evaluation score.

In addition to improving the evaluation score by iteratively improving the pseudo-labels on Test, it's possible to combine several existing submissions into one that ought to gain an even higher evaluation score. This is illustrated here for a simple scenario where there are 4 existing submissions, with evaluations scores 0.9934, 0.9935, 0.9936, and 0.9940.

In TABLE 4, these 4 submissions are arranged in ascending order of evaluation score, and the first 2 pseudo-labels, \(\hat{y}_0\) and \(\hat{y}_{1}\), are shown for each submission.

The first thing to do is roughly gauge how the pseudo-labels change with increasing score, by calculating the average difference between adjacent submissions, \(\Delta\). For the first pseudo-label, \(\hat{y}_0\), this is: $$ \Delta = \frac{(0.5 - 0.3) + (0.8 - 0.6)}{2} = 0.2 $$ and for the second pseudo-label, it's: $$ \Delta = \frac{(0.6 - 0.8) + (0.4 - 0.6)}{2} = - 0.2 $$ \(\hat{y}_0\) has an upward trend of 0.2, whilst \(\hat{y}_1\) has a downward trend of 0.2.

This suggests that the score is likely to increase further if \(\hat{y}_{0}\) is further increased and if \(\hat{y}_1\) is further decreased. The exact adjustment is determined by the following expression: $$ \hat{y}_{new} = \left\{ \begin{array}{rl} (1 - \alpha \Delta) \hat{y}_{best} + \alpha\Delta & \text{if } \Delta \ge 0 \\ (1 + \alpha \Delta) \hat{y}_{best} & \text{otherwise} \quad . \end{array} \right. $$ Notice that it limits the adjusted pseudo-label to within the range between 0 and 1, as it needs to be. Applying this expression to the example here, the new value for the first pseudo-label should be: $$ \hat{y}_0 = (1 - 0.2\alpha) \cdot 0.8 + 0.2\alpha = 0.84 $$ and the new value for the second pseudo-label should be: $$ \hat{y}_1 = (1 - 0.2\alpha) \cdot 0.4 = 0.32 $$ As expected, \(\hat{y}_0\) has increased, and \(\hat{y}_1\) has decreased. \(\alpha = 1\) is used here, but in practice, any value between 1 and 2 can be used.

Applying this post-processing technique to submissions has proven to help increase the overall evaluation score on the leaderboard.

Team Lingua Franca are rafiko1 + leecming.