Natural Language Inference by Integrating Deep and Shallow Representations with Knowledge Distillation

Natural language understanding models often make use of surface patterns or idiosyncratic biases in a given dataset to make predictions pertaining to natural language inference (NLI) tasks. Unfortunately, this renders the resulting model vulnerable to out-of-distribution datasets to which the identified features are inapplicable, thereby leading to erroneous results. Many of the methods developed for out-of-distribution datasets have proven effective; however, they also tend to impose a tradeoff in performance when applied to in-distribution datasets. In this paper, we use a teacher model providing knowledge for the student ensemble model as basic information for training. The student ensemble model then integrates information of deep and shallow representations to extend learning performance to a wide range of examples. The evaluation demonstrates that the proposed model outperformed state-of-the-art models when applied to in-distribution as well as out-of-distribution datasets.