A sequential and intensive weighted language modeling scheme for multi-task learning-based natural language understanding
- 주제(키워드) Language modeling , Multi-task learning , Natural language understanding , Neural networks , Supervised learning
- 관리정보기술 faculty
- 등재 SCIE, SCOPUS
- 발행기관 MDPI AG
- 발행년도 2021
- 세부유형 Article
- URI http://www.dcollection.net/handler/ewha/000000181733
- 본문언어 영어
- Published As http://dx.doi.org/10.3390/app11073095
초록/요약
Multi-task learning (MTL) approaches are actively used for various natural language processing (NLP) tasks. The Multi-Task Deep Neural Network (MT-DNN) has contributed significantly to improving the performance of natural language understanding (NLU) tasks. However, one drawback is that confusion about the language representation of various tasks arises during the training of the MT-DNN model. Inspired by the internal-transfer weighting of MTL in medical imaging, we introduce a Sequential and IntensiveWeighted Language Modeling (SIWLM) scheme. The SIWLM consists of two stages: (1) Sequential weighted learning (SWL), which trains a model to learn entire tasks sequentially and concentrically, and (2) Intensive weighted learning (IWL), which enables the model to focus on the central task. We apply this scheme to the MT-DNN model and call this model the MTDNN-SIWLM. Our model achieves higher performance than the existing reference algorithms on six out of the eight GLUE benchmark tasks. Moreover, our model outperforms MT-DNN by 0.77 on average on the overall task. Finally, we conducted a thorough empirical investigation to determine the optimal weight for each GLUE task. © 2021 by the authors.
more