An Experimental Investigation of Discourse Expectations in Neural Language Models
- 주제(키워드) BERT , coreference resolution , discourse expectation , GPT-2 , implicit causality bias , LSTM , neural language model , next sentence prediction , surprisal
- 설명문(URI) https://www.scopus.com/record/display.uri?eid=2-s2.0-85141426090&origin=resultslist&sort=plf-f&src=s&st1=An+experimental+investigation+of+discourse+expectations+in+neural+language+models&sid=8cbe4a3a35baa7dccb2e3543f638f205&sot=b&sdt=b&sl=88&s=TITLE%28An+experimental+investigation+of+discourse+expectations+in+neural+language+models%29&relpos=0&citeCnt=0&searchTerm=
- 설명문(URI) https://www.kci.go.kr/kciportal/ci/sereArticleSearch/ciSereArtiView.kci?sereArticleSearchBean.artiId=ART002890062
- 등재 SCOPUS, KCI등재
- 발행기관 The Korean Association for the Study of English Language and Linguistics
- 발행년도 2022
- URI http://www.dcollection.net/handler/ewha/000000204060
- 본문언어 영어
- Published As https://doi.org/10.15738/kjell.22..202210.1101
- 저작권 이화여자대학교 논문은 저작권에 의해 보호받습니다.
초록/요약
The present study reports on three language processing experiments with most up-to-date neural language models from a psycholinguistic perspective. We investigated whether and how discourse expectations demonstrated in the psycholinguistics literature are manifested in neural language models, using the language models whose architectures and assumptions are considered most appropriate for the given language processing tasks. We first attempted to perform a general assessment of a neural model’s discourse expectations about story continuity or coherence (Experiment 1), based on the next sentence prediction module of the bidirectional transformer-based model BERT (Devlin et al. 2019). We also studied language models’ expectations about reference continuity in discursive contexts in both comprehension (Experiment 2) and production (Experiment 3) settings, based on so-called Implicit Causality biases. We used the unidirectional (or left-to-right) RNN-based model LSTM (Hochreiter and Schmidhuber 1997) and the transformer-based generation model GPT-2 (Radford et al. 2019), respectively. The results of the three experiments showed, first, that neural language models are highly successful in distinguishing between reasonably expected and unexpected story continuations in human communication and also that they exhibit human-like bias patterns in reference expectations in both comprehension and production contexts. The results of the present study suggest language models can closely simulate the discourse processing features observed in psycholinguistic experiments with human speakers. The results also suggest language models can, beyond simply functioning as a technology for practical purposes, serve as a useful research tool and/or object for the study of human discourse processing.
more