Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Год журнала: 2022, Номер unknown

Опубликована: Янв. 1, 2022

Social intelligence and Theory of Mind (TOM), i.e., the ability to reason about different mental states, intents, reactions all people involved, allows humans effectively navigate understand everyday social interactions. As NLP systems are used in increasingly complex situations, their grasp dynamics becomes crucial.In this work, we examine open question modern from an empirical theorybased perspective. We show that one today's largest language models (GPT-3; Brown et al., 2020) lacks kind out-of-the box, using two tasks: SocialIQa (Sap 2019), which measure models' intents participants interactions, ToMi (Le, Boureau, Nickel, measures whether can infer states realities situations.Our results struggle substantially at these tasks, with well-below-human accuracies 55% 60% on ToMi, respectively. To conclude, draw theories pragmatics contextualize shortcoming large models, by examining limitations stemming data, neural architecture, training paradigms. Challenging prevalent narrative only scale is needed, posit person-centric approaches might be more effective towards Mind.

Язык: Английский

---

Benchmarks for Automated Commonsense Reasoning: A Survey

ACM Computing Surveys, Год журнала: 2023, Номер 56(4), С. 1 - 41

Опубликована: Сен. 11, 2023

More than one hundred benchmarks have been developed to test the commonsense knowledge and reasoning abilities of artificial intelligence (AI) systems. However, these are often flawed, many aspects common sense remain untested. Consequently, there is currently no reliable way measuring what extent existing AI systems achieved abilities. This article surveys development uses benchmarks. It enumerates 139 that developed: 102 text-based, 18 image-based, 12 video-based, 7 based in simulated physical environments. gives more detailed descriptions twelve these, three from each category. various methods used construct discusses nature sense, role AI, goals served by constructing benchmarks, desirable features flaws gap concludes with a number recommendations for future benchmarks; most importantly, creators invest work needed ensure benchmark examples consistently high quality.

Язык: Английский

---

How games can make behavioural science better

Nature, Год журнала: 2023, Номер 613(7944), С. 433 - 436

Опубликована: Янв. 17, 2023

Язык: Английский

---

Historical Psychology

Current Directions in Psychological Science, Год журнала: 2023, Номер 32(2), С. 176 - 183

Опубликована: Март 12, 2023

A growing body of evidence suggests that many aspects psychology have evolved culturally over historical time. combination approaches, including experimental data collected the past 75 years, cross-cultural comparisons, and studies immigrants, points to systematic changes in psychological domains as diverse conformity, attention, emotion, morality, olfaction. However, these approaches can go back time only for a few decades typically fail provide continuous measures cultural change, posing challenge testing deeper processes. To tackle this most directly, computational methods emerging from natural language processing be adapted extract information large-scale corpora. Here, we first review benefits science then present three useful classes text-analytic techniques inquiry: dictionary-based methods, distributed-representational human-annotation-based methods. These represent an excellent suite methodologies used examine record “dead minds.” Finally, discuss importance going beyond English-centric text analysis foster more generalizable inclusive human behavior. We propose should incorporate further develop variety reliably quantify processes gave rise contemporary social, political, phenomena.

Язык: Английский

---

Speech and language markers of neurodegeneration: a call for global equity

Brain, Год журнала: 2023, Номер 146(12), С. 4870 - 4879

Опубликована: Июль 27, 2023

Abstract In the field of neurodegeneration, speech and language assessments are useful for diagnosing aphasic syndromes characterizing other disorders. As a complement to classic tests, scalable low-cost digital tools can capture relevant anomalies automatically, potentially supporting quest globally equitable markers brain health. However, this promise remains unfulfilled due limited linguistic diversity in scientific works clinical instruments. Here we argue cross-linguistic research as core strategy counter problem. First, survey contributions study primary progressive aphasia three most prevalent neurodegenerative disorders worldwide—Alzheimer’s disease, Parkinson’s behavioural variant frontotemporal dementia. Second, address two forms unfairness literature: neglect world’s 7000 languages preponderance English-speaking cohorts. Third, review studies showing that dysfunctions given disorder may vary depending on patient’s English speakers offer suboptimal benchmark groups. Finally, highlight different approaches, initiatives research, identifying challenges their deployment. Overall, seek inspire timely actions looming source inequity neurology.

Язык: Английский

---

Event Knowledge in Large Language Models: The Gap Between the Impossible and the Unlikely

Cognitive Science, Год журнала: 2023, Номер 47(11)

Опубликована: Ноя. 1, 2023

Abstract Word co‐occurrence patterns in language corpora contain a surprising amount of conceptual knowledge. Large models (LLMs), trained to predict words context, leverage these achieve impressive performance on diverse semantic tasks requiring world An important but understudied question about LLMs’ abilities is whether they acquire generalized knowledge common events. Here, we test five pretrained LLMs (from 2018's BERT 2023's MPT) assign higher likelihood plausible descriptions agent−patient interactions than minimally different implausible versions the same event. Using three curated sets minimal sentence pairs (total n = 1215), found that possess substantial event knowledge, outperforming other distributional models. In particular, almost always possible versus impossible events ( The teacher bought laptop vs. ). However, show less consistent preferences for likely unlikely nanny tutored boy follow‐up analyses, (i) LLM scores are driven by both plausibility and surface‐level features, (ii) generalize well across syntactic variants (active passive constructions) (synonymous sentences), (iii) some errors mirror human judgment ambiguity, (iv) serves as an organizing dimension internal representations. Overall, our results aspects naturally emerge from linguistic patterns, also highlight gap between representations possible/impossible likely/unlikely

Язык: Английский

---

Establishing psychological universals

Nature Reviews Psychology, Год журнала: 2023, Номер 2(4), С. 199 - 200

Опубликована: Март 3, 2023

Язык: Английский

---

A fine-grained comparison of pragmatic language understanding in humans and language models

Опубликована: Янв. 1, 2023

Pragmatics and non-literal language understanding are essential to human communication, present a long-standing challenge for artificial models. We perform fine-grained comparison of models humans on seven pragmatic phenomena, using zero-shot prompting an expert-curated set English materials. ask whether (1) select interpretations speaker utterances, (2) make similar error patterns as humans, (3) use linguistic cues solve the tasks. find that largest achieve high accuracy match patterns: within incorrect responses, favor literal over heuristic-based distractors. also preliminary evidence sensitive cues. Our results suggest behaviors can emerge in without explicitly constructed representations mental states. However, tend struggle with phenomena relying social expectation violations.

Язык: Английский

---

Language experience predicts music processing in a half-million speakers of fifty-four languages

Current Biology, Год журнала: 2023, Номер 33(10), С. 1916 - 1925.e4

Опубликована: Апрель 26, 2023

Язык: Английский

---

Reproducible research practices and transparency across linguistics

Glossa Psycholinguistics, Год журнала: 2023, Номер 2(1)

Опубликована: Ноя. 9, 2023

Scientific studies of language span across many disciplines and provide evidence for social,  cultural, cognitive, technological, biomedical human nature behavior. As it becomes increasingly empirical quantitative, linguistics has been facing challenges limitations the scientific practices that pose barriers to reproducibility replicability. One the  proposed solutions widely acknowledged replicability crisis implementation transparency practices,  e.g., open access publishing, preregistrations, sharing study materials, data, analyses, performing replications, declaring conflicts interest. Here, we have assessed prevalence these in 600 randomly sampled journal articles from two time points. In line with similar other disciplines, found 35% were published rates protocols below 10%. None reported 1% 10% had conflict interest statements. These not increased noticeably between 2008/2009 2018/2019, pointing remaining slow adoption reproducible research linguistics. To facilitate practices, a range recommendations implementing improving

Язык: Английский

---