Document Type
Article
Publication Date
7-2023
Publication Title
Advances in Cognitive Systems
Abstract
The ability to recognize that pairs or sets of language expressions “mean the same thing” is a cognitive task for which meaning representation is clearly a central issue. This paper uses the task of paraphrasing to study meaning representation in a cognitive system. The main claim is that a consequential part of the meaning representation for a natural language expression is a set of language-free structures that are not part of the expression in question. To support this claim, we construct a corpus of paraphrase pairs using a system that has a non-linguistic meaning represen- tation decoupled from the linguistic system that generates natural language from it. This corpus of paraphrase pairs is special in that it represents a full range of syntactic and lexical difference in its constituent sentences. We conduct an extensive analysis that compares the performance of a neural network model and humans on the paraphrase detection task. We find that, unlike humans, the model fails to recognize paraphrases when the sentences use different words and syntactic structures to convey the same meaning. As the neural network model is trained only on linguistic items, the discrepancy points to the existence of a substantial non-linguistic part of meaning formation.
Volume
10
First Page
85
Last Page
102
Rights
© 2023 Cognitive Systems Foundation. All rights reserved.
Recommended Citation
Macbeth, Jamie C.; Chang, Ella; Chen, Jingyu Gin; Hua, Yining; Grandic, Sandra; and Zheng, Winnie X., "Humans Against Large Language Models on Hard Paraphrase Detection Tasks" (2023). Computer Science: Faculty Publications, Smith College, Northampton, MA.
https://scholarworks.smith.edu/csc_facpubs/368
Comments
Archived as published.