To access this work you must either be on the Smith College campus OR have valid Smith login credentials.

On Campus users: To access this work if you are on campus please Select the Download button.

Off Campus users: To access this work from off campus, please select the Off-Campus button and enter your Smith username and password when prompted.

Non-Smith users: You may request this item through Interlibrary Loan at your own library.

Publication Date

2024-5

First Advisor

Jamie C. Macbeth

Document Type

Honors Project

Degree Name

Bachelor of Arts

Department

Computer Science

Keywords

spatial reasoning, large language models, symbolic AI, natural language processing

Abstract

This thesis assesses GPT-4’s reasoning capabilities by probing it with SHRDLU, a computer program known for its reasoning capabilities that understands and interacts with objects in a virtual “blocks” environment. This project developed a comprehensive dataset comprised of numerous SHRDLU dialogs, which allows for a detailed and quantifiable assessment of GPT-4 capabilities, specifically in spatial reasoning involving containment relationships. The project sets up experiments with different prompting in levels of difficulty to test GPT-4’s understanding of spatial concepts, containment relationships, and its ability to reason through complex scenarios involving object manipulation. The findings reveal that GPT-4 performs well with basic tasks but struggles with complex spatial relationships in a long series of manipulations.

Rights

©2024 Kexin Zhao. Access limited to the Smith College community and other researchers while on campus. Smith College community members also may access from off-campus using a Smith College log-in. Other off-campus researchers may request a copy through Interlibrary Loan for personal use.

Language

English

Comments

77 pages: color illustrations, charts. Includes bibliographical references (pages 52-54).

Download

Smith Only:
Off Campus Download

Share

COinS