Towards Learning to Reason: Comparing LLMs with Neuro-Symbolic on Arithmetic Relations in Abstract Reasoning

Tracking #: 794-1785

Flag : Review Assignment Stage

Authors: 

Michael Hersche
Giacomo Camposampiero
Roger Wattenhofer
Abu Sebastian
Abbas Rahimi

Responsible editor: 

Guest Editors NeSy 2024

Submission Type: 

Article in Special Issue (note in cover letter)

Full PDF Version: 

Cover Letter: 

Dear Dr. Besold, I am writing to submit our manuscript, entitled "Towards Learning to Reason: Comparing LLMs with Neuro-Symbolic on Arithmetic Relations in Abstract Reasoning," for consideration to the special issue in the Neurosymbolic AI journal covering the 18th International Conference on Neural-Symbolic Learning and Reasoning (NeSy 2024). In the initial work, "Towards Learning Abductive Reasoning using VSA Distributed Representations," published at NeSy 2024, we presented a neuro-symbolic approach (ARLC) that can learn to reason with distributed vector-symbolic architectures (VSAs) representations and operators. This approach achieved state-of-the-art accuracy on I-RAVEN, a dataset containing Raven's progressive matrices (RPM) tests. The present paper significantly extends on ARLC: (1) Develop new SOTA LLM benchmarks. We benchmarked two prominent LLMs (GPT-4 and Llama03 70B) on the I-RAVEN dataset. Our advanced prompting techniques lead to SOTA LLM accuracy on I-RAVEN. (2) New I-RAVEN-X benchmark. We introduce a new dataset with larger RPM matrices (3x10 instead of 3x3) and configurable dynamic ranges (from 10 up to 1000). This allows us to thoroughly analyze and reveal LLM's weakness in understanding arithmetic relations. (3) Benchmark ARLC on new I-RAVEN-X. We train and evaluate ARLC on I-RAVEN-X. It notably outperforms the LLM baselines on the overall task and in performing arithmetic relations. We are looking forward to your evaluation. Sincerely, Dr. Michael Hersche IBM Research – Zurich Säumerstrasse 4, 8803 Rüschlikon, Switzerland Phone: +41 44 724 8894 michael.hersche@ibm.com

Tags: 

  • Under Review