Core-LLM-Bench: A Controlled Neurosymbolic Benchmark for Ontology-Grounded Reasoning in Large Language Models

Tracking #: 1022-2061

Flag : Review Assignment Stage

Authors: 

Julie Loesch
Sara Falahatkar
Nicole Kilk
Rishabh Jakhar
Raghava Mutharaju
Michel Dumontier
Remzi Celebi

Responsible editor: 

Guest Editors Benchmarks 2026

Submission Type: 

Article in Special Issue (note in cover letter)

Full PDF Version: 

Cover Letter: 

Dear Editors, On behalf of my coauthors, I am pleased to submit our manuscript entitled “CORE-LLM-Bench: A Controlled Neurosymbolic Benchmark for Ontology-Grounded Reasoning in Large Language Models” for consideration as a benchmark paper in the Neurosymbolic Artificial Intelligence special issue. This manuscript is a substantially extended version of our CORE-LLM-Bench paper accepted at the Neurosymbolic AI conference. The journal version further develops the benchmark along several methodological and experimental dimensions. Specifically, the extended version incorporates the Pizza ontologies, refines binary-question generation to preserve balanced positive–negative pairs, and improves natural-language and abstracted representations. It also introduces a unified explanation schema for binary and open-ended questions, including complete explanations for multi-answer queries, and revises the measurement of explanation-based reasoning complexity. Furthermore, the manuscript presents a new reasoning-tag-level analysis examining how the symbolic constructs present in minimum complete explanations are associated with model performance across tasks and knowledge representations. The finalized benchmark contains 9,048 ontology-grounded questions evaluated across three knowledge representations and three language models, resulting in 81,432 model observations. Our findings highlight the importance of evaluating ontology-grounded reasoning across multiple dimensions, including task formulation, representation, context depth, semantic abstraction, and symbolic explanation structure, rather than relying on aggregate accuracy alone. To support transparency and reproducibility, CORE-LLM-Bench v1.1.1 has been publicly released through Zenodo, GitHub, and Hugging Face: - Zenodo: https://zenodo.org/records/23138373 - GitHub: https://github.com/jloe2911/CORE-LLM-Bench/releases/tag/v1.1.1 - Hugging Face: https://huggingface.co/datasets/jloe2911/CORE-LLM-Bench/tree/v1.1.1 We believe that the extended benchmark and its explanation-grounded analyses will provide a valuable resource for the neurosymbolic AI community, supporting more systematic evaluation of large language models on verifiable symbolic reasoning tasks. Thank you for considering our manuscript. We look forward to your response. Kind regards, Julie Loesch On behalf of all coauthors

Tags: 

  • Under Review