Special Call for Neurosymbolic Benchmark Papers

Neurosymbolic Benchmark Papers are short-to-medium-sized papers (typically up to 12 pages) presenting and describing novel neurosymbolic benchmarks with corresponding tasks, metrics, and datasets. They should have the potential to become well-known references for evaluating existing and newly developed neurosymbolic solutions. 

We welcome benchmarks and datasets at different levels of maturity that provide novel scientific results, e.g., a bespoke benchmark that provides new insights into the performance of state-of-the-art tools on a specific task. Besides novel benchmarks, which are of primary interest for this call, we also welcome benchmarks that have already enjoyed significant adoption by third parties in the context of research, industry, or a specific user community (e.g., biomedicine) from which novel scientific results emerge.  

The paper should describe, in concise and clear terms, the motivation and merit for the benchmark as a novel contribution to neurosymbolic AI, highlight the targeted gaps, detail its theoretical framework and practical instantiation, formalize and detail its task, dataset and metrics, report its key statistics and illustrative examples, apply it to evaluate a relevant set of baseline approaches following clear research questions or hypotheses, and reflect on its limitations. The paper should also make it clear what property/ies, tasks, or uses of neurosymbolic systems are being addressed with the benchmark, and describe its content as a guide to its usage in various (possibly unforeseen) applications. As a neurosymbolic contribution, it should be clear how the benchmark, as well as its dataset and metrics, connect to both neural and symbolic considerations. 

Benchmark and/or datasets should follow open-access scientific guidelines, i.e., they should be published using a stable URL (e.g., Zenodo), include a name, version date and number, licensing, availability, topic coverage, source of the data, purpose and method of creation and maintenance, reported usage, etc. Ideally, benchmarks should be accompanied by software (e.g., a Python package, a HuggingFace dataset module, or a web demo) that facilitates their use. 

Benchmark contributions will be evaluated along the following dimensions: (1) Relevance and novelty of the benchmark contribution within neurosymbolic AI, e.g., in capturing NeSy system properties or evaluating a task; (2) Clarity and completeness of the description, in terms of (theoretical) justification for the design and in line with the identified research gaps; (3) Usefulness and impact of the benchmark, e.g., in highlighting strengths and weaknesses of systems; (4) Quality of the benchmark/metrics/dataset, supported through, for example, study execution, human performance or agreement, as well as objective metrics of its diversity and coverage.

Prospective authors should feel free to contact the guest editors with any questions related to this call, e.g., to assess whether a particular topic is suitable in principle.

Deadline

May 15, 2026

Earlier submissions will be processed as they come in. 

Special Call Editors

- Filip Ilievski (Vrije Universiteit Amsterdam, The Netherlands)

- Claudia d’Amato (University of Bari, Italy)

Contact email: nai-benchmarks@googlegroups.com 

Special Call Editorial Board Members

Roberto Barile

Alessandra Mileo

Diliso Ivan

N. Fanizzi

Pierre Monnin  

Raghava Mutharaju

Reka Marta Sabou

Ilaria Tiddi  

Ute Schmid  

Philipp Cimiano

Tarek Besold

Roberto Confalonieri  

Michael Cochez

Hande McGinty  

Louis Mahon

Ernesto Jiménez-Ruiz

Stephan Scheele

Ana Ozaki

Yifan Jiang

Jaleed Khan

Deborah McGuinness

Sundong Kim

​​Wael AbdAlmageed

Anna Lisa Gentile

Michael Witbrock

Vaishak Belle

Emile van Krieken

Hossein Khojasteh

Author Guidelines

Submissions shall be made through the journal's website at https://neurosymbolic-ai-journal.com/. Prospective authors must take notice of the submission guidelines posted at https://neurosymbolic-ai-journal.com/content/author-guidelines.

Note that you need to request an account on the website to submit a paper. Please indicate in the cover letter that it is a benchmark manuscript. In the paper submission form, please select "Benchmark". All manuscripts will be reviewed based on the journal's open and transparent review policy and will be made available online during the review process.