Special Call for Neurosymbolic Benchmark Papers

Neurosymbolic Benchmark Papers are short-to-medium-sized papers (typically up to 12 pages) presenting and describing novel neurosymbolic benchmarks with corresponding tasks, metrics, and datasets. They should have the potential to become well-known references for evaluating existing and newly developed neurosymbolic solutions. 

We welcome benchmarks and datasets at different levels of maturity that provide novel scientific results, e.g., a bespoke benchmark that provides new insights into the performance of state-of-the-art tools on a specific task. Besides novel benchmarks, which are of primary interest for this call, we also welcome benchmarks that have already enjoyed significant adoption by third parties in the context of research, industry, or a specific user community (e.g., biomedicine) from which novel scientific results emerge.  

The paper should describe, in concise and clear terms, the motivation and merit for the benchmark as a novel contribution to neurosymbolic AI, highlight the targeted gaps, detail its theoretical framework and practical instantiation, formalize and detail its task, dataset and metrics, report its key statistics and illustrative examples, apply it to evaluate a relevant set of baseline approaches following clear research questions or hypotheses, and reflect on its limitations. The paper should also make it clear what property/ies, tasks, or uses of neurosymbolic systems are being addressed with the benchmark, and describe its content as a guide to its usage in various (possibly unforeseen) applications. As a neurosymbolic contribution, it should be clear how the benchmark, as well as its dataset and metrics, connect to both neural and symbolic considerations. 

Benchmark and/or datasets should follow open-access scientific guidelines, i.e., they should be published using a stable URL (e.g., Zenodo), include a name, version date and number, licensing, availability, topic coverage, source of the data, purpose and method of creation and maintenance, reported usage, etc. Ideally, benchmarks should be accompanied by software (e.g., a Python package, a HuggingFace dataset module, or a web demo) that facilitates their use. 

Benchmark contributions will be evaluated along the following dimensions: (1) Relevance and novelty of the benchmark contribution within neurosymbolic AI, e.g., in capturing NeSy system properties or evaluating a task; (2) Clarity and completeness of the description, in terms of (theoretical) justification for the design and in line with the identified research gaps; (3) Usefulness and impact of the benchmark, e.g., in highlighting strengths and weaknesses of systems; (4) Quality of the benchmark/metrics/dataset, supported through, for example, study execution, human performance or agreement, as well as objective metrics of its diversity and coverage.

Prospective authors should feel free to contact the guest editors with any questions related to this call, e.g., to assess whether a particular topic is suitable in principle.

Deadline

May 15, 2026

Earlier submissions will be processed as they come in. 

Special Call Editors

- Filip Ilievski (Vrije Universiteit Amsterdam, The Netherlands)

- Claudia d’Amato (University of Bari, Italy)

Contact email: nai-benchmarks@googlegroups.com 

Special Call Editorial Board Members

1. Wael AbdAlmageed (Clemson University)

2. Roberto Barile (Università di Bari)

3. Vaishak Belle (University of Edinburgh)

4. Tarek Besold (Sony AI)

5. Philipp Cimiano (Bielefeld University)

6. Michael Cochez (Åbo Akademi)

7. Roberto Confalonieri (University of Padua)

8. Stefano De Giorgis (VU Amsterdam)

9. Ivan Diliso (Università di Bari)

10. Nicola Fanizzi (Università di Bari)

11. Anna Lisa Gentile (IBM)

12. Yifan Jiang (USC Information Sciences Institute)

13. Ernesto Jiménez-Ruiz (St George’s, University of London)

14. Jaleed Khan (Elsewhen)

15. Hossein Khojasteh (VU Amsterdam)

16. Sundong Kim (Gwangju Institute of Science and Technology)

17. Emile van Krieken (VU Amsterdam)

18. Louis Mahon (University of Edinburgh)

19. Hande McGinty (Kansas State University)

20. Deborah McGuinness (Rensselaer Polytechnic Institute)

21. Alessandra Mileo (Dublin City University)

22. Pierre Monnin (Inria)

23. Raghava Mutharaju (IIIT Delhi)

24. Ana Ozaki (University of Oslo)

25. Reka Marta Sabou (WU Vienna)

26. Stephan Scheele (OTH Regensburg)

27. Ute Schmid (University of Bamberg)

28. Ilaria Tiddi (VU Amsterdam)

Author Guidelines

Submissions shall be made through the journal's website at https://neurosymbolic-ai-journal.com/. Prospective authors must take notice of the submission guidelines posted at https://neurosymbolic-ai-journal.com/content/author-guidelines.

Note that you need to request an account on the website to submit a paper. Please indicate in the cover letter that it is a benchmark manuscript. In the paper submission form, please select "Benchmark". All manuscripts will be reviewed based on the journal's open and transparent review policy and will be made available online during the review process.