Reviewer has chosen not to be Anonymous

Overall Impression: Good

Content:
Technical Quality of the paper: Average
Originality of the paper: Yes, but limited
Adequacy of the bibliography: Yes

Presentation:
Adequacy of the abstract: Yes
Introduction: background and motivation: Limited
Organization of the paper: Needs improvement
Level of English: Satisfactory
Overall presentation: Good

Detailed Comments:

The paper "A Scoping Review of Neurosymbolic Reasoning over Ontologies and Knowledge Graphs: Datasets, Evaluation Practices, and Open Challenges" provides an overview of 63 articles about neurosymbolic systems utilizing ontologies and knowledge graphs for various tasks (e.g., logical query answering, link prediction, subsumption check). The authors focus on categorization of selected methods according to Kautz neurosymbolic paradigms, review 83 benchmark datasets and provide exhaustive dataset-to-task mapping, report most common metrics from the literature and summarize task-to-metric mapping. The survey then discusses dataset limitations and evaluation challenges and provides a custom comprehensive evaluation framework aiming to assess multiple aspects of proposed neurosymbolic methods (e.g., dataset characteristics, noise tolerance, reasoning capabilities etc.). The proposed work provides an interesting overview of neurosymbolic methods which integrate ontologies and knowledge graphs as symbolic components and discussions about datasets and evaluation criteria for various reasoning tasks, yet there are several things which could strengthen the survey and make it more suitable as introductory reading: e.g., some works on knowledge graph embeddings are missing and the text lacks background on knowledge graphs, ontologies and Description Logics; also, some technical details about methods reported are missing. Please, see the comments below.

Major comments:

1. Kautz categorization of neurosymbolic frameworks is a large part of the paper and one of the main contributions. It would be good to clarify this in the title of the paper (now it only mentions evaluation strategies and dataset overview).

2. In Section 'Introduction' or 'Background' more detailed information on knowledge graphs and ontologies should be added: what is a knowledge graph (formally), what is an ontology, what are Description Logics (syntax and semantics), ABox / TBox axioms, OWL / RDF, which tasks support ontology and knowledge graph reasoning (e.g., link prediction, subsumption prediction, query answering etc.), maybe brief mentioning of symbolic reasoners (ELK, HermiT etc.). Without proper introduction of all the terminology it would be difficult for a beginner to understand the rest of the paper. This also applies to datasets mentioned in the introductory part: what is FB15k, WN18 etc., which data do they contain?

3. In Section 'Introduction' some prior surveys on neurosymbolic systems operating with knowledge graphs and ontologies are mentioned, yet this list is not complete. E.g. consider the works below:

DeLong, Lauren Nicole, Ramon Fernández Mir, and Jacques D. Fleuriot. "Neurosymbolic AI for reasoning over knowledge graphs: A survey." IEEE Transactions on Neural Networks and Learning Systems 36.5 (2024): 7822-7842.

Seeliger, Arne, Matthias Pfaff, and Helmut Krcmar. "Semantic web technologies for explainable machine learning models: A literature review." PROFILES/SEMEX@ ISWC 2465 (2019): 1-16.

Steinmetz, Nadine, and Kai-Uwe Sattler. "What is in the KGQA benchmark datasets? Survey on challenges in datasets for question answering on knowledge graphs." Journal on Data Semantics 10.3 (2021): 241-265.

Dai, Yuanfei, et al. "A survey on knowledge graph embedding: Approaches, applications and benchmarks." Electronics 9.5 (2020): 750.

Zamazal, Ondřej. "A survey of ontology benchmarks for semantic web ontology tools." international Journal on Semantic Web and information Systems (iJSWiS) 16.1 (2020): 47-68.

It should be stated explicitly what novelty this survey brings compared to other similar works.

4. In Section 'Methods', subsection 'Stage 1: Identifying the research questions' RQ 1 should be more specific: the objective is not just to list the works with focus on reasoning over ontologies and knowledge graphs, yet to categorize selected works according to Kautz neurosymbolic taxonomy.

5. 'Methods' ('Stage 3: Study Selection', 'Stage 4: Charting the data' and 'Stage 5: Collating, summarizing, and reporting the results' subsections) and the beginning of 'Results' sections (everything up until 'Neurosymbolic systems (RQ 1)') need to be reorganized since there are some parts that repeat each other. I would suggest to put everything from 'Results' until 'Neurosymbolic systems (RQ 1)' to the 'Methods' section and start 'Results' with Kautz categorization. I also suggest to move the entire 'Methods' section to the Appendix since this is not the most important part of the paper.

'Stage 3: Study Selection' from 'Methods' does not include the information how many papers were selected at the end and after each stage of relevant work selection. This part can be merged with paragraphs 1-2 of 'Literature Search' from 'Results' which also reveal the process of article collection.

'Stage 4: Charting the data' from 'Methods', paragraph 3 (about benchmark datasets) needs to be joined with paragraph 3 of 'Literature Search' from 'Results' which includes information about benchmark datasets as well.

Paragraph 4 of 'Literature Search' from 'Results' about notable benchmark families can be omitted since benchmarks are discussed further in the paper.

Paragraph 2 of 'Results' section includes information that neurosymbolic systems were characterized across 4 dimensions including neural and symbolic components. It would be useful to add very brief information regarding these components (neural architectures, algorithms, symbolic reasoners, ontologies/KGs) for each work in Kautz categorization since some work descriptions are too high-level (e.g., for BEUrRE: 'combines neural uncertainty modeling with symbolic constraints that encourage global consistency').

Figure 1, 'Neural component' part needs also 'ontology embeddings' mentioning since many discussed works apply this form of ontology representation for underlying problem solving.

Figure 2 needs to be moved to 'Methods' section as well since it describes the workflow of article selection.

In Figure 2, 'Identification', the right rectangle: could you please clarify which 'automation tools' were used to assess records eligibility? In the 'Screening' part, could you please clarify which sort of 'retrieval' was applied (rectangles in the middle)?

6. In 'Neurosymbolic systems (RQ 1)' subsection of 'Results' section, it would be good to unify theme names and their purposes. E.g., theme names in Type 1 systems usually reflect problems to solve ('predicting ontology relations', 'template-based symbolic output selection'), themes in Type 2 systems focus more on methods ('symbolic enrichment followed by neural prediction', 'neural parametrization and constraint-based filtering in symbolic inference'), and some themes in Type 4 systems mix both methods and problems to solve (e.g., 'logic-regularized neural representations for complex query answering'). Since Kautz categorization is about methods, it would be relevant to stick by methods while choosing theme names in all system types and organize the works descriptions accordingly.

In this part in general some important and famous works for knowledge graph embeddings are missed, e.g. TransE, DistMult, TransH, RotatE etc.

In Type 1 systems, theme 3 'predicting ontology relations, especially subsumptions' includes one box-embedding approach, yet other geometric ontology embedding methods are operating like this: they encode individuals and concepts as geometric regions and interpret logical operators as operations over geometric objects. Despite that, some geometric ontology embedding methods (e.g., ELEmbeddings, Box2EL) are included in Type 5 systems as per Kautz although they solve similar problems and represent ontologies in a similar way.

In Type 1 systems, why EmEL++ is not grouped together with Box2EL, ELEmbeddings and other similar methods? It basically extends ELEmbeddings to include role hierarchy and role chains. As for ELK inferences, Box2EL authors use ELK inferences as ground truth to test for entailed knowledge as well. Also, why this work is not coupled with 'predicting ontology relations' work since it also evaluates ontology completion?

In Type 1 systems, EBR does not directly 'use <...> reasoner outputs as ground truth' yet rather use symbolic reasoners as SOTA methods to compare against.

In Type 2 systems, I would place DPLogic and DiffLogic into Neuro|Symbolic type since these methods have two co-equal components, KG embeddings and MLN which exchange information via EM loop.

In Type 5 systems for the second theme, it is argued that discussed methods 'support ranking or predicting axioms', yet they predict axioms via ranking (via rank calculation of axioms from the test set among similar ones).

7. In 'Evaluation Methodology (RQ 2)' subsection of 'Results' section, please elaborate more on datasets which are used just in one work among selected ones; why they are not widely adopted? Also, add more information about OWL benchmark suites reported (LUBM, ORE, OWL2Bench), which ontologies are included there, how do they look like, in which works were they mentioned, for which tasks were they applied in these works? These datasets collections are not reported in dataset tables.

'The co-existence of KG-style benchmarks and OWL-centric resources suggests that current neurosymbolic evaluation spans two partially distinct traditions: one emphasizes performance on predictive KG tasks, whereas the other emphasizes deductive reasoning' - this claim might not be true: some of reported ontology embedding methods (e.g., ELEmbeddings, Box2EL etc.) try to predict new knowledge (e.g., missing subsumptions), not entailed axioms. Similar claim can be found in 'Conclusion' section, RQ 3.

For Tables 8 and 11 from the Appendix, it would be interesting to add ontology DL expressivity for ontology datasets to see which methods target certain DL expressivity for ontology reasoning tasks.

8. For 'Tasks and Metrics (RQ 2.2)' subsection of 'Results' section, it would be useful to specify whether reported metrics are task-specific or general. As for metrics reported beyond the 10 most common ones, it would be useful not just to list them, yet to mention what they measure, how they are calculated, what do they mean. It would be good to discuss here which limitations do reported metrics have, how they miss some aspects of methods evaluation etc., for now only critical analysis of benchmark sets is present.

In the text accompanying Table 9, in G3 part comments about 'Membership' are missing although membership task is mentioned in the table.

'Ontology-centric structure induction was also prominent.' - could you please clarify what does this sentence mean?

G5 and G6 topics are mentioned briefly, it would be good to explain what they mean (although this part is contained in the Appendix entirely).

9. 'Discussion' section, 'Limitations and future improvements (RQ 3)' subsection:

'First, the benchmark landscape is fragmented across two partially distinct traditions: widely adopted KG completion datasets <...> and ontology-centric OWL/DL benchmarks (e.g., LUBM, ORE and OWL2Bench).' - what about other ontologies or LQA datasets?

As for Table 10 (benchmark limitations), were these limitations extracted from the results of Query 4 from the Appendix?

As for Figure 3, are these recommendations universal for both KGs and ontologies? Are there any specific metrics / evaluation strategies tailored for KGs or ontologies?

In 'Evaluation metrics' part of the five-dimensional evaluation framework, why MRR/Hits@K, precision/recall/F1/accuracy are chosen among standard metrics?

10. Supporting Material, 'Full Search Queries' part: in Query 6 'Extended coverage', were "knowledge graph embedding" approach included and "link prediction" task? If no, why not?

Minor comments:

1. The font in Figure 1 is too small, please enlarge it for easier readability.

2. On page 15 there's a typo: 'ALCconstraints' -> 'ALC constraints'.

3. On page 31 in the text accompaning Table 9, it might be better to renumber G-themes so that G1 appears first in the text, G2 second etc., not G2 first, then G3, then G4, and finally G1.

Reviewer has chosen to be Anonymous

Overall Impression: Weak

Content:
Technical Quality of the paper: Weak
Originality of the paper: Yes, but limited
Adequacy of the bibliography: Yes, but see detailed comments

Presentation:
Adequacy of the abstract: Yes
Introduction: background and motivation: Limited
Organization of the paper: Poor
Level of English: Satisfactory
Overall presentation: Weak

Detailed Comments:

The paper's scope is relevant and could be beneficial for many researchers approaching the NeSy domain.
However, the paper presents, in my opinion, two main problems:

1. Kautz Neuro-symbolic taxonomy adoption
While it is true that Kautz taxonomy is well known and has been adopted in several works, one of the most problematic aspects is that it does not provide very rigorous types definition. It stemmed out of a nice lecture, but the majority of NeSy systems, if analysed in their actual componency, can be classified by more than one type, and that's because this taxonomy lacks the level of detail that in this context would be necessary. Adopting it as the backbone to classify previous existing work unfortunately does not grant the immediate understandability of the classified approaches, which is, instead, very relevant as scope of a survey paper.

2. Lack of clarity: the scope of a survey paper is to provide a clear description of (a specific aspect of) the state of the art in a domain. The organisation of the paper is, in my opinion, poor, and it is very difficult to get an idea of the domain from this work.
Some examples (detailed in the followings):

- tables express in a more schematic way what it is said in the text, and the text briefly summarises what can be found in tables, but they do not add anything one to the other. This results in several tables with a column "Task" which is barely mentioned, and which is instead potentially fairly relevant, whose entries are keywords left to be interpreted by the reader.

- the reason for clustering papers in "Types of neurosymbolic reasoners" is left to a few lines sentences such as "overall, type X systems in our corpus..." while it should be clearly stated and highlighted at the very beginning, including what are the details of that specific cluster, architectural patterns, nuances based on their scope, etc. This information is mostly missing from the main paper, or sparse, or not detailed enough.

For the above mentioned reasons, to the best of my knowledge and understanding, the paper would need a substantial revision and rework, being, in its current state, not suitable for publication.

Additional general remarks:

It is not always clear how the clustering of "themes" is done. They start appearing on p. 11, and they are used then to introduce clusters of works considered, but it is not clarified if they are extracted bottom up, analysing the selected corpus, if they are regularities

The paper would benefit of more clear figures, showing difference in architectures, possibly with the adoption of a more standard terminology, such as the Boxology approach [1]

[1]: van Bekkum, M., de Boer, M., van Harmelen, F., Meyer-Vitali, A., & Teije, A. T. (2021). Modular design patterns for hybrid learning and reasoning systems: a taxonomy, patterns and use cases. Applied Intelligence, 51(9), 6528-6546.

==== Detailed Review ====

p. 2: "Knowledge graphs and ontologies provide structured and explicit representations of domain knowledge that are accessible to both neural and symbolic reasoning modules" --> what does it mean in this context "that are accessible"?

p. 2: "knowledge representation and reasoning (KRR) frameworks, particularly knowledge graphs (KGs) and ontologies" --> would you define knowledge graphs and ontologies as frameworks in this context?

p. 2: "and YAGO-based resources" --> which resources are you referring to? re reference points at YAGO, but here are mentioned "YAGO-based rescources"

p.4: Table 1 here is not necessary, one possible rework would be to describe in more detail the specific type in the corresponding part of the paper.

p. 4: the Background section is pretty short, and the same 4-5 papers appears repeated in different sentences before and after Table 1.

p.6: RQ1 is very broad, what does it mean here to "reason over ontologies and knowledge graphs" ?

p. 6: "The initial search retrieved 984 records from Google Scholar, 202 from the ACM Digital Library, and 81 from IEEE Xplore" --> which timespan did you consider?

p. 7: it is probably not necessary to declare both inclusion and exclusion criteria, at least not in this detail, especially if one criterion is e.g. “English only” which automatically excludes "Publications not written in English."

p. 8: "(3) omission of required components such as ontologies..." --> omission in the sense that it was not included as material or that it was not used? Also, the definition of KG and ontology is per se ambiguous, what were the inclusion criteria here?

p. 8: "The database search yielded 984 records from Google Scholar, 202 from the ACM Digital Library, and 81 from IEEE Xplore." --> this exact sentence is present on page 6. Maybe it would be useful to provide some more statistics? For example the percentage of journal papers? Of conference papers? etc.

p.9: if "Neural Systems" and "Symbolic Systems" are "out of scope", why do you have them in the image? It would be enough a brief sentence in the text, but the image should highlight the main and most relevant aspects.

p. 9: "Datasets were excluded if they primarily contained image-based or lexical data" --> why?

p. 10: Fig. 2: what is the meaning of the arrows here?

p. 10: here the paper would benefit of some diagrammatic language such as Boxology

p. 11: "In our corpus, one theme is template-based symbolic output selection, where HTL (Hereditary attentive Tree-LSTM) (Gomes et al., 2022) represents..." --> how come that it is one “theme”, and then there is only one paper cited here? a big, well organised table with clusters and dimension is necessary for a survey paper

p. 11: "Table 2. Type 1 neurosymbolic reasoners." --> are they only "reasoners"?

p. 11 Table 2: if one of the criteria is the shape of the ontology - KG, why are these not described here? not even the domain?

p. 12: "A second theme is..." here again, a figure / diagram of the components of the architecture could be useful

p. 12: "ontology evolution" --> have you defined ontology evolution before?

p. 14: "themes" division here is chaotic, it would be clearer if divided in paragraphs

p. 15: at the top of the table it would be good to have a short description of the neuro-symbolic architecture type, otherwise it is difficult to follow

p. 15: "to satisfy ALC constraints" --> I think there is a missing space after ALC, but also: what if the reader does not have familiarity with logic language acronyms? a short description would be beneficial

p. 16: given that a survey paper should be the reading to start from to have an idea of a domain, here what is provided is a list of works, organised (subjectively) according to a taxonomy, but no further explanation is provided.

p. 16: "Overall, Type 4 systems..." --> This is what the paper should provide, but not as a single sentence, but detailed grouping of different aspects of several works

p. 18: "constraints for completion and approximate deduction." --> What do you mean here with “approximate deduction”?

p. 22: "Similarly, the commonsense reasoner..." --> why is it a "commonsense reasoner"?

p. 24: "Overall, these findings suggest that neurosymbolic evaluation currently draws on..." --> this formulations should be avoided in survey papers, these are not findings, it should be the systematic analysis of a set of papers and organisation according to certain dimensions, clustering, and explicit description of the rationale

p. 25: Table 8 could benefit of a rework to make it more readable, maybe adding one column per each work considered and some ticks, if they are all relevant, otherwise show only meaningful clusters

p. 26-28: In some cases the column "Domain" it is the actual domain, in others it is the extended name of the Dataset, in others it seems to be the scope…?

p. 29: "ComplexWebQuestions" overlaps with next cell

p. 30: This list of metrics lacks a bit of context, e.g. Hits@K out of how many works focused on the same task? and is this related to which task specifically?

p. 31: Table 9 starts at the end of the page, and then again at the beginning of the subsequent page, but lacking the “G1” specification?

p. 35: Fig. 3 actually helps a lot, but a. it should appear much earlier in the paper, not as a surprise towards the end, it is one of the main outcomes and it deserves to be highlighted. However, it could be formatted in a better way: it is not clear why the entries are on two columns, given that you have the space. It is necessary also to motivate the choice of the dimensions in advance, not after the limitations

p. 37: "Our synthesis highlights three main limitations of..." --> this is another outcome, identifying some limitations allows to extend and refine the area of research, it deserves to be listed among the main contributions