By Anonymous User
Review Details
Reviewer has chosen to be Anonymous
Overall Impression: Weak
Content:
Technical Quality of the paper: Weak
Originality of the paper: Yes, but limited
Adequacy of the bibliography: Yes, but see detailed comments
Presentation:
Adequacy of the abstract: Yes
Introduction: background and motivation: Limited
Organization of the paper: Poor
Level of English: Satisfactory
Overall presentation: Weak
Detailed Comments:
This manuscript considers the question "what can knowledge graph alignment (KGA) approaches gain with Neuro-Symbolic approaches?". This is indeed an interesting question, a paper on which could be a very important contribution, as there is, as the authors say, a discrepancy between logical methods and learning methods based on lexical, structural, and semantic data.
However, I find the contribution of this paper to be unclear. I would expect that the paper would specify what exactly state-of-the-art methods in KGA are, what exactly are their problems, and how could they be bridged by a NeSy integration. Instead, the paper is composed of: a KGA section that focuses on defining the problem (this section does include a section on SotA methods, but their presentation focuses only on the features they use without an analysis of their pros and cons); a NeSy section that talks about NeSy broadly, not in the context of KGA; and a "challenges and opportunities" section that doesn't seem to follow on the setup of the previous sections, and points challenges that are intuitive but come out of the blue. With this organization, the reader gets lost in what exactly the paper is trying to do.
Section 1 reads relatively well. Here, I was puzzled by the phrase "Recent subsymbolic approaches, including linguistic and structural models like Attention Networks and Graph Neural Networks..." - what exactly is the linguistic approach here, attention networks? Also, later in the paper deep neural networks are considered NeSy methods - how come their structured variant is considered "subsymbolic" in §1?
Section 2:
- The notion of a KG seems somewhat narrow to graphs that rely on RDF and OWL formalisms. Note that the most widely used KGs of today, like Wikidata or ConceptNet, do not align with this formalism (while Wikidata can be translated to RDF, there is no OWL in it, and its Qualifier model is generally considered a different class of models). Property graphs neither. Does this mean that KGA is not relevant for those KGs?
- "The consistency principle [16] stipulated that every named concept ... should be satisfiable" - please describe why the consistency is applied to named concepts and not to attributes, relations, etc.
- As §2.1 has very few citations, it is unclear whether this section is considered a novel contribution (as in, formalization of the task), or whether it is adapted from prior work. Parts such as the softconsist formula trigger this question, it is good to know where this formula comes from. Similarly, 2.1.2 makes various claims (e.g., "the majority of existing ... are unsupervised" and "the candidate generation ... uses ..."), which would also benefit from support. And 2.1.3 again has no pointers to prior work.
- What is K^M (undefined)?
- §2.2 makes an effort to review the state of the art, which has several issues:
1) the review classification comes from work on ontology matching. The paper never discusses the relationship between the envisioned KGA and OM. Given the definition in 2.1 and the focus on OWL schemas, it seems like they have a lot of overlap. In that case, it is unclear what is the novelty of §2.2. If there is a difference, then the authors should motivate why the reuse of a classification of another task is appropriate.
2) the review focuses on what the systems use as knowledge sources, but at no point, is there a review of how the systems operate (what is the method). Without knowing what the method is, it becomes hard to appreciate the methodological challenges presented in §4. Instead, the section has a lot of vague phrasings that seem to deliberately avoid providing more detail: "is put forth by", "capitalizing on", "harnessing", "directing attention", "several works have been published in this area", ...
3) the classification seems rather inconsistent to me. The first class has only two subclasses with coarse granularity, while the second class has six that are very finely split (e.g., string-based vs language-based; graph-based vs instance-based).
4) at the end of §2.2, the authors single out two methods that are "go-to choices". There is, again, no description of how these methods work, how they are different than other methods, and why they are favored over other methods.
5) There is a mention of the "Bio-ML track" which is "of particular significance" because "it is the first to emphasize machine learning... for ontology alignment". Again, what is the relation between ontology alignment and KGA is not specified.
Section 3:
- I found this section to be unaligned with the topic of the paper. The section does not talk about KGA at all, instead, it provides general motivations for NeSy.
- Also, there is again a variety of arguments that are left semi-addressed:
1) "symbolic approaches face challenges" (I count only one challenge in the given paragraph, that of reliance on human input);
2) a mention of the "current limitations" of deep learning without contextualizing these in the context of KGA;
3) "Due to these limitations, ... CYC ... have largely fallen out of favor" - it is unclear that CYC has "fallen out of favor" only because it relied on people (e.g., it was largely proprietary), plus CYC did gradually shift towards including automated modules as well.
- §3.1 talks about integration, summarizing prior work by Hitzler et al. This section is also not focused on KGA, so its relevance to the paper is unclear, as is its novelty.
Section 4:
- Section 4 is arguably where the main novelty of the paper lies. The section points out issues of KGA methods and discusses how they are addressed or can be addressed. My main problem with this section is that it is not well-aligned with the rest of the paper. The mapping and repair categories, which are its organizing factors, are mentioned before but just in passing. The five challenges relate to other challenges provided in the paper, but their solutions (cells in the table) do not clearly align with the prior description of the methods. What would have been a nice contribution is to evaluate the adequacy of prior method categories (or methods) with respect to each of the challenges, and indicate which of them are most promising to solve each challenge.
- I was confused by the sudden proposal of an approach in 4.1, followed by a statement that a "similar architecture was already introduced". Maybe it would be better to start with that architecture and discuss what could be improved?
- §4.2 indicates that LLMs rely on large corpora and KG inference must be induced from thousands of examples. However, this argument misses the fact that these LLMs are actually already available, and recent paradigms like in-context learning allow us to perform inference with them without fine-tuning and without a large number of examples. Interestingly, a similar argument comes two paragraphs later, so it is perhaps a matter of consolidation.
- §4.3 (achieving correctness) as a separate challenge seems confusing. Isn't producing correct alignments the basic objective of the task? According to §2, this seems to be the case. Discussing the accuracy can certainly be beneficial, as long as the paper provides indications of what aspects of the model or the task are currently challenging and how these can be improved.
- §4.5 makes a point that all methods are non-transparent. While this statement is hard to judge as the workings of the methods are not provided in the paper, §2.2 seems to allude that some of the methods provide an intrinsic way to derive explanations. §4.5 would thus be also improved by a more granular distinction between the method families that address a challenge and those that don't.
- In §4.5, it sounds like the post-hoc explanations are an invention of the semantic web community. This is certainly not the case, as popular post-hoc explainable AI methods like LIME and SHAP mainly come from machine learning research.
Section 5:
- the conclusion seems very generic to me. With phrases like "we advocate for research endeavours that transcend singular methodologies", and an "aspiration ... that collectively surpasses prevailing state-of-the-art algorithms", it sounds like a truism rather than a summary of a vision for bringing NeSy to KGA
In summary, I do see a lot of value in the premise of this paper and I can certainly see informative content in it, especially the challenges. However, to make it a paper that can serve as a review of the state-of-the-art NeSy systems in KGA, I suggest that the paper presentation, contributions, and argumentation are thoroughly revised and re-focused.
